Ensure Kubernetes Deployment Complete and All Containers Updated and Available

Question

The deployment status indicates that you can see the observedGeneration vs generation deployments and when observedGeneration >= generation then the deployment succeeded. This is good, but I'm interested to know when the new container is actually running in all my containers, so if I get into the service, I know for sure that I’m pushing the server representing the last deployed container.

Another tip from K8S Slack member:

 kubectl get deployments | grep <deployment-name> | sed 's/ /,/g' | cut -d ' ' -f 4 

I deployed a bad image, resulting in ErrImagePull , but the deployment still reports the correct number of 8 replicas of the date (there were 7 replicas available).

Answer 1

Update # 2: Kubernetes 1.5 will ship with a much better version of kubectl rollout status and will be further improved in version 1.6, possibly replacing my custom / script solution below.

Update # 1: I turned my answer into a script hosted on Github that got a small amount of enhancement PRs today.

Original answer:

First of all, I believe that the kubectl command you received is incorrect: it replaces all spaces with commas, but then tries to get the 4th field after separating with spaces.

To verify that the deployment (or upgrade) made it for all containers, I think you should check if the number of available replicas matches the number of replicas required. That is, the columns AVAILABLE and DESIRED in the output of kubectl are equal. So far, you can get the number of available replicas (5th column) through

kubectl get deployment nginx | tail -n +2 | awk '{print $5}'

and the number of replicas required (2nd column) through

kubectl get deployment nginx | tail -n +2 | awk '{print $2}'

a cleaner way is to use kubectl output, especially if you want to accept the generation requirement, which is also mentioned in the official documentation.

Here's a quick bash script that I wrote that expects to get the deployment name on the command line, waits for the observed generation to become specified, and then expects the available replicas to reach the number specified:

 #!/bin/bash set -o errexit set -o pipefail set -o nounset deployment= get_generation() { get_deployment_jsonpath '{.metadata.generation}' } get_observed_generation() { get_deployment_jsonpath '{.status.observedGeneration}' } get_replicas() { get_deployment_jsonpath '{.spec.replicas}' } get_available_replicas() { get_deployment_jsonpath '{.status.availableReplicas}' } get_deployment_jsonpath() { local readonly _jsonpath="$1" kubectl get deployment "${deployment}" -o "jsonpath=${_jsonpath}" } if [[ $# != 1 ]]; then echo "usage: $(basename $0) <deployment>" >&2 exit 1 fi readonly deployment="$1" readonly generation=$(get_generation) echo "waiting for specified generation ${generation} to be observed" while [[ $(get_observed_generation) -lt ${generation} ]]; do sleep .5 done echo "specified generation observed." readonly replicas="$(get_replicas)" echo "specified replicas: ${replicas}" available=-1 while [[ ${available} -ne ${replicas} ]]; do sleep .5 available=$(get_available_replicas) echo "available replicas: ${available}" done echo "deployment complete."