Skip to content

CKAD: Application Observability and Maintenance

15% of the CKAD exam, 5 competencies. Back to the CKAD overview.

Each competency lists what to know and the official documentation to read, then a demo you can run. Demos run on the demo lab, a throwaway kind cluster on your own machine; set it up once before you start.

Know

  • API versions are removed on a schedule. A manifest using a removed version fails with “no matches for kind”. Check the deprecation guide before each upgrade.

Read

Demo: Find the versions your cluster serves

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace ckad-deprecations
    kubectl config set-context --current --namespace=ckad-deprecations
  2. List served API versions and check one resource.

    Terminal window
    kubectl api-versions | grep -E 'policy|autoscaling|flowcontrol'
    kubectl explain horizontalpodautoscaler | head -3
    kubectl explain horizontalpodautoscaler --api-version=autoscaling/v2 | head -3
  3. Apply a manifest written for an API that was removed.

    Terminal window
    kubectl apply -f - <<'EOF'
    # expected to fail: policy/v1beta1 PodDisruptionBudget was removed in v1.25
    apiVersion: policy/v1beta1
    kind: PodDisruptionBudget
    metadata:
    name: old
    spec:
    minAvailable: 1
    selector:
    matchLabels:
    app: web
    EOF
  4. See whether anything is still calling deprecated APIs.

    Terminal window
    kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis || echo "no deprecated API calls recorded"

What you should see: policy/v1beta1 isn’t listed and the apply fails with no matches for kind "PodDisruptionBudget" in version "policy/v1beta1". Fix the manifest by changing apiVersion to policy/v1. The metric names any deprecated API a client has called since the API server started; check it before every upgrade.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace ckad-deprecations

On this site

Know

  • Startup gates the others, liveness restarts, readiness controls traffic. Liveness should never check dependencies.

Read

Demo: Startup, readiness, and liveness doing different jobs

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace ckad-probes
    kubectl config set-context --current --namespace=ckad-probes
  2. A slow-starting app with all three probes.

    Terminal window
    kubectl apply -f - <<'EOF'
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: probed
    spec:
    replicas: 2
    selector:
    matchLabels:
    app: probed
    template:
    metadata:
    labels:
    app: probed
    spec:
    containers:
    - name: app
    image: registry.k8s.io/e2e-test-images/agnhost:2.66.1
    command: ["sh", "-c", "sleep 20; exec /agnhost netexec --http-port=8080"]
    startupProbe:
    httpGet: {path: /healthz, port: 8080}
    periodSeconds: 5
    failureThreshold: 12
    readinessProbe:
    httpGet: {path: /readyz, port: 8080}
    periodSeconds: 5
    livenessProbe:
    httpGet: {path: /healthz, port: 8080}
    periodSeconds: 10
    EOF
    kubectl expose deployment probed --port=80 --target-port=8080
    kubectl get pods -l app=probed -w # READY goes 0/1 to 1/1 after about 20s; Ctrl-C
  3. Make one pod fail its liveness probe.

    Terminal window
    POD=$(kubectl get pod -l app=probed -o jsonpath='{.items[0].metadata.name}')
    kubectl exec $POD -- wget -qO- 'http://localhost:8080/exit?code=1' || true
    kubectl get endpointslices -l kubernetes.io/service-name=probed -o jsonpath='{range .items[*].endpoints[*]}{.targetRef.name}{" ready="}{.conditions.ready}{"\n"}{end}'
    sleep 30; kubectl get pod $POD
    kubectl describe pod $POD | sed -n '/Events:/,$p'

What you should see: Without the startup probe, a liveness probe would kill the app during its 20-second start. During the failure the pod is marked not ready in the EndpointSlice, so traffic goes to the other pod, and the container restarts. Readiness takes a pod out of traffic; liveness restarts it.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace ckad-probes

Practise

  • Add an HTTP readiness probe and a TCP liveness probe, then break each one and compare the events.

On this site

Use built-in CLI tools to monitor Kubernetes applications

Section titled “Use built-in CLI tools to monitor Kubernetes applications”

Know

  • kubectl get -w, describe, top, events --for, and -o wide or JSONPath output answer most questions.

Read

Demo: Watch an app with kubectl alone

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace ckad-monitor
    kubectl config set-context --current --namespace=ckad-monitor
  2. Something to watch.

    Terminal window
    kubectl create deployment web --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --replicas=3 -- /agnhost netexec --http-port=8080
    kubectl rollout status deployment/web
  3. Sort, pick columns, and watch.

    Terminal window
    kubectl get pods -o wide --sort-by=.metadata.creationTimestamp
    kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount,IP:.status.podIP
    kubectl top pods --sort-by=cpu
    kubectl events --watch &
    kubectl scale deployment web --replicas=5; sleep 10; kill %1
  4. Wait for a condition instead of polling.

    Terminal window
    kubectl scale deployment web --replicas=2
    kubectl wait --for=jsonpath='{.status.readyReplicas}'=2 deployment/web --timeout=60s

What you should see: Custom columns give a one-line health view per pod, events --watch streams scaling events as they happen, and kubectl wait blocks until the condition is true: the right tool in scripts and in the exam.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace ckad-monitor

Practise

  • List pods sorted by restart count using --sort-by.

Know

  • kubectl logs with -c, --previous, -f, --since, --tail, and -l for a label selector across pods.

Read

Demo: Follow, filter, and find logs fast

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace ckad-logs
    kubectl config set-context --current --namespace=ckad-logs
  2. Several pods that log.

    Terminal window
    kubectl create deployment logger --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 --replicas=3 -- sh -c 'i=0; while true; do i=$((i+1)); echo "$(hostname) line $i"; sleep 1; done'
    kubectl rollout status deployment/logger
  3. Read them in different ways.

    Terminal window
    kubectl logs deployment/logger --tail=2
    kubectl logs -l app=logger --prefix --since=5s
    kubectl logs -l app=logger --max-log-requests=5 --tail=1 --timestamps
    timeout 5 kubectl logs -f -l app=logger --prefix || true

What you should see: logs deployment/logger reads one pod picked for you; a label selector reads all of them, --prefix says which line came from which pod, and --since cuts the noise. Logs are stdout and stderr only: anything the app writes to a file inside the container won’t show up here.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace ckad-logs

Know

  • Events and status first, then logs, then kubectl exec or kubectl debug with an ephemeral container.

Read

Demo: A pod that runs but doesn’t serve

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace ckad-debug
    kubectl config set-context --current --namespace=ckad-debug
  2. A Service and Deployment that look fine but disagree on the port.

    Terminal window
    kubectl create deployment api --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 -- /agnhost netexec --http-port=9090
    kubectl expose deployment api --port=80 --target-port=8080
    kubectl rollout status deployment/api
    kubectl run client --rm -i --restart=Never --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- wget -qO- -T 3 http://api/hostname
  3. Narrow it down: is the pod serving, and on which port?

    Terminal window
    kubectl get endpointslices -l kubernetes.io/service-name=api -o jsonpath='{.items[0].ports}{"\n"}'
    kubectl port-forward deployment/api 9090:9090 &
    sleep 2; curl -s localhost:9090/hostname; echo; kill %1
    POD=$(kubectl get pod -l app=api -o name | head -1)
    kubectl debug -it $POD --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 --target=agnhost -- sh -c 'netstat -tlnp'
  4. Fix the Service.

    Terminal window
    kubectl patch service api -p '{"spec":{"ports":[{"port":80,"targetPort":9090}]}}'
    kubectl run client --rm -i --restart=Never --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- wget -qO- -T 3 http://api/hostname; echo

What you should see: The endpoints exist (the selector is right) but list port 8080. Port-forwarding to 9090 works, and netstat in the debug container shows the app listening on 9090. The Service’s targetPort was wrong; after the patch the call works.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace ckad-debug

On this site


Domain and competency names are from the CNCF CKAD curriculum (v1.37), cncf/curriculum, licensed CC BY 4.0. Summaries, demos, and exercises are original to this site.