CKAD: Application Observability and Maintenance
15% of the CKAD exam, 5 competencies. Back to the CKAD overview.
Each competency lists what to know and the official documentation to read, then a demo you can run. Demos run on the demo lab, a throwaway kind cluster on your own machine; set it up once before you start.
Understand API deprecations
Section titled “Understand API deprecations”Know
- API versions are removed on a schedule. A manifest using a removed version fails with “no matches for kind”. Check the deprecation guide before each upgrade.
Read
Demo: Find the versions your cluster serves
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace ckad-deprecationskubectl config set-context --current --namespace=ckad-deprecations -
List served API versions and check one resource.
Terminal window kubectl api-versions | grep -E 'policy|autoscaling|flowcontrol'kubectl explain horizontalpodautoscaler | head -3kubectl explain horizontalpodautoscaler --api-version=autoscaling/v2 | head -3 -
Apply a manifest written for an API that was removed.
Terminal window kubectl apply -f - <<'EOF'# expected to fail: policy/v1beta1 PodDisruptionBudget was removed in v1.25apiVersion: policy/v1beta1kind: PodDisruptionBudgetmetadata:name: oldspec:minAvailable: 1selector:matchLabels:app: webEOF -
See whether anything is still calling deprecated APIs.
Terminal window kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis || echo "no deprecated API calls recorded"
What you should see: policy/v1beta1 isn’t listed and the apply fails with no matches for kind "PodDisruptionBudget" in version "policy/v1beta1". Fix the manifest by changing apiVersion to policy/v1. The metric names any deprecated API a client has called since the API server started; check it before every upgrade.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace ckad-deprecationsOn this site
Implement probes and health checks
Section titled “Implement probes and health checks”Know
- Startup gates the others, liveness restarts, readiness controls traffic. Liveness should never check dependencies.
Read
Demo: Startup, readiness, and liveness doing different jobs
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace ckad-probeskubectl config set-context --current --namespace=ckad-probes -
A slow-starting app with all three probes.
Terminal window kubectl apply -f - <<'EOF'apiVersion: apps/v1kind: Deploymentmetadata:name: probedspec:replicas: 2selector:matchLabels:app: probedtemplate:metadata:labels:app: probedspec:containers:- name: appimage: registry.k8s.io/e2e-test-images/agnhost:2.66.1command: ["sh", "-c", "sleep 20; exec /agnhost netexec --http-port=8080"]startupProbe:httpGet: {path: /healthz, port: 8080}periodSeconds: 5failureThreshold: 12readinessProbe:httpGet: {path: /readyz, port: 8080}periodSeconds: 5livenessProbe:httpGet: {path: /healthz, port: 8080}periodSeconds: 10EOFkubectl expose deployment probed --port=80 --target-port=8080kubectl get pods -l app=probed -w # READY goes 0/1 to 1/1 after about 20s; Ctrl-C -
Make one pod fail its liveness probe.
Terminal window POD=$(kubectl get pod -l app=probed -o jsonpath='{.items[0].metadata.name}')kubectl exec $POD -- wget -qO- 'http://localhost:8080/exit?code=1' || truekubectl get endpointslices -l kubernetes.io/service-name=probed -o jsonpath='{range .items[*].endpoints[*]}{.targetRef.name}{" ready="}{.conditions.ready}{"\n"}{end}'sleep 30; kubectl get pod $PODkubectl describe pod $POD | sed -n '/Events:/,$p'
What you should see: Without the startup probe, a liveness probe would kill the app during its 20-second start. During the failure the pod is marked not ready in the EndpointSlice, so traffic goes to the other pod, and the container restarts. Readiness takes a pod out of traffic; liveness restarts it.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace ckad-probesPractise
- Add an HTTP readiness probe and a TCP liveness probe, then break each one and compare the events.
On this site
Use built-in CLI tools to monitor Kubernetes applications
Section titled “Use built-in CLI tools to monitor Kubernetes applications”Know
kubectl get -w,describe,top,events --for, and-o wideor JSONPath output answer most questions.
Read
Demo: Watch an app with kubectl alone
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace ckad-monitorkubectl config set-context --current --namespace=ckad-monitor -
Something to watch.
Terminal window kubectl create deployment web --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --replicas=3 -- /agnhost netexec --http-port=8080kubectl rollout status deployment/web -
Sort, pick columns, and watch.
Terminal window kubectl get pods -o wide --sort-by=.metadata.creationTimestampkubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount,IP:.status.podIPkubectl top pods --sort-by=cpukubectl events --watch &kubectl scale deployment web --replicas=5; sleep 10; kill %1 -
Wait for a condition instead of polling.
Terminal window kubectl scale deployment web --replicas=2kubectl wait --for=jsonpath='{.status.readyReplicas}'=2 deployment/web --timeout=60s
What you should see: Custom columns give a one-line health view per pod, events --watch streams scaling events as they happen, and kubectl wait blocks until the condition is true: the right tool in scripts and in the exam.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace ckad-monitorPractise
- List pods sorted by restart count using
--sort-by.
Utilize container logs
Section titled “Utilize container logs”Know
kubectl logswith-c,--previous,-f,--since,--tail, and-lfor a label selector across pods.
Read
Demo: Follow, filter, and find logs fast
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace ckad-logskubectl config set-context --current --namespace=ckad-logs -
Several pods that log.
Terminal window kubectl create deployment logger --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 --replicas=3 -- sh -c 'i=0; while true; do i=$((i+1)); echo "$(hostname) line $i"; sleep 1; done'kubectl rollout status deployment/logger -
Read them in different ways.
Terminal window kubectl logs deployment/logger --tail=2kubectl logs -l app=logger --prefix --since=5skubectl logs -l app=logger --max-log-requests=5 --tail=1 --timestampstimeout 5 kubectl logs -f -l app=logger --prefix || true
What you should see: logs deployment/logger reads one pod picked for you; a label selector reads all of them, --prefix says which line came from which pod, and --since cuts the noise. Logs are stdout and stderr only: anything the app writes to a file inside the container won’t show up here.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace ckad-logsDebugging in Kubernetes
Section titled “Debugging in Kubernetes”Know
- Events and status first, then logs, then
kubectl execorkubectl debugwith an ephemeral container.
Read
Demo: A pod that runs but doesn’t serve
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace ckad-debugkubectl config set-context --current --namespace=ckad-debug -
A Service and Deployment that look fine but disagree on the port.
Terminal window kubectl create deployment api --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 -- /agnhost netexec --http-port=9090kubectl expose deployment api --port=80 --target-port=8080kubectl rollout status deployment/apikubectl run client --rm -i --restart=Never --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- wget -qO- -T 3 http://api/hostname -
Narrow it down: is the pod serving, and on which port?
Terminal window kubectl get endpointslices -l kubernetes.io/service-name=api -o jsonpath='{.items[0].ports}{"\n"}'kubectl port-forward deployment/api 9090:9090 &sleep 2; curl -s localhost:9090/hostname; echo; kill %1POD=$(kubectl get pod -l app=api -o name | head -1)kubectl debug -it $POD --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 --target=agnhost -- sh -c 'netstat -tlnp' -
Fix the Service.
Terminal window kubectl patch service api -p '{"spec":{"ports":[{"port":80,"targetPort":9090}]}}'kubectl run client --rm -i --restart=Never --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- wget -qO- -T 3 http://api/hostname; echo
What you should see: The endpoints exist (the selector is right) but list port 8080. Port-forwarding to 9090 works, and netstat in the debug container shows the app listening on 9090. The Service’s targetPort was wrong; after the patch the call works.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace ckad-debugOn this site
Domain and competency names are from the CNCF CKAD curriculum (v1.37), cncf/curriculum, licensed CC BY 4.0. Summaries, demos, and exercises are original to this site.