CKA: Troubleshooting
30% of the CKA exam, 5 competencies. Back to the CKA overview.
Each competency lists what to know and the official documentation to read, then a demo you can run. Demos run on the demo lab, a throwaway kind cluster on your own machine; set it up once before you start.
Troubleshoot clusters and nodes
Section titled “Troubleshoot clusters and nodes”Know
- Check
kubectl get nodes, thenkubectl describe nodeconditions. On the node:systemctl status kubelet,journalctl -u kubelet, and the container runtime. - Common causes: kubelet stopped or misconfigured, expired certificates, CNI missing, disk or memory pressure.
Read
Demo: Take a node down and bring it back
-
Stop the kubelet on a worker and watch the control plane notice.
Terminal window docker exec kind-worker systemctl stop kubeletkubectl get nodes -w # wait for kind-worker to show NotReady (about 40s), then Ctrl-C -
Diagnose it the way you would on a real node.
Terminal window kubectl describe node kind-worker | sed -n '/Conditions:/,/Addresses:/p'docker exec kind-worker systemctl status kubelet --no-pager | head -5docker exec kind-worker journalctl -u kubelet --no-pager -n 20 -
Fix it, then drain and uncordon as you would for maintenance.
Terminal window docker exec kind-worker systemctl start kubeletkubectl wait --for=condition=Ready node/kind-worker --timeout=120skubectl drain kind-worker --ignore-daemonsets --delete-emptydir-datakubectl get node kind-workerkubectl uncordon kind-worker
What you should see: The node goes NotReady with conditions reading Kubelet stopped posting node status, the unreachable taint is added, and the journal shows the stop. After the drain the node shows SchedulingDisabled until you uncordon it.
Practise
- Stop the kubelet on a worker, watch the node go NotReady, find the cause from
journalctl, and fix it.
On this site
Troubleshoot cluster components
Section titled “Troubleshoot cluster components”Know
- On kubeadm clusters the control plane runs as static pods from
/etc/kubernetes/manifests. A typo there stops the component;crictl ps -aandcrictl logsshow why. - Check API server health with
kubectl get --raw=/readyz?verbose.
Read
Demo: Break the scheduler, then fix it
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace cka-componentskubectl config set-context --current --namespace=cka-components -
Control plane components are static pods: the kubelet runs whatever is in this directory.
Terminal window docker exec kind-control-plane ls /etc/kubernetes/manifestskubectl get pods -n kube-system -l tier=control-plane -
Back up the scheduler manifest and point it at a config file that doesn’t exist.
Terminal window docker exec kind-control-plane cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/kube-scheduler.yaml.bakdocker exec kind-control-plane sed -i 's|--kubeconfig=/etc/kubernetes/scheduler.conf|--kubeconfig=/etc/kubernetes/nope.conf|' /etc/kubernetes/manifests/kube-scheduler.yamlsleep 30; kubectl create deployment stuck --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 -- /agnhost pausekubectl get pods -n kube-system -l component=kube-scheduler; kubectl get pods -
Find the error where it is written, then restore.
Terminal window docker exec kind-control-plane crictl ps -a --name kube-schedulerdocker exec kind-control-plane sh -c 'crictl logs $(crictl ps -a --name kube-scheduler -q | head -1)' 2>&1 | tail -5docker exec kind-control-plane cp /root/kube-scheduler.yaml.bak /etc/kubernetes/manifests/kube-scheduler.yamlsleep 30; kubectl get pods
What you should see: The new pod stays Pending with no events at all, because nothing is scheduling. The scheduler container keeps exiting and its log names the missing file. When the API server itself is down, kubectl is useless, so crictl on the node is the tool to remember.
Note: The restore step matters: without it the lab cluster stays broken. If you lose the backup, recreate the cluster.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace cka-componentsPractise
- Break the kube-scheduler manifest, observe new pods stay Pending with no events, and repair it.
On this site
Monitor cluster and application resource usage
Section titled “Monitor cluster and application resource usage”Know
kubectl top nodesandkubectl top podsneed metrics-server. Sort with--sort-by=cpuor--sort-by=memory.
Read
Demo: Find the pod using the most
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace cka-usagekubectl config set-context --current --namespace=cka-usage -
Generate some load and read it with
top.Terminal window kubectl create deployment burner --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sh -c 'while true; do :; done'kubectl set resources deployment burner --requests=cpu=100m,memory=64Mi --limits=cpu=500m,memory=256Misleep 60; kubectl top pods --containerskubectl top nodes -
Compare usage with what was requested on each node.
Terminal window kubectl describe node kind-worker | sed -n '/Allocated resources:/,/Events:/p'
What you should see: top shows the burner near its 500m CPU limit (it is throttled, not killed). The node’s “Allocated resources” shows requests, which is what the scheduler counts; usage and requests are different numbers and both matter.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace cka-usagePractise
- Find the pod using the most memory across all namespaces in one command.
On this site
Manage and evaluate container output streams
Section titled “Manage and evaluate container output streams”Know
kubectl logsreads stdout and stderr. Use-cfor a container,--previousfor the last crashed instance,-fto follow,--sinceto limit.- Logs live on the node under
/var/log/podsand are rotated by the kubelet.
Read
Demo: Logs from many pods, and from a dead container
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace cka-logskubectl config set-context --current --namespace=cka-logs -
Two containers in one pod, writing to stdout and stderr.
Terminal window kubectl apply -f - <<'EOF'apiVersion: v1kind: Podmetadata:name: chattylabels:app: chattyspec:containers:- name: outimage: registry.k8s.io/e2e-test-images/busybox:1.37.0-2command: ["sh", "-c", "while true; do echo out $(date +%T); sleep 2; done"]- name: errimage: registry.k8s.io/e2e-test-images/busybox:1.37.0-2command: ["sh", "-c", "while true; do echo err $(date +%T) >&2; sleep 3; done"]EOFkubectl wait --for=condition=Ready pod/chatty --timeout=120skubectl logs chatty -c err --tail=2kubectl logs -l app=chatty --all-containers --prefix --since=10s -
Logs of a container that already crashed.
Terminal window kubectl run crasher --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sh -c 'echo "about to fail"; exit 3'sleep 20; kubectl logs crasher --previous -
Where the kubelet keeps those logs on the node.
Terminal window NODE=$(kubectl get pod chatty -o jsonpath='{.spec.nodeName}')docker exec $NODE sh -c 'ls /var/log/pods/ | grep cka-logs'
What you should see: --prefix labels each line with pod and container. --previous shows the crashed run’s output; without it you would see the current, possibly empty, run. On the node, logs live under /var/log/pods/, which is what log shippers read.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace cka-logsPractise
- Get the logs of the previous instance of a crashing init container.
On this site
Troubleshoot services and networking
Section titled “Troubleshoot services and networking”Know
- Work hop by hop: DNS name, Service selector and EndpointSlices, kube-proxy or CNI forwarding, then the container port.
- A
connection refusedis usually the port; a timeout is usually a policy or routing drop.
Read
Demo: A Service that routes nowhere
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace cka-svckubectl config set-context --current --namespace=cka-svc -
Create a Service whose selector doesn’t match the pods.
Terminal window kubectl create deployment api --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 -- /agnhost netexec --http-port=8080kubectl create service clusterip api --tcp=80:8080kubectl patch service api -p '{"spec":{"selector":{"app":"apii"}}}'kubectl run client --rm -i --restart=Never --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- wget -qO- -T 3 http://api/hostname -
Work through the chain: Service, EndpointSlice, pod labels.
Terminal window kubectl get endpointslices -l kubernetes.io/service-name=apikubectl get service api -o jsonpath='{.spec.selector}{"\n"}'kubectl get pods --show-labels -
Fix the selector and check DNS resolution while you are there.
Terminal window kubectl patch service api -p '{"spec":{"selector":{"app":"api"}}}'kubectl run client --rm -i --restart=Never --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sh -c 'nslookup api; wget -qO- -T 3 http://api/hostname'
What you should see: The first call times out and the EndpointSlice has no endpoints. The labels show app=api while the selector says apii. After the fix the name resolves to the ClusterIP and the call returns a pod name. Empty endpoints is the first thing to check for any “Service doesn’t work” problem.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace cka-svcPractise
- Break a Service selector and fix it using only EndpointSlices and labels.
On this site
Domain and competency names are from the CNCF CKA curriculum (v1.37), cncf/curriculum, licensed CC BY 4.0. Summaries, demos, and exercises are original to this site.