Skip to content

CKA: Troubleshooting

30% of the CKA exam, 5 competencies. Back to the CKA overview.

Each competency lists what to know and the official documentation to read, then a demo you can run. Demos run on the demo lab, a throwaway kind cluster on your own machine; set it up once before you start.

Know

  • Check kubectl get nodes, then kubectl describe node conditions. On the node: systemctl status kubelet, journalctl -u kubelet, and the container runtime.
  • Common causes: kubelet stopped or misconfigured, expired certificates, CNI missing, disk or memory pressure.

Read

Demo: Take a node down and bring it back

  1. Stop the kubelet on a worker and watch the control plane notice.

    Terminal window
    docker exec kind-worker systemctl stop kubelet
    kubectl get nodes -w # wait for kind-worker to show NotReady (about 40s), then Ctrl-C
  2. Diagnose it the way you would on a real node.

    Terminal window
    kubectl describe node kind-worker | sed -n '/Conditions:/,/Addresses:/p'
    docker exec kind-worker systemctl status kubelet --no-pager | head -5
    docker exec kind-worker journalctl -u kubelet --no-pager -n 20
  3. Fix it, then drain and uncordon as you would for maintenance.

    Terminal window
    docker exec kind-worker systemctl start kubelet
    kubectl wait --for=condition=Ready node/kind-worker --timeout=120s
    kubectl drain kind-worker --ignore-daemonsets --delete-emptydir-data
    kubectl get node kind-worker
    kubectl uncordon kind-worker

What you should see: The node goes NotReady with conditions reading Kubelet stopped posting node status, the unreachable taint is added, and the journal shows the stop. After the drain the node shows SchedulingDisabled until you uncordon it.

Practise

  • Stop the kubelet on a worker, watch the node go NotReady, find the cause from journalctl, and fix it.

On this site

Know

  • On kubeadm clusters the control plane runs as static pods from /etc/kubernetes/manifests. A typo there stops the component; crictl ps -a and crictl logs show why.
  • Check API server health with kubectl get --raw=/readyz?verbose.

Read

Demo: Break the scheduler, then fix it

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace cka-components
    kubectl config set-context --current --namespace=cka-components
  2. Control plane components are static pods: the kubelet runs whatever is in this directory.

    Terminal window
    docker exec kind-control-plane ls /etc/kubernetes/manifests
    kubectl get pods -n kube-system -l tier=control-plane
  3. Back up the scheduler manifest and point it at a config file that doesn’t exist.

    Terminal window
    docker exec kind-control-plane cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/kube-scheduler.yaml.bak
    docker exec kind-control-plane sed -i 's|--kubeconfig=/etc/kubernetes/scheduler.conf|--kubeconfig=/etc/kubernetes/nope.conf|' /etc/kubernetes/manifests/kube-scheduler.yaml
    sleep 30; kubectl create deployment stuck --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 -- /agnhost pause
    kubectl get pods -n kube-system -l component=kube-scheduler; kubectl get pods
  4. Find the error where it is written, then restore.

    Terminal window
    docker exec kind-control-plane crictl ps -a --name kube-scheduler
    docker exec kind-control-plane sh -c 'crictl logs $(crictl ps -a --name kube-scheduler -q | head -1)' 2>&1 | tail -5
    docker exec kind-control-plane cp /root/kube-scheduler.yaml.bak /etc/kubernetes/manifests/kube-scheduler.yaml
    sleep 30; kubectl get pods

What you should see: The new pod stays Pending with no events at all, because nothing is scheduling. The scheduler container keeps exiting and its log names the missing file. When the API server itself is down, kubectl is useless, so crictl on the node is the tool to remember.

Note: The restore step matters: without it the lab cluster stays broken. If you lose the backup, recreate the cluster.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace cka-components

Practise

  • Break the kube-scheduler manifest, observe new pods stay Pending with no events, and repair it.

On this site

Monitor cluster and application resource usage

Section titled “Monitor cluster and application resource usage”

Know

  • kubectl top nodes and kubectl top pods need metrics-server. Sort with --sort-by=cpu or --sort-by=memory.

Read

Demo: Find the pod using the most

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace cka-usage
    kubectl config set-context --current --namespace=cka-usage
  2. Generate some load and read it with top.

    Terminal window
    kubectl create deployment burner --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sh -c 'while true; do :; done'
    kubectl set resources deployment burner --requests=cpu=100m,memory=64Mi --limits=cpu=500m,memory=256Mi
    sleep 60; kubectl top pods --containers
    kubectl top nodes
  3. Compare usage with what was requested on each node.

    Terminal window
    kubectl describe node kind-worker | sed -n '/Allocated resources:/,/Events:/p'

What you should see: top shows the burner near its 500m CPU limit (it is throttled, not killed). The node’s “Allocated resources” shows requests, which is what the scheduler counts; usage and requests are different numbers and both matter.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace cka-usage

Practise

  • Find the pod using the most memory across all namespaces in one command.

On this site

Manage and evaluate container output streams

Section titled “Manage and evaluate container output streams”

Know

  • kubectl logs reads stdout and stderr. Use -c for a container, --previous for the last crashed instance, -f to follow, --since to limit.
  • Logs live on the node under /var/log/pods and are rotated by the kubelet.

Read

Demo: Logs from many pods, and from a dead container

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace cka-logs
    kubectl config set-context --current --namespace=cka-logs
  2. Two containers in one pod, writing to stdout and stderr.

    Terminal window
    kubectl apply -f - <<'EOF'
    apiVersion: v1
    kind: Pod
    metadata:
    name: chatty
    labels:
    app: chatty
    spec:
    containers:
    - name: out
    image: registry.k8s.io/e2e-test-images/busybox:1.37.0-2
    command: ["sh", "-c", "while true; do echo out $(date +%T); sleep 2; done"]
    - name: err
    image: registry.k8s.io/e2e-test-images/busybox:1.37.0-2
    command: ["sh", "-c", "while true; do echo err $(date +%T) >&2; sleep 3; done"]
    EOF
    kubectl wait --for=condition=Ready pod/chatty --timeout=120s
    kubectl logs chatty -c err --tail=2
    kubectl logs -l app=chatty --all-containers --prefix --since=10s
  3. Logs of a container that already crashed.

    Terminal window
    kubectl run crasher --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sh -c 'echo "about to fail"; exit 3'
    sleep 20; kubectl logs crasher --previous
  4. Where the kubelet keeps those logs on the node.

    Terminal window
    NODE=$(kubectl get pod chatty -o jsonpath='{.spec.nodeName}')
    docker exec $NODE sh -c 'ls /var/log/pods/ | grep cka-logs'

What you should see: --prefix labels each line with pod and container. --previous shows the crashed run’s output; without it you would see the current, possibly empty, run. On the node, logs live under /var/log/pods/, which is what log shippers read.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace cka-logs

Practise

  • Get the logs of the previous instance of a crashing init container.

On this site

Know

  • Work hop by hop: DNS name, Service selector and EndpointSlices, kube-proxy or CNI forwarding, then the container port.
  • A connection refused is usually the port; a timeout is usually a policy or routing drop.

Read

Demo: A Service that routes nowhere

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace cka-svc
    kubectl config set-context --current --namespace=cka-svc
  2. Create a Service whose selector doesn’t match the pods.

    Terminal window
    kubectl create deployment api --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 -- /agnhost netexec --http-port=8080
    kubectl create service clusterip api --tcp=80:8080
    kubectl patch service api -p '{"spec":{"selector":{"app":"apii"}}}'
    kubectl run client --rm -i --restart=Never --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- wget -qO- -T 3 http://api/hostname
  3. Work through the chain: Service, EndpointSlice, pod labels.

    Terminal window
    kubectl get endpointslices -l kubernetes.io/service-name=api
    kubectl get service api -o jsonpath='{.spec.selector}{"\n"}'
    kubectl get pods --show-labels
  4. Fix the selector and check DNS resolution while you are there.

    Terminal window
    kubectl patch service api -p '{"spec":{"selector":{"app":"api"}}}'
    kubectl run client --rm -i --restart=Never --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sh -c 'nslookup api; wget -qO- -T 3 http://api/hostname'

What you should see: The first call times out and the EndpointSlice has no endpoints. The labels show app=api while the selector says apii. After the fix the name resolves to the ClusterIP and the call returns a pod name. Empty endpoints is the first thing to check for any “Service doesn’t work” problem.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace cka-svc

Practise

  • Break a Service selector and fix it using only EndpointSlices and labels.

On this site


Domain and competency names are from the CNCF CKA curriculum (v1.37), cncf/curriculum, licensed CC BY 4.0. Summaries, demos, and exercises are original to this site.