Skip to content

Probes and resource limits

Draft: not yet run against a live cluster. Treat output as representative and check it on yours.

Most “random restarts” in production are not random. They are the kubelet doing exactly what the pod spec told it to: kill a container that crossed its memory limit, throttle one that hit its CPU limit, or restart one whose liveness probe failed. This chapter is about reading the spec the way the kubelet does.

  1. The kubelet turns requests and limits into cgroup settings. The CPU request becomes a CPU weight, the CPU limit becomes a CFS quota (CPU time allowed per 100ms period), and the memory limit becomes the cgroup memory maximum.

  2. The pod gets a QoS class from those numbers. Guaranteed if every container has CPU and memory requests equal to its limits. BestEffort if no container has any requests or limits. Burstable for everything in between. Check it with kubectl get pod <pod> -o jsonpath='{.status.qosClass}'.

  3. The kernel enforces the limits. A container that uses its whole CPU quota early in a period waits for the next period. A container that allocates past its memory limit triggers the kernel OOM killer inside its cgroup, and the kubelet reports OOMKilled.

  4. The kubelet runs the probes. Each probe runs every periodSeconds (default 10) and fails if it doesn’t answer within timeoutSeconds (default 1). It takes failureThreshold consecutive failures (default 3) to act. Probes can be httpGet, tcpSocket, exec, or grpc.

  5. Startup gates the others. While a startup probe is defined and hasn’t succeeded, liveness and readiness probes don’t run. That’s what makes slow starters safe.

limits.yaml
apiVersion: v1
kind: Pod
metadata:
name: hungry
spec:
containers:
- name: hog
image: polinux/stress:1.0.4
command: ['stress']
args: ['--vm', '1', '--vm-bytes', '200M', '--vm-hang', '1']
resources:
requests: { cpu: 100m, memory: 100Mi }
limits: { cpu: 200m, memory: 100Mi }
Terminal window
kubectl apply -f limits.yaml
kubectl get pod hungry -w
kubectl get pod hungry -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'

The container asks for 200M with a 100Mi limit, so it’s OOM-killed on every start and ends up in CrashLoopBackOff. The last line shows reason: OOMKilled and exitCode: 137. The image is a third-party stress tool used only for this demonstration; replace it with your own if your policy requires approved images.

A container uses more CPU than its limit. What happens?

It’s throttled: it waits until the next CFS period to run again. It’s never killed for CPU.

What QoS class does a pod get if its only container has requests but no limits?

Burstable. Guaranteed needs requests equal to limits for both CPU and memory in every container.

Why shouldn't a liveness probe call the database?

Because a database outage would then fail liveness on every replica, and the kubelet would restart all of them in a loop. Liveness should only detect a stuck process.

What does a failing readiness probe do that a failing liveness probe doesn't, and vice versa?

Readiness removes the pod from Service endpoints without restarting it. Liveness restarts the container but doesn’t, by itself, remove it from endpoints.

Maps to: CKA: Workloads and Scheduling; CKAD: Application Observability and Maintenance. Check the current exam curriculum, since domain names change between revisions.