Node pressure and eviction
Draft: not yet run against a live cluster. Treat output as representative and check it on yours.
Pods don’t only die from their own mistakes. When a node runs short of memory, disk, or process IDs, the kubelet starts evicting pods to save the node, and the pods it picks are often not the ones causing the problem.
Under the hood
Section titled “Under the hood”-
The kubelet watches eviction signals:
memory.available,nodefs.available,nodefs.inodesFree,imagefs.available, andpid.available. On Linux, the default hard thresholds includememory.available<100Mi,nodefs.available<10%,imagefs.available<15%, andnodefs.inodesFree<5%. Your distribution or managed platform may set different values. -
Crossing a threshold sets a node condition and a taint.
MemoryPressure,DiskPressure, orPIDPressurebecomesTrue, and a matching taint (for examplenode.kubernetes.io/memory-pressure) stops new pods from being scheduled there. -
For disk pressure, the kubelet first reclaims by deleting unused images and dead containers. Only then does it evict pods.
-
It ranks pods for eviction. First pods whose usage of the starved resource exceeds their requests, then by pod priority, then by how far over their requests they are.
BestEffortpods (no requests) are always over, so they go first.Guaranteedpods within their requests go last. -
The evicted pod stays as a record in phase
Failedwith reasonEvicted. Its Deployment or other controller creates a replacement.
Minimal example
Section titled “Minimal example”Watch the signals and conditions on a node:
kubectl describe node <node> | sed -n '/Conditions:/,/Addresses:/p'kubectl get --raw "/api/v1/nodes/<node>/proxy/stats/summary" | head -40Make a pod fill its node’s ephemeral storage, with a limit so only it is evicted:
apiVersion: v1kind: Podmetadata: name: disk-hogspec: containers: - name: hog image: busybox:1.36 command: ['sh', '-c', 'dd if=/dev/zero of=/tmp/fill bs=1M count=600; sleep 3600'] resources: limits: { ephemeral-storage: 500Mi }The kubelet evicts it once it writes past 500Mi. kubectl get pod disk-hog shows Evicted, and kubectl describe says the container was using more ephemeral storage than its limit.
How it breaks
Section titled “How it breaks”Self-check
Section titled “Self-check”A container shows OOMKilled with exit code 137. Was it evicted?
No. That’s the kernel enforcing the container’s own memory limit. Eviction shows on the pod as reason Evicted.
Under memory pressure, which goes first: a BestEffort pod or a Guaranteed pod using less than its requests?
The BestEffort pod. It has no requests, so any usage exceeds them.
What does the kubelet do before evicting pods for disk pressure?
It reclaims node-level resources: deleting unused container images and dead containers.
Maps to: CKA: Troubleshooting; CKA: Workloads and Scheduling. Check the current exam curriculum, since domain names change between revisions.