Skip to content

Node pressure and eviction

Draft: not yet run against a live cluster. Treat output as representative and check it on yours.

Pods don’t only die from their own mistakes. When a node runs short of memory, disk, or process IDs, the kubelet starts evicting pods to save the node, and the pods it picks are often not the ones causing the problem.

  1. The kubelet watches eviction signals: memory.available, nodefs.available, nodefs.inodesFree, imagefs.available, and pid.available. On Linux, the default hard thresholds include memory.available<100Mi, nodefs.available<10%, imagefs.available<15%, and nodefs.inodesFree<5%. Your distribution or managed platform may set different values.

  2. Crossing a threshold sets a node condition and a taint. MemoryPressure, DiskPressure, or PIDPressure becomes True, and a matching taint (for example node.kubernetes.io/memory-pressure) stops new pods from being scheduled there.

  3. For disk pressure, the kubelet first reclaims by deleting unused images and dead containers. Only then does it evict pods.

  4. It ranks pods for eviction. First pods whose usage of the starved resource exceeds their requests, then by pod priority, then by how far over their requests they are. BestEffort pods (no requests) are always over, so they go first. Guaranteed pods within their requests go last.

  5. The evicted pod stays as a record in phase Failed with reason Evicted. Its Deployment or other controller creates a replacement.

Watch the signals and conditions on a node:

Terminal window
kubectl describe node <node> | sed -n '/Conditions:/,/Addresses:/p'
kubectl get --raw "/api/v1/nodes/<node>/proxy/stats/summary" | head -40

Make a pod fill its node’s ephemeral storage, with a limit so only it is evicted:

disk-hog.yaml
apiVersion: v1
kind: Pod
metadata:
name: disk-hog
spec:
containers:
- name: hog
image: busybox:1.36
command: ['sh', '-c', 'dd if=/dev/zero of=/tmp/fill bs=1M count=600; sleep 3600']
resources:
limits: { ephemeral-storage: 500Mi }

The kubelet evicts it once it writes past 500Mi. kubectl get pod disk-hog shows Evicted, and kubectl describe says the container was using more ephemeral storage than its limit.

A container shows OOMKilled with exit code 137. Was it evicted?

No. That’s the kernel enforcing the container’s own memory limit. Eviction shows on the pod as reason Evicted.

Under memory pressure, which goes first: a BestEffort pod or a Guaranteed pod using less than its requests?

The BestEffort pod. It has no requests, so any usage exceeds them.

What does the kubelet do before evicting pods for disk pressure?

It reclaims node-level resources: deleting unused container images and dead containers.

Maps to: CKA: Troubleshooting; CKA: Workloads and Scheduling. Check the current exam curriculum, since domain names change between revisions.