Broken in Prod
Learn Kubernetes from the way it fails. Every chapter shows the real output, the root cause, and how it’s run in regulated production.
Common starting points
CrashLoopBackOffYour container keeps exiting and Kubernetes keeps restarting it with a growing delayImagePullBackOffThe node can’t pull the container imageInvalidImageNameThe image reference in the pod spec isn’t a valid reference, so the kubelet won’t even try to pull itCreateContainerConfigErrorA ConfigMap, Secret, or key the container needs doesn’t exist in the pod’s namespace, so the container can’t be configuredCreateContainerErrorThe runtime accepted the pod but failed to create the container: a bad command, a mount problem, or a name conflictInit:CrashLoopBackOffAn init container keeps failing, so the app containers never startOOMKilledThe container exceeded its memory limit and the kernel killed it, exit code 137Liveness probe failedThe kubelet is restarting a container because its liveness probe keeps failingReadiness probe failedThe pod is running but not Ready, so Services send it no trafficStuck in ContainerCreatingThe pod is scheduled, but the node can’t finish setting it up: sandbox, network, volumes, or imageStuck in TerminatingA pod was deleted but never goes awayFailedSchedulingThe pod is Pending because no node passed the scheduler’s filtersEvictedThe kubelet removed the pod to protect a node that ran short of memory, disk, or process IDsNode NotReadyA node stopped reporting healthyCannot evict pod: disruption budgetA drain or eviction is blocked by a PodDisruptionBudgetPVC stuck in PendingA PersistentVolumeClaim isn’t bindingFailedMountThe kubelet can’t mount a volume into the podMulti-Attach errorA ReadWriteOnce disk is still attached to another node, so the new pod can’t startnode(s) didn't match PersistentVolume's node affinityThe pod’s volume is pinned to a zone or node where nothing can run the podNo endpoints available for serviceA Service has no Ready backendsDNS resolution failure in podsPods can’t resolve names, or resolution is slowIngress returns 502 or 503The edge proxy answers, but with Bad Gateway or Service UnavailableForbidden: cannot list resourceAn RBAC denialx509: certificate signed by unknown authorityThe client doesn’t trust the certificate it was shownThe connection to the server localhost:8080 was refusedkubectl has no cluster configured, so it falls back to localhost:8080Forbidden: violates PodSecurityPod Security Admission rejected the podForbidden: exceeded quotaA ResourceQuota blocked the podFailed calling webhookAn admission webhook is unreachable or slow, so the API server rejects requestsProgressDeadlineExceededA Deployment’s rollout stopped making progressField is immutableAn apply tried to change a field Kubernetes doesn’t allow to change, such as a Deployment selector or a Job templateStrict decoding error: unknown fieldThe manifest contains a field the API server doesn’t recognise: a typo, wrong indentation, or a field from a newer versionNo matches for kind in versionThe cluster doesn’t serve this kind at this API version: a removed API after an upgrade, or a CRD that isn’t installedFailedGetResourceMetricThe HorizontalPodAutoscaler can’t read CPU or memory metrics, so it won’t scaleetcd: database space exceededetcd hit its storage quota and stopped accepting writesx509: certificate has expired or is not yet validA cluster certificate expired, or a clock is wrong
No page for that yet. Browse all 35 Kubernetes errors.
Learn: Kubernetes
A study path through all five CNCF Kubernetes exams, built on the official curricula and linked to the official docs.
- 1. How a pod gets from YAML to running: draft
- 2. Probes and resource limits: draft
- 3. Services, DNS, and kube-proxy: draft
- 4. ConfigMaps and Secrets: draft
- 5. Storage and CSI: draft
- 6. RBAC and service accounts: draft
- 7. Ingress and Gateway API: draft
- 8. Network policies: draft
- 9. Deployments and rollouts: draft
- 10. Node pressure and eviction: draft
- 11. Control plane failures: draft
- 12. Upgrades: draft
Dashed: drafted, not yet run on a live cluster. Solid: verified, with the versions it was tested on.
Planned tracksLinuxDocker and containersOpenShiftGitLab CI
Start the certification pathEvery chapter, seven parts
- 1 Mental model
- 2 Under the hood
- 3 Minimal example
- 4 How it breaks
- 5 Production call
- 6 Regulated production
- 7 Self-check
For regulated production
Each chapter ends with what changes when an assessor is watching.
35 real Kubernetes errors
Every message is linked to the source line that produces it, checked against Kubernetes v1.37.2. Linux, Docker and containers, OpenShift, GitLab CI errors are planned.
Browse errors by technologyContributing
PlannedNot open yet. Later you’ll be able to write up errors you’ve debugged, reviewed before they’re published.
What’s planned