ENGINEERING THROUGH FAILURE
Brokenin Prod
Learn Kubernetes by understanding what breaks, why it breaks, and how engineers resolve failures in real production environments.
- 35 real Kubernetes errors
- 5 certification guides
- Free, no ads, no sign-up
$ kubectl get pods -n payments
NAME READY STATUS RESTARTS AGE
api-7d9f6c8b5-x2kqp 0/1 CrashLoopBackOff 5 (40s ago) 6m
$ kubectl describe pod api-7d9f6c8b5-x2kqp -n payments | grep -A2 "Last State"
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
# root cause: the container hit its 256Mi memory limit
$ kubectl set resources deployment/api -n payments --limits=memory=512Mi
deployment.apps/api resource requirements updated
$ kubectl rollout status deployment/api -n payments
deployment "api" successfully rolled out
Troubleshoot an error
Paste the message, event, or status you’re looking at. Every page leads with what you see, then the cause and the fix, in order.
Common starting points
CrashLoopBackOffYour container keeps exiting and Kubernetes keeps restarting it with a growing delayImagePullBackOffThe node can’t pull the container imageInvalidImageNameThe image reference in the pod spec isn’t a valid reference, so the kubelet won’t even try to pull itCreateContainerConfigErrorA ConfigMap, Secret, or key the container needs doesn’t exist in the pod’s namespace, so the container can’t be configuredCreateContainerErrorThe runtime accepted the pod but failed to create the container: a bad command, a mount problem, or a name conflictInit:CrashLoopBackOffAn init container keeps failing, so the app containers never startOOMKilledThe container exceeded its memory limit and the kernel killed it, exit code 137Liveness probe failedThe kubelet is restarting a container because its liveness probe keeps failingReadiness probe failedThe pod is running but not Ready, so Services send it no trafficStuck in ContainerCreatingThe pod is scheduled, but the node can’t finish setting it up: sandbox, network, volumes, or imageStuck in TerminatingA pod was deleted but never goes awayFailedSchedulingThe pod is Pending because no node passed the scheduler’s filtersEvictedThe kubelet removed the pod to protect a node that ran short of memory, disk, or process IDsNode NotReadyA node stopped reporting healthyCannot evict pod: disruption budgetA drain or eviction is blocked by a PodDisruptionBudgetPVC stuck in PendingA PersistentVolumeClaim isn’t bindingFailedMountThe kubelet can’t mount a volume into the podMulti-Attach errorA ReadWriteOnce disk is still attached to another node, so the new pod can’t startnode(s) didn't match PersistentVolume's node affinityThe pod’s volume is pinned to a zone or node where nothing can run the podNo endpoints available for serviceA Service has no Ready backendsDNS resolution failure in podsPods can’t resolve names, or resolution is slowIngress returns 502 or 503The edge proxy answers, but with Bad Gateway or Service UnavailableForbidden: cannot list resourceAn RBAC denialx509: certificate signed by unknown authorityThe client doesn’t trust the certificate it was shownThe connection to the server localhost:8080 was refusedkubectl has no cluster configured, so it falls back to localhost:8080Forbidden: violates PodSecurityPod Security Admission rejected the podForbidden: exceeded quotaA ResourceQuota blocked the podFailed calling webhookAn admission webhook is unreachable or slow, so the API server rejects requestsProgressDeadlineExceededA Deployment’s rollout stopped making progressField is immutableAn apply tried to change a field Kubernetes doesn’t allow to change, such as a Deployment selector or a Job templateStrict decoding error: unknown fieldThe manifest contains a field the API server doesn’t recognise: a typo, wrong indentation, or a field from a newer versionNo matches for kind in versionThe cluster doesn’t serve this kind at this API version: a removed API after an upgrade, or a CRD that isn’t installedFailedGetResourceMetricThe HorizontalPodAutoscaler can’t read CPU or memory metrics, so it won’t scaleetcd: database space exceededetcd hit its storage quota and stopped accepting writesx509: certificate has expired or is not yet validA cluster certificate expired, or a clock is wrong
No page matches that yet.
Try pasting just the reason, for example or . You can also browse all 35 errors, or email the message so it can be added.
Learn
Kubernetes learning paths
A study path through all five CNCF Kubernetes certifications, built on the official curricula and linked to the official documentation. Study guides are available; their hands-on demos haven’t been run end to end yet, and the cards say so.
- KCNAAssociateKubernetes and Cloud Native AssociateKubernetes and cloud native fundamentals: the vocabulary everything else builds on.13 topics12 demosMultiple choiceGuide availableDemos not yet run
- CKAProfessionalCertified Kubernetes AdministratorInstall, run, and troubleshoot clusters: storage, networking, workloads, and the control plane.27 topics27 demosHands-on examGuide availableDemos not yet run
- CKADProfessionalCertified Kubernetes Application DeveloperDesign, build, and debug applications on Kubernetes as a developer.24 topics24 demosHands-on examGuide availableDemos not yet run
- KCSAAssociateKubernetes and Cloud Native Security AssociateThe security model: threat modelling, the 4Cs, and platform controls.42 topics34 demosMultiple choiceGuide availableDemos not yet run
- CKSSpecialistCertified Kubernetes Security SpecialistHarden clusters, the supply chain, and runtime, under exam time pressure.26 topics26 demosHands-on exam, needs CKAGuide availableDemos not yet run
Deep dives: how each part breaks
12 chapters that break Kubernetes on purpose, one subsystem at a time.
0 of 12 verified on a cluster
- Pod lifecycle and scheduling(draft)
- Probes and resource limits(draft)
- Services, DNS, and kube-proxy(draft)
- ConfigMaps and Secrets(draft)
- Storage and CSI(draft)
- RBAC and service accounts(draft)
- Ingress and Gateway API(draft)
- Network policies(draft)
- Deployments and rollouts(draft)
- Node pressure and eviction(draft)
- Control plane failures(draft)
- Upgrades(draft)
Draft: written, not yet run on a live cluster. Verified: run, with the versions listed on the page.
Method
How Broken in Prod works
Every deep dive follows the same seven parts, so you always know where the failure, the decision, and the audit view are.
- 01
Mental model
What the thing is for and where it sits, in one diagram.
- 02
Under the hood
What each component actually does, step by step.
- 03
Minimal example
Copy-paste commands that run on a laptop with kind.
- 04
How it breaks
Real failure output, the root cause, and the diagnosis in order.
- 05
Production call
An opinionated default, and what we would never do.
- 06
Regulated production
What changes under PCI DSS, SAMA CSF, or ISO 27001.
- 07
Self-check
Five questions, mapped to certification objectives.
Regulated production
What changes when an assessor is watching
Each deep dive ends with the operational evidence, security controls, and production decisions that matter in banking and payments. It shows engineering practice, not legal advice.
PCI DSS v4.0
Systems that store, process, or transmit payment card data.
Evidence assessors commonly ask for
- Network security controls around the cardholder data environment (Requirement 1)
- Access limited by business need (Requirement 7)
- Logged and reviewed access (Requirement 10)
Where the chapters help
NetworkPolicy as segmentation evidence, RBAC reviews, and audit log retention.
SAMA CSF
The Saudi Central Bank’s framework for its regulated member organizations.
Evidence assessors commonly ask for
- Change management (3.3.7)
- Identity and access management (3.3.5)
- Cyber security event management (3.3.14)
Where the chapters help
Rollouts with approvals and rollback, least-privilege service accounts, and events a SOC can use.
ISO/IEC 27001:2022
An information security management system, audited against Annex A controls.
Evidence assessors commonly ask for
- Configuration management (A.8.9)
- Logging and monitoring (A.8.15, A.8.16)
- Change management (A.8.32)
Where the chapters help
Declarative configuration in Git, drift you can show, and logs that answer who changed what.
Broken in Prod isn’t certified against, audited for, or endorsed by any of these frameworks. Control references point to where a topic usually comes up; your assessor and your own control mapping decide what applies.
Error library
Real errors, real messages
33 of 35 messages are checked against the Kubernetes v1.37.2 source line that prints them, with a link to that line on every page.
CrashLoopBackOffYour container keeps exiting and Kubernetes keeps restarting it with a growing delayNAME READY STATUS RESTARTS AGEChecked against Kubernetes v1.37.2 sourceOOMKilledThe container exceeded its memory limit and the kernel killed it, exit code 137Last State: TerminatedChecked against Kubernetes v1.37.2 sourceFailedSchedulingThe pod is Pending because no node passed the scheduler’s filtersWarning FailedScheduling default-scheduler 0/6 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient memory, 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/6 nodes are available: 3 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod.Checked against Kubernetes v1.37.2 sourceMulti-Attach errorA ReadWriteOnce disk is still attached to another node, so the new pod can’t startWarning FailedAttachVolume attachdetach-controller Waiting for detach for volume "pvc-3f2a…" Volume is already used by pod(s) db-0Checked against Kubernetes v1.37.2 sourceNo endpoints available for serviceA Service has no Ready backendserror: unable to forward port because pod is not running. Current status=PendingChecked against Kubernetes v1.37.2 sourceForbidden: cannot list resourceAn RBAC denialError from server (Forbidden): deployments.apps is forbidden: User "jane@example.com" cannot patch resource "deployments" in API group "apps" in the namespace "shop"Checked against Kubernetes v1.37.2 sourcex509: certificate signed by unknown authorityThe client doesn’t trust the certificate it was shownUnable to connect to the server: x509: certificate signed by unknown authorityChecked against Go go1.27.4 sourceNo matches for kind in versionThe cluster doesn’t serve this kind at this API version: a removed API after an upgrade, or a CRD that isn’t installederror: resource mapping not found for name: "web" namespace: "shop" from "web.yaml": no matches for kind "Ingress" in version "extensions/v1beta1"Checked against Kubernetes v1.37.2 sourceetcd: database space exceededetcd hit its storage quota and stopped accepting writesError from server: etcdserver: mvcc: database space exceededChecked against etcd v3.6.16 source
Roadmap
Planned, not released
Kubernetes is the only open track. These are next, in no fixed order and with no dates promised.
- Planned
Linux
Processes, systemd, filesystems, permissions, and networking on the hosts underneath everything.
- Planned
Docker and containers
Images, layers, namespaces, cgroups, and why distroless images break your shell.
- Planned
OpenShift
Operators, Security Context Constraints, Routes, upgrades, and running it under PCI DSS.
- Planned
GitLab CI
Runners, pipelines, caching, artifacts, and secure delivery to clusters.
- Planned
Community contributions
Not open yet. Later you’ll be able to write up errors you’ve debugged, reviewed before publishing.
What’s planned