ENGINEERING THROUGH FAILURE

Brokenin Prod

Learn Kubernetes by understanding what breaks, why it breaks, and how engineers resolve failures in real production environments.

  • 35 real Kubernetes errors
  • 5 certification guides
  • Free, no ads, no sign-up
~/incidents/payments-apiDiagnosing
$ kubectl get pods -n payments
NAME                  READY   STATUS             RESTARTS      AGE
api-7d9f6c8b5-x2kqp   0/1     CrashLoopBackOff   5 (40s ago)   6m
$ kubectl describe pod api-7d9f6c8b5-x2kqp -n payments | grep -A2 "Last State"
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
# root cause: the container hit its 256Mi memory limit
$ kubectl set resources deployment/api -n payments --limits=memory=512Mi
deployment.apps/api resource requirements updated
$ kubectl rollout status deployment/api -n payments
deployment "api" successfully rolled out
Symptom, cause, fix: the order every error page follows. Read the OOMKilled walkthrough

Troubleshoot an error

Paste the message, event, or status you’re looking at. Every page leads with what you see, then the cause and the fix, in order.

Try
Category

Common starting points

  1. CrashLoopBackOffYour container keeps exiting and Kubernetes keeps restarting it with a growing delayPods and containers
  2. ImagePullBackOffThe node can’t pull the container imagePods and containers
  3. OOMKilledThe container exceeded its memory limit and the kernel killed it, exit code 137Pods and containers
  4. FailedSchedulingThe pod is Pending because no node passed the scheduler’s filtersScheduling and nodes
  5. Multi-Attach errorA ReadWriteOnce disk is still attached to another node, so the new pod can’t startStorage
  6. Forbidden: cannot list resourceAn RBAC denialAccess and admission

Browse the full error directory

Learn

Kubernetes learning paths

A study path through all five CNCF Kubernetes certifications, built on the official curricula and linked to the official documentation. Study guides are available; their hands-on demos haven’t been run end to end yet, and the cards say so.

  1. KCNAAssociateKubernetes and Cloud Native AssociateKubernetes and cloud native fundamentals: the vocabulary everything else builds on.13 topics12 demosMultiple choiceGuide availableDemos not yet run
  2. CKAProfessionalCertified Kubernetes AdministratorInstall, run, and troubleshoot clusters: storage, networking, workloads, and the control plane.27 topics27 demosHands-on examGuide availableDemos not yet run
  3. CKADProfessionalCertified Kubernetes Application DeveloperDesign, build, and debug applications on Kubernetes as a developer.24 topics24 demosHands-on examGuide availableDemos not yet run
  4. KCSAAssociateKubernetes and Cloud Native Security AssociateThe security model: threat modelling, the 4Cs, and platform controls.42 topics34 demosMultiple choiceGuide availableDemos not yet run
  5. CKSSpecialistCertified Kubernetes Security SpecialistHarden clusters, the supply chain, and runtime, under exam time pressure.26 topics26 demosHands-on exam, needs CKAGuide availableDemos not yet run

Method

How Broken in Prod works

Every deep dive follows the same seven parts, so you always know where the failure, the decision, and the audit view are.

  1. 01

    Mental model

    What the thing is for and where it sits, in one diagram.

  2. 02

    Under the hood

    What each component actually does, step by step.

  3. 03

    Minimal example

    Copy-paste commands that run on a laptop with kind.

  4. 04

    How it breaks

    Real failure output, the root cause, and the diagnosis in order.

  5. 05

    Production call

    An opinionated default, and what we would never do.

  6. 06

    Regulated production

    What changes under PCI DSS, SAMA CSF, or ISO 27001.

  7. 07

    Self-check

    Five questions, mapped to certification objectives.

Regulated production

What changes when an assessor is watching

Each deep dive ends with the operational evidence, security controls, and production decisions that matter in banking and payments. It shows engineering practice, not legal advice.

PCI DSS v4.0

Systems that store, process, or transmit payment card data.

Evidence assessors commonly ask for

  • Network security controls around the cardholder data environment (Requirement 1)
  • Access limited by business need (Requirement 7)
  • Logged and reviewed access (Requirement 10)

Where the chapters help

NetworkPolicy as segmentation evidence, RBAC reviews, and audit log retention.

SAMA CSF

The Saudi Central Bank’s framework for its regulated member organizations.

Evidence assessors commonly ask for

  • Change management (3.3.7)
  • Identity and access management (3.3.5)
  • Cyber security event management (3.3.14)

Where the chapters help

Rollouts with approvals and rollback, least-privilege service accounts, and events a SOC can use.

ISO/IEC 27001:2022

An information security management system, audited against Annex A controls.

Evidence assessors commonly ask for

  • Configuration management (A.8.9)
  • Logging and monitoring (A.8.15, A.8.16)
  • Change management (A.8.32)

Where the chapters help

Declarative configuration in Git, drift you can show, and logs that answer who changed what.

Broken in Prod isn’t certified against, audited for, or endorsed by any of these frameworks. Control references point to where a topic usually comes up; your assessor and your own control mapping decide what applies.

Error library

Real errors, real messages

33 of 35 messages are checked against the Kubernetes v1.37.2 source line that prints them, with a link to that line on every page.

All 35 errors

Roadmap

Planned, not released

Kubernetes is the only open track. These are next, in no fixed order and with no dates promised.