Skip to content

etcd: database space exceeded

etcd reached its backend storage quota, raised a NOSPACE alarm, and now rejects writes. Reads still work, so the cluster looks alive but nothing can change.

On managed Kubernetes and on OpenShift, don’t run the steps below yourself: open a support case or follow the platform’s documented procedure.

Run etcdctl against your etcd endpoints with the cluster’s client certificates (on kubeadm they live under /etc/kubernetes/pki/etcd/):

Terminal window
export ETCDCTL_API=3
etcdctl --endpoints=<endpoints> --cacert=<ca> --cert=<cert> --key=<key> endpoint status --write-out=table
etcdctl --endpoints=<endpoints> --cacert=<ca> --cert=<cert> --key=<key> alarm list

DB SIZE near the quota and alarm:NOSPACE confirm it.

  1. Snapshot first: etcdctl ... snapshot save /backup/etcd-$(date +%F-%H%M).db.
  2. Compact old revisions up to the current one (get the revision from endpoint status --write-out=json): etcdctl ... compact <revision>.
  3. Defragment each member, one at a time, never all at once: etcdctl --endpoints=<one-member> ... defrag. Defragmentation blocks that member while it runs.
  4. Disarm the alarm: etcdctl ... alarm disarm.
  5. Find out what filled it. Common causes: a controller writing an object in a loop, huge numbers of Events, or very large ConfigMaps or Secrets.

Alert on database size against the quota, make sure the API server’s automatic compaction is running, and defragment on a schedule during quiet hours. Raise the quota (8 GiB is a common ceiling) only alongside the monitoring.

Read next: Control plane failures.