etcd: database space exceeded
etcd reached its backend storage quota, raised a NOSPACE alarm, and now rejects writes. Reads still work, so the cluster looks alive but nothing can change.
On managed Kubernetes and on OpenShift, don’t run the steps below yourself: open a support case or follow the platform’s documented procedure.
Diagnose
Section titled “Diagnose”Run etcdctl against your etcd endpoints with the cluster’s client certificates (on kubeadm they live under /etc/kubernetes/pki/etcd/):
export ETCDCTL_API=3etcdctl --endpoints=<endpoints> --cacert=<ca> --cert=<cert> --key=<key> endpoint status --write-out=tableetcdctl --endpoints=<endpoints> --cacert=<ca> --cert=<cert> --key=<key> alarm listDB SIZE near the quota and alarm:NOSPACE confirm it.
Recover, in this order
Section titled “Recover, in this order”- Snapshot first:
etcdctl ... snapshot save /backup/etcd-$(date +%F-%H%M).db. - Compact old revisions up to the current one (get the revision from
endpoint status --write-out=json):etcdctl ... compact <revision>. - Defragment each member, one at a time, never all at once:
etcdctl --endpoints=<one-member> ... defrag. Defragmentation blocks that member while it runs. - Disarm the alarm:
etcdctl ... alarm disarm. - Find out what filled it. Common causes: a controller writing an object in a loop, huge numbers of Events, or very large ConfigMaps or Secrets.
Prevent
Section titled “Prevent”Alert on database size against the quota, make sure the API server’s automatic compaction is running, and defragment on a schedule during quiet hours. Raise the quota (8 GiB is a common ceiling) only alongside the monitoring.
Read next: Control plane failures.