How to Back Up and Restore etcd Before You Need It

Every Deployment, Secret, ConfigMap, ServiceAccount and custom resource in a self-managed cluster lives in etcd. Lose it without a snapshot and you have a fleet of healthy nodes with no idea what they are supposed to run. Keep one, and a dead control plane is twenty minutes of work.

Most teams already have a snapshot cron job. Far fewer have ever fed one of those files back into an etcd process and watched a cluster come up, and that second half is where the surprises are.

What a snapshot covers and what it does not

An etcd snapshot is the entire Kubernetes keyspace at a moment in time. It is not your application data. PersistentVolume contents live on the storage backend and need their own backups. The cluster PKI under /etc/kubernetes/pki sits on the control plane node's disk, not in etcd, and a restore without matching certificates gives you a cluster nobody can authenticate to.

Treat the snapshot and /etc/kubernetes as one unit. Back them up together, restore them together.

Taking a snapshot worth having

The snapshot save subcommand talks to one server. The Kubernetes documentation is explicit that it only operates on the etcd server endpoint given by the client URL, so aiming it at a load balancer in front of three members gets you a snapshot of whichever member answered.

On a kubeadm control plane node the certificates are in /etc/kubernetes/pki/etcd, so the command is:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd-$(date +%Y%m%d-%H%M%S).db

ETCDCTL_API=3 has been the default since etcd 3.4, and setting it anyway costs nothing on an older binary.

Then move the file off the node. A snapshot that lives on the same disk as the data directory it protects is not a backup, it is a second copy of the same failure.

Verify the snapshot rather than storing it

Two subcommands changed homes, and this is the detail that breaks copied runbooks. In etcd 3.6, etcdctl snapshot status and etcdctl snapshot restore were both removed. They live in etcdutl now, the utility that operates on data files rather than over the network.

etcdutl snapshot status /var/backups/etcd-20260825-030000.db -w table

That prints the hash, revision, total keys and total size. The last two are worth alerting on. A key count that is a fraction of yesterday's, or a database size that halves overnight, means something changed that nobody intended.

Integrity is the other half. A file produced by etcdctl snapshot save carries an integrity hash which etcdutl snapshot restore checks. A file copied straight out of a data directory has no hash and will only restore with --skip-hash-check, which is a solid reason to prefer the snapshot command over cp.

The restore

Restoring is not putting the file back where it came from. etcdutl snapshot restore builds a new data directory and refuses to run if the target already exists, which is deliberate protection against overwriting live data.

It also rewrites cluster identity. The etcd documentation notes that the restore overwrites the member ID and cluster ID, so the member loses its former identity, and the Kubernetes documentation adds that the restored cluster comes up with a new cluster token and cluster ID. The API servers have to be pointed at the restored cluster before they start.

For a single member control plane, the sequence below is destructive. Moving the static Pod manifests out of /etc/kubernetes/manifests stops the API server and etcd on that node, which is exactly the intent, and nothing schedules or reports until they are back.

sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/
sudo mv /etc/kubernetes/manifests/etcd.yaml /root/

sudo etcdutl snapshot restore /var/backups/etcd-20260825-030000.db \
  --name master-1 \
  --initial-cluster master-1=https://10.0.0.11:2380 \
  --initial-cluster-token etcd-restore-20260825 \
  --initial-advertise-peer-urls https://10.0.0.11:2380 \
  --data-dir /var/lib/etcd-restore

sudo mv /var/lib/etcd /var/lib/etcd-old
sudo mv /var/lib/etcd-restore /var/lib/etcd
sudo mv /root/etcd.yaml /etc/kubernetes/manifests/
sudo mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/

Keep /var/lib/etcd-old until the cluster has been healthy for a day.

With three members, run the same restore on each node using that node's own --name and --initial-advertise-peer-urls, listing all three peers in --initial-cluster and using an identical token everywhere. All members must be restored from the same snapshot. Mixing files gives you a cluster that disagrees with itself about its own history.

Then confirm before you celebrate:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --write-out=table endpoint status

kubectl get nodes
kubectl get pods -A

The alarm that reads like an outage

etcd sets a space quota on its backend database. When a member goes over it, etcd raises a cluster wide alarm and the cluster accepts only reads and deletes, which looks from the outside like a control plane that has stopped taking writes for no reason.

etcdctl endpoint health --cluster
etcdctl alarm list
etcdctl defrag --cluster
etcdctl alarm disarm

Defragmentation on a live member blocks reads and writes while it rebuilds state, so work through members one at a time rather than passing --cluster on a busy production cluster.

Rehearse the restore, not the snapshot

Book an hour, build a throwaway single node cluster, restore last night's production snapshot into it and run kubectl get deploy -A. The number you get from that hour is your real recovery time, and it is the only number worth putting in a disaster recovery document.

If you would rather have this built, timed and documented against your own cluster, our disaster recovery and backup work starts here, and Kubernetes management keeps the snapshots verified afterwards. The same discipline applied to application databases is covered in How to Build a Backup You Have Actually Restored.