etcd backup and restore lab for Kubernetes administrators

Published 3 August 2026 · 14 minute read

An etcd snapshot is not a recovery plan until you have validated it, restored it into a new data directory, updated the control-plane configuration correctly, and proved that the Kubernetes API and controllers return to healthy operation.

What etcd protects

etcd stores Kubernetes API state: Deployments, Secrets, ConfigMaps, RBAC objects, custom resources, and controller intent. Container images, external volumes, cloud databases, and application data outside the Kubernetes API require separate backup systems.

1. Discover the real endpoint and certificates

Do not copy certificate paths from a tutorial without checking the running static Pod. Inspect the manifest and process arguments.

sudo grep -E -- '--(advertise-client-urls|cert-file|key-file|trusted-ca-file|data-dir)' \
  /etc/kubernetes/manifests/etcd.yaml

kubectl -n kube-system get pod -l component=etcd -o wide

Record the endpoint, CA certificate, client certificate, client key, data directory, and current cluster health. Recovery commands are trustworthy only when they match the actual control plane.

2. Create a marker object

Create a harmless object before the snapshot so you have a clear integrity check after restoration.

kubectl create namespace restore-proof
kubectl -n restore-proof create configmap marker \
  --from-literal=created-before-snapshot=true

3. Save and validate the snapshot

Use the etcd client version supplied by the cluster or an explicitly compatible version. Recent tooling may expose snapshot operations through `etcdutl`; follow the syntax supported by your installed release.

sudo ETCDCTL_API=3 etcdctl snapshot save /var/lib/etcd/snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

sudo ETCDCTL_API=3 etcdctl snapshot status /var/lib/etcd/snapshot.db --write-out=table

Copy the snapshot off the original data volume. Record its checksum, size, revision, and validation output.

4. Introduce a controlled state change

Delete the marker or create a second object after the snapshot. This produces a visible difference between snapshot state and current state.

kubectl delete namespace restore-proof
kubectl create namespace after-snapshot

5. Restore into a new data directory

Never overwrite the active directory in place. Stop the static Pod safely by moving its manifest, restore into a new directory, set correct ownership, and update the manifest to reference the restored path.

sudo mv /etc/kubernetes/manifests/etcd.yaml /tmp/etcd.yaml.disabled
sudo etcdutl snapshot restore /var/lib/etcd/snapshot.db \
  --data-dir=/var/lib/etcd-restored
sudo chown -R root:root /var/lib/etcd-restored

Edit the disabled manifest so both the `--data-dir` argument and the hostPath volume point to the restored directory. Return the manifest to `/etc/kubernetes/manifests/` and allow the kubelet to recreate etcd.

6. Prove recovery at multiple layers

• etcd endpoint health succeeds.
• The API server becomes reachable.
• The pre-snapshot marker exists again.
• The post-snapshot namespace is absent, as expected.
• Controllers reconcile Deployments and new Pods can be scheduled.
• CoreDNS and a representative application request succeed.

kubectl get namespace restore-proof
kubectl get namespace after-snapshot
kubectl get --raw=/readyz?verbose
kubectl create deployment restore-test --image=nginx
kubectl rollout status deployment/restore-test --timeout=120s

Common restore failures

Wrong certificate identity, incompatible client tooling, incorrect ownership, updating only the command but not the hostPath, restoring into the active directory, and forgetting that a multi-member cluster requires member-aware recovery. Capture component logs before changing additional variables.

Turn the lab into an operational control

Schedule snapshots, encrypt and copy them off-cluster, define retention, monitor failures, and rehearse restoration. Measure recovery time and document who can access the keys. A successful snapshot command is only the beginning.