ETCD is the Kubernetes database. Everything in the cluster is stored here:
If ETCD is lost, the cluster state is lost. This is why backups are critical.
Before doing anything, inspect the ETCD static pod manifest. This file contains:
cat /etc/kubernetes/manifests/etcd.yaml
Useful fields:
--cert-file=
--key-file=
--trusted-ca-file=
--data-dir=
kubectl get pods -n kube-system | grep etcd
Expected output:
etcd-controlplane
Check ETCD health:
export ETCDCTL_API=3
etcdctl endpoint health \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
//health check that works ************ POS AWS 🤬🤬🤬
ETCDCTL_API=3 /snap/etcd/249/bin/etcdctl endpoint health \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
Expected output:
https://127.0.0.1:2379 is healthy
List all keys stored in ETCD:
ETCDCTL_API=3 etcdctl get / --prefix --keys-only \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
You may see:
/registry/pods
/registry/deployments
/registry/services
/registry/secrets
View a specific object:
ETCDCTL_API=3 etcdctl get /registry/pods/default/nginx \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
This is the command you should memorize.
ETCDCTL_API=3 etcdctl snapshot save /opt/etcd-backup.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
Verify backup exists:
ls -lh /opt/etcd-backup.db
Check snapshot details:
ETCDCTL_API=3 etcdctl snapshot status /opt/etcd-backup.db
Expected output:
HASH
REVISION
TOTAL KEYS
TOTAL SIZE
Restore the backup into a new data directory.
ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restored
Verify directory exists:
ls /var/lib/etcd-restored
Edit the ETCD manifest:
vi /etc/kubernetes/manifests/etcd.yaml
Find:
--data-dir=/var/lib/etcd
Replace with:
--data-dir=/var/lib/etcd-restored
volumes:
- hostPath:
path: /var/lib/etcd
Replace with:
volumes:
- hostPath:
path: /var/lib/etcd-restored
When the manifest changes, kubelet automatically recreates the ETCD pod.
Watch ETCD recover:
kubectl get pods -n kube-system -w
Verify cluster access:
kubectl get nodes
Expected:
NAME STATUS
controlplane Ready
cat /etc/kubernetes/manifests/etcd.yaml
export ETCDCTL_API=3
etcdctl snapshot save /opt/snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=<ca.crt> \
--cert=<cert.crt> \
--key=<key.key>
etcdctl snapshot restore /opt/snapshot.db \
--data-dir=/var/lib/etcd-restore
1. Find certs in etcd.yaml
2. snapshot save
3. snapshot restore
4. edit etcd.yaml
5. change data-dir
6. change hostPath
7. save
8. verify cluster