Backup and Restore Kubernetes ETCD database
Create a test secret and check
kubectl create secret generic test-secret \
--from-literal=username='svcaccount' \
--from-literal=password='S0mthingS0Str0ng!'
kubectl get secret test-secret
kubectl get nodes
Define a variable for the endpoint to etcd
ENDPOINT=https://127.0.0.1:2379
Install etcdctl
Check running version and install etcd locally
kubectl exec -it etcd-c1-cp1 -n kube-system -- /bin/sh -c 'ETCDCTL_API=3 /usr/local/bin/etcd --version' | head
Set release version and download etcdctl
export RELEASE="3.5.4"
wget https://github.com/etcd-io/etcd/releases/download/v${RELEASE}/etcd-v${RELEASE}-linux-amd64.tar.gz
Install etcdctl
tar -zxvf etcd-v${RELEASE}-linux-amd64.tar.gz
cd etcd-v${RELEASE}-linux-amd64
sudo cp etcdctl /usr/local/bin
Take the backup saving it to /var/lib/dat-backup.db
Be sure to copy that to remote storage when doing this for real
sudo ETCDCTL_API=3 etcdctl --endpoints=$ENDPOINT \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /var/lib/dat-backup.db
Read the metadata from the backup/snapshot to print out the snapshot’s status
sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status /var/lib/dat-backup.db
Delete the test secret to create backup/restore proof
kubectl delete secret test-secret
kubectl get secret test-secret
Restore the backup from /var/lib/dat-backup.db
Values for command below cab be found in /etc/kubernetes/manifests/etcd.yaml
ETCDCTL_API=3 etcdctl snapshot restore \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
--endpoints=$ENDPOINT \
--data-dir="/var/lib/etcdbkp" \
--initial-cluster="c1-cp1=https://<IP>:2380" \
--name="c1-cp1" \
--initial-advertise-peer-urls="https://<IP>:2380" \
/var/lib/dat-backup.db
Modify etcd.yaml
Specify the volume mountpath and host path as well in etcd.yaml configuration file. Usually this file resides under /etc/kubernetes/manifests directory.
vi /etc/kubernetes/manifests/etcd.yaml
Note: There are 3 values of data-dir. Please change all of them. If not you will encounter many problems.
...
spec:
containers:
- command:
...
--data-dir=/var/lib/etcdbkp
...
volumeMounts:
- mountPath: /var/lib/etcdbkp
...
- hostPath:
path: /var/lib/etcdbkp
...
Restart the kubelet
systemctl daemon-reload
systemctl restart kubelet
Check ETCD Container
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps
Check Member List
sudo ETCDCTL_API=3 etcdctl --endpoints=$ENDPOINT \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
member list
Check test secret and nodes
kubectl get secret test-secret
kubectl get nodes
Source:
https://hijackson.com/kokekloud-backup-and-restore-etcd/
https://www.cyberithub.com/how-to-backup-and-restore-kubernetes-etcd-database-step-by-step/