通过备份 Etcd 来完美恢复 Kubernetes 中的误删数据

误删除或者服务器宕机,会导致 Etcd 数据的丢失或某个节点的 Etcd 数据异常时,当误删时,需要恢复数据,这个在实际环境当中是不可避免的。以下描述删除两个 namespace 下的 Pod,如何恢复对应 namespace 的数据。

1、操作环境

  • 3 个(master、etcd)+1 个 node
  • 新建 1 个 namespace 下且创建 Pod 和 default namespace 下创建 Pod

2、准备
数据误删除已经在 Etcd 备份

3、数据环境的模拟
3.1、新建 test namespace,在该 ns 下创建 Pod,创建成功之后,此时这些数据已存在 Etcd 中

kubectl get pod -n test
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-7d78fc5688-ntvtd       1/1     Running   0          12h
productpage-v1-844495cb4b-qn6l9   1/1     Running   0          12h
ratings-v1-55ccf46fb4-pwqw7       1/1     Running   0          12h
reviews-v1-68bb7b8c4f-h6vr9       1/1     Running   0          12h

3.2、在 default namespace 创建 Pod,如下所示,创建成功之后,此时这些数据已存在 Etcd 中

kubectl get pod
NAME                      READY   STATUS    RESTARTS   AGE
nginx-6db489d4b7-jdfkw    1/1     Running   0          12h
nginx1-675bf6c9f-jndrx    1/1     Running   0          12h
nginx2-6dfc958b55-vgsc4   1/1     Running   0          12h
nginx3-8f65b44f9-2pk4l    1/1     Running   0          12h
nginx4-9bb966479-svhs6    1/1     Running   0          12h
nginx5-7d4d998c6c-7xlp8   1/1     Running   0          12h
nginx6-79444687cf-rc8vk   1/1     Running   0          12h

3.3、 在某目录下已存放包含上述产生数据的 Etcd 快照

ls
member  snapshot.db

3.4、模拟数据的删除
default 和 test namespace 下 Pod 都被干掉

kubectl get pod
No resources found in default namespace.
[root@node1 etcd-2021-01-19_02:00:01]# kubectl get pod -n test
No resources found in test namespace.

4、数据恢复
停止所有 master 上 kube-apiserver 服务,因为和 Etcd 交互是 kube-apiserver。此步骤不会影响正常运行的 Pod。

4.1、所有 master 机器上操作如下指令,将 kube-apiserver.yaml 文件移除当前目录下,作用是停止 kube-apiserver 服务

mv /etc/kubernetes/manifests/kube-apiserver.yaml .

4.2、检查 kube-apiserver 服务已停止

kubectl get pod -n kube-system | grep kube-apiserver
Unable to connect to the server: EOF

4.3、所有 Etcd 机器上,停止 Etcd 运行,此步操作不会影响现有的 Pod,只是不能新建、删除 Pod

systemctl stop etcd

4.4、查看 Etcd 服务是否已停止

docker ps -a | grep etcd
18dfbbfb8317   kubesphere/etcd:v3.3.12            "/usr/local/bin/etcd"    15 hours ago   Exited (0) 45 seconds ago             etcd3

4.5、移除所有 Etcd 存储目录下数据,不同环境下,存储目录可能不一样,可以通过 systemctl status etcd 查看 Etcd 配置参数

mv /var/lib/etcd /var/lib/etcd.bak

4.6、将 node1 机器上的快照 snapshot.db 分别拷贝到另外两台 Etcd 机器上

scp snapshot.db 192.168.0.26:/root

scp snapshot.db 192.168.0.28:/root

5、恢复备份
不同环境下,目录可能不一样,可以通过 systemctl status etcd 查看 Etcd 配置参数。特别需要注意 name、initial-cluster、initial-cluster-token、initial-advertise-peer-urls 和 data-dir 参数的值。

5.1、在第一台 Etcd 节点上,注意需要 ETCDCTL_API=3、name 值、IP 值、snapshot.db 文件目录和 data-dir 目录。

export ETCDCTL_API=3
一条指令,可以直接在终端上修改里面参数
etcdctl snapshot restore snapshot.db --name etcd1 --initial-cluster "etcd1=https://192.168.0.25:2380,etcd2=https://192.168.0.26:2380,etcd3=https://192.168.0.28:2380" --initial-cluster-token k8s_etcd --initial-advertise-peer-urls https://192.168.0.25:2380 --data-dir=/var/lib/etcd
和上面指令一样作用,把长的指令以换行形式展现
etcdctl snapshot restore snapshot.db --name etcd1 \
--initial-cluster "etcd1=https://192.168.0.25:2380,etcd2=https://192.168.0.26:2380,etcd3=https://192.168.0.28:2380" \
--initial-cluster-token k8s_etcd \
--initial-advertise-peer-urls https://192.168.0.25:2380 \
--data-dir=/var/lib/etcd

2021-01-19 11:17:06.773113 I | mvcc: restore compact to 96139
2021-01-19 11:17:06.800086 I | etcdserver/membership: added member 7370b1d3dc967c [https://192.168.0.25:2380] to cluster e4d7f96e88cc9d71
2021-01-19 11:17:06.800159 I | etcdserver/membership: added member 2ef3cfc4ca48ad38 [https://192.168.0.26:2380] to cluster e4d7f96e88cc9d71
2021-01-19 11:17:06.800190 I | etcdserver/membership: added member 3a0c86c4c744477c [https://192.168.0.28:2380] to cluster e4d7f96e88cc9d71

5.2、第二台和第三台 Etcd 恢复数据,同样需要改变 ETCDCTL_API=3、name 值、IP 值、snapshot.db 文件目录和 data-dir 目录。

export ETCDCTL_API=3
一条指令,可以直接在终端上修改里面参数
etcdctl snapshot restore snapshot.db --name etcd2 --initial-cluster "etcd1=https://192.168.0.25:2380,etcd2=https://192.168.0.26:2380,etcd3=https://192.168.0.28:2380" --initial-cluster-token k8s_etcd --initial-advertise-peer-urls https://192.168.0.26:2380 --data-dir=/var/lib/etcd
和上面指令一样作用,把长的指令以换行形式展现
etcdctl snapshot restore snapshot.db --name etcd2 \
--initial-cluster "etcd1=https://192.168.0.25:2380,etcd2=https://192.168.0.26:2380,etcd3=https://192.168.0.28:2380" \
--initial-cluster-token k8s_etcd \
--initial-advertise-peer-urls https://192.168.0.26:2380 \
--data-dir=/var/lib/etcd

2021-01-19 11:19:59.857363 I | mvcc: restore compact to 96139
2021-01-19 11:19:59.873793 I | etcdserver/membership: added member 7370b1d3dc967c [https://192.168.0.25:2380] to cluster e4d7f96e88cc9d71
2021-01-19 11:19:59.873837 I | etcdserver/membership: added member 2ef3cfc4ca48ad38 [https://192.168.0.26:2380] to cluster e4d7f96e88cc9d71
2021-01-19 11:19:59.873852 I | etcdserver/membership: added member 3a0c86c4c744477c [https://192.168.0.28:2380] to cluster e4d7f96e88cc9d71

export ETCDCTL_API=3
一条指令,可以直接在终端上修改里面参数
etcdctl snapshot restore snapshot.db --name etcd3 --initial-cluster "etcd1=https://192.168.0.25:2380,etcd2=https://192.168.0.26:2380,etcd3=https://192.168.0.28:2380" --initial-cluster-token k8s_etcd --initial-advertise-peer-urls https://192.168.0.28:2380 --data-dir=/var/lib/etcd
和上面指令一样作用,把长的指令以换行形式展现
etcdctl snapshot restore snapshot.db --name etcd3 \
--initial-cluster "etcd1=https://192.168.0.25:2380,etcd2=https://192.168.0.26:2380,etcd3=https://192.168.0.28:2380" \
--initial-cluster-token k8s_etcd \
--initial-advertise-peer-urls https://192.168.0.28:2380 \
--data-dir=/var/lib/etcd

2021-01-19 11:22:21.423215 I | mvcc: restore compact to 96139
2021-01-19 11:22:21.438319 I | etcdserver/membership: added member 7370b1d3dc967c [https://192.168.0.25:2380] to cluster e4d7f96e88cc9d71
2021-01-19 11:22:21.438357 I | etcdserver/membership: added member 2ef3cfc4ca48ad38 [https://192.168.0.26:2380] to cluster e4d7f96e88cc9d71
2021-01-19 11:22:21.438371 I | etcdserver/membership: added member 3a0c86c4c744477c [https://192.168.0.28:2380] to cluster e4d7f96e88cc9d71

5.3、上面三台 Etcd 启动

systemctl start etcd

5.4、检查 Etcd 集群状态,注意:如果有证书,需要加上证书;在哪台 Etcd 机器执行,注意找到对应的证书文件及最后面加 endpoint health 字样。

一条指令,可以直接在终端上修改里面参数
etcdctl --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-node3.pem --key=/etc/ssl/etcd/ssl/node-node3-key.pem --endpoints=https://192.168.0.25:2379,https://192.168.0.26:2379,https://192.168.0.28:2379 endpoint health
和上面指令一样作用,把长的指令以换行形式展现
etcdctl --cacert=/etc/ssl/etcd/ssl/ca.pem \
--cert=/etc/ssl/etcd/ssl/node-node3.pem \
--key=/etc/ssl/etcd/ssl/node-node3-key.pem \
--endpoints=https://192.168.0.25:2379,https://192.168.0.26:2379,https://192.168.0.28:2379 \
endpoint health

https://192.168.0.28:2379 is healthy: successfully committed proposal: took = 11.664519ms
https://192.168.0.26:2379 is healthy: successfully committed proposal: took = 5.04665ms
https://192.168.0.25:2379 is healthy: successfully committed proposal: took = 1.837265ms

5.5、三台 Etcd 全部正常,分别到每台 master 启动 kube-apiserver

mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/

5.6、检查 Kubernetes 集群和创建的 Pod 是否恢复正常,包括 Etcd 服务、kube-apiserver 服务及对应 ns 下删除的 Pod。

kubectl get pod -n kube-system | grep kube-apiserver
kube-apiserver-node1                           1/1     Running   0          16h
kube-apiserver-node2                           1/1     Running   0          16h
kube-apiserver-node3                           1/1     Running   0          16h


[root@node1 ssl]# docker ps -a | grep etcd
156b19a72bf0   kubesphere/etcd:v3.3.12      "/usr/local/bin/etcd"    15 minutes ago   Up 15 minutes                           etcd1


kubectl get pod
NAME                      READY   STATUS              RESTARTS   AGE
nginx-6db489d4b7-jdfkw    1/1     Running             0          13h
nginx1-675bf6c9f-jndrx    1/1     Running             0          13h
nginx2-6dfc958b55-vgsc4   1/1     Running             0          13h
nginx3-8f65b44f9-2pk4l    1/1     Running             0          13h
nginx4-9bb966479-svhs6    1/1     Running             0          13h
nginx5-7d4d998c6c-7xlp8   0/1     ContainerCreating   0          13h
nginx6-79444687cf-rc8vk   1/1     Running             0          13h


[root@node1 ssl]# kubectl get pod -n test
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-7d78fc5688-ntvtd       1/1     Running   0          13h
productpage-v1-844495cb4b-qn6l9   1/1     Running   0          13h
ratings-v1-55ccf46fb4-pwqw7       1/1     Running   0          13h
reviews-v1-68bb7b8c4f-h6vr9       1/1     Running   0          13h
上一篇:VScode 运行代码显示:Code is already running!


下一篇:PG_插件-pg_hint_plan