虚拟机上k8s部署好的第二天用时总是出现的各种问题
open /run/flannel/subnet.env: no such file or directory
open /run/flannel/subnet.env: no such file or directory
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9a5eade3c13f1eeeb000df80e942ed22e59d2c532def6f1f281fd2ebefdcfa2c" network for pod "mcw01dep-nginx1-69bc6f5957-lzpdv": networkPlugin cni failed to set up pod "mcw01dep-nginx1-69bc6f5957-lzpdv_default" network: open /run/flannel/subnet.env: no such file or directory
scp /run/flannel/subnet.env 结点
scp /run/flannel/subnet.env 10.0.0.6:/run/flannel/
问题一：查看网络状态报错：RTNETLINK answers: File exists错误解决方法
CentOS7 Failed to start LSB: Bring up/down networking
RTNETLINK answers: File exists错误解决方法
https://blog.csdn.net/u010719917/article/details/79423180
chkconfig --level 35 network on
chkconfig --level 0123456 NetworkManager off
service NetworkManager stop
service network stop
service network start
如果还不行，重启系统看看
service network start 出现RTNETLINKanswers:Fileexists错误解决 或者
/etc/init.d/network start 出现RTNETLINKanswers:Fileexists错误解决
(其实两者是等效的，其实前者执行的就是这个命令)
在centos下出现该故障的原因是启动网络的两个服务有冲突：
/etc/init.d/network 和
/etc/init.d/NetworkManager 这两个服务有冲突吧。
从根本上说是NetworkMaganager（NM）的带来的冲突，停用NetworkManager即可解决，重启即可。
1.切换到root账户，并用chkconfig命令查看network 和NetworkManager两个服务的开机启动配置情况；
=====
只是执行如下三个命令就成功了
service NetworkManager stop
service network stop
service network start
问题二：ping外网时，Destination Host Unreachable。from 内网ip
排查过程
[root@mcw7 ~]$ ping www.baidu.com
PING www.a.shifen.com (220.181.38.149) 56(84) bytes of data.
From bogon (172.16.1.137) icmp_seq=1 Destination Host Unreachable 查看能通外网的路由表
[root@mcw8 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
0.0.0.0 172.16.1.2 0.0.0.0 UG 101 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
172.16.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 查看不通外网的路由表，发现缺少了一条关于10.0.0.2的路由，
应该加一条如上的路由试试0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37 加错路由了，删除
route add -host 10.0.0.137 gw 10.0.0.2
[root@mcw7 ~]$ route add -host 10.0.0.137 gw 10.0.0.2
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
10.0.0.137 10.0.0.2 255.255.255.255 UGH 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37 删除路由 -host后面的ip，在路由的第一列，目标地址。我这里应该填0.0.0.0。目标地址是任意的，指定gw是10.0.0.2
[root@mcw7 ~]$ route del -host 10.0.0.137 dev ens33
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37
[root@mcw7 ~]$ -host是指去往的目的主机，这里子网掩码应该设置为0.0.0.0，需要手动删除重建。旗帜貌似多了H，不知道干嘛的
[root@mcw7 ~]$ route add -host 0.0.0.0 gw 10.0.0.2
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 255.255.255.255 UGH 0 0 0 ens33
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37 删除指定目的主机，指定网卡接口
[root@mcw7 ~]$ route del -host 0.0.0.0 dev ens33
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37 看提示信息，指定掩码用netmask，genmask和mask和是255的那种，取补集
[root@mcw7 ~]$ route add -host 0.0.0.0 MASK 0.0.0.0 gw 10.0.0.2
Usage: inet_route [-vF] del {-host|-net} Target[/prefix] [gw Gw] [metric M] [[dev] If]
inet_route [-vF] add {-host|-net} Target[/prefix] [gw Gw] [metric M]
[netmask N] [mss Mss] [window W] [irtt I]
[mod] [dyn] [reinstate] [[dev] If]
inet_route [-vF] add {-host|-net} Target[/prefix] [metric M] reject
inet_route [-FC] flush NOT supported
[root@mcw7 ~]$
[root@mcw7 ~]$
[root@mcw7 ~]$ route add -host 0.0.0.0 netmask 0.0.0.0 gw 10.0.0.2
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 255.255.255.255 UGH 0 0 0 ens33
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37
[root@mcw7 ~]$ 再次删除 route del 指定目的主机，指定接口
[root@mcw7 ~]$ route del -host 0.0.0.0 dev ens33
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37
真正解决前的描述
删除默认网关
[root@mcw7 ~]$ route del -host 0.0.0.0 dev ens33
SIOCDELRT: No such process
[root@mcw7 ~]$
[root@mcw7 ~]$ route del -host 0.0.0.0 dev ens37
SIOCDELRT: No such process
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
0.0.0.0 172.16.1.2 0.0.0.0 UG 101 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
172.16.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
[root@mcw7 ~]$ route del -host 0.0.0.0
SIOCDELRT: No such process
[root@mcw7 ~]$ route del default
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.1.2 0.0.0.0 UG 100 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
172.16.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
[root@mcw7 ~]$ route del default
[root@mcw7 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
172.16.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
[root@mcw7 ~]$
真正的解决这个问题
参考：https://www.cnblogs.com/skgoo/p/13559964.html
mcw8上ping不能到外网，显示包来自服务器内网ip。
[root@mcw8 ~]$ ping www.baidu.com
PING www.a.shifen.com (39.156.66.14) 56(84) bytes of data.
From mcw8 (172.16.1.138) icmp_seq=1 Destination Host Unreachable mcw9上能ping通外网，显示包来着外网百度ip
[root@mcw9 ~]$ ping www.baidu.com
PING www.a.shifen.com (39.156.66.18) 56(84) bytes of data.
64 bytes from 39.156.66.18 (39.156.66.18): icmp_seq=1 ttl=128 time=43.2 ms 查看mcw9正常网关，是有10.0.0.2的网关ip
[root@mcw9 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
0.0.0.0 172.16.1.2 0.0.0.0 UG 101 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
172.16.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 查看mcw8异常网络的路由，没有外网的网关10.0.0.2。
[root@mcw8 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 给mcw8添加默认网关，上面之前添加各种路由，结果genmask都不对，不能变成0.0.0.0。而使用如下命令，才实现了
Destination是0.0.0.0，Gateway是10.0.0.2，Genmask是0.0.0.0 ，Flags是UG，Iface是ens33。然后才成功访问外网
[root@mcw8 ~]$ route add default gw 10.0.0.2
[root@mcw8 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 0 0 0 ens33
0.0.0.0 172.16.1.2 0.0.0.0 UG 0 0 0 ens37
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens33
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 ens37
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens37
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 在mcw8上可以正常访问外网了
[root@mcw8 ~]$ ping www.baidu.com
PING www.a.shifen.com (39.156.66.14) 56(84) bytes of data.
64 bytes from 39.156.66.14 (39.156.66.14): icmp_seq=1 ttl=128 time=23.5 ms
64 bytes from 39.156.66.14 (39.156.66.14): icmp_seq=2 ttl=128 time=36.7 ms
由于第二次部署flannel下面网络不通了，网站访问不了(查域名是禁止查询的域名)，但是我以前有把这个文件内容保存下来。这样我直接把文件内容复制进来，直接部署就可以了。如下
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[machangwei@mcw7 ~]$ ls
mcw.txt mm.yml scripts tools
[machangwei@mcw7 ~]$ kubectl apply -f mm.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
因为忘记init的加入集群的命令了。所以当我要kubeadm init，然后执行kubeadm reset之后，原本有的容器都没了
排查过程，以及IPtables规则的导出和导入
[root@mcw7 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@mcw7 ~]$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@mcw7 ~]$
重设然后重初始化后，网络也没有的
[root@mcw7 ~]$ docker ps|grep kube-flannel
[root@mcw7 ~]$ 进入普通用户重新部署网络报错
[machangwei@mcw7 ~]$ ls
mcw.txt mm.yml scripts tools
[machangwei@mcw7 ~]$ kubectl apply -f mm.yml
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
[machangwei@mcw7 ~]$ 查询之前重设的信息。发现说不能清除CNI的信息
[root@mcw7 ~]$ echo y|kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: [preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni] The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command. If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables. The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file. 移除文件不管用
[root@mcw7 ~]$ mv /etc/cni/net.d /etc/cni/net.dbak
[root@mcw7 ~]$ ipvsadm --clear
-bash: ipvsadm: command not found 查看了一大堆，不知道咋弄
[root@mcw7 ~]$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes health check service ports */
KUBE-EXTERNAL-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
INPUT_direct all -- anywhere anywhere
INPUT_ZONES_SOURCE all -- anywhere anywhere 既然无法清除，那么直接从其它机子导出导入一份规则
导出：
[root@mcw9 ~]$ iptables-save > /root/iptables_beifen.txt
[root@mcw9 ~]$ cat iptables_beifen.txt
# Generated by iptables-save v1.4.21 on Fri Jan 7 23:05:39 2022
*filter
:INPUT ACCEPT [1676:135745]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [896:67997]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri Jan 7 23:05:39 2022
# Generated by iptables-save v1.4.21 on Fri Jan 7 23:05:39 2022
*nat
:PREROUTING ACCEPT [32:2470]
:INPUT ACCEPT [32:2470]
:OUTPUT ACCEPT [8:528]
:POSTROUTING ACCEPT [8:528]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Fri Jan 7 23:05:39 2022
[root@mcw9 ~]$ cat mcw7上导入规则。出错，文件有问题。第一行注释一下吧
[root@mcw7 ~]$ iptables-restore</root/daoru.txt
iptables-restore: line 1 failed
[root@mcw7 ~]$ cat daoru.txt #命令
ptables-save v1.4.21 on Fri Jan 7 23:05:39 2022
*filter
:INPUT ACCEPT [1676:135745] 导入，防火墙规则一致了
https://blog.csdn.net/jake_tian/article/details/102548306
[root@mcw7 ~]$ iptables-restore</root/daoru.txt
[root@mcw7 ~]$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT)
target prot opt source destination Chain DOCKER (1 references)
target prot opt source destination Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere ============
再次执行，试一试 重试
[root@mcw7 ~]$ echo y|kubeadm reset 再看防火墙，貌似是没有变化
[root@mcw7 ~]$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT)
target prot opt source destination Chain DOCKER (1 references)
target prot opt source destination Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
[root@mcw7 ~]$ 重新初始化后
[root@mcw7 ~]$ kubeadm init --apiserver-advertise-address 10.0.0.137 --pod-network-cidr=10.244.0.0/24 --image-repository=registry.aliyuncs.com/google_containers
kubeadm join 10.0.0.137:6443 --token 1e2kkw.ivkth6zzkbx72z4u \
--discovery-token-ca-cert-hash sha256:fb83146082fb33ca2bff56a525c1e575b5f2587ab1be566f9dd3d7e8d7845462 [root@mcw7 ~]$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes health check service ports */
KUBE-EXTERNAL-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL all -- anywhere anywhere Chain FORWARD (policy ACCEPT)
target prot opt source destination
KUBE-FORWARD all -- anywhere anywhere /* kubernetes forwarding rules */
KUBE-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere Chain DOCKER (1 references)
target prot opt source destination Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere Chain KUBE-EXTERNAL-SERVICES (2 references)
target prot opt source destination Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000 你可能无法记住做题的步骤，但是你能根据笔记把题很快做出来，还有把握保证是对的
你可能无法记住部署的步骤，执行的每一个命令，但是你能根据自己以前的笔记很快做出来 原来这个问题跟防火墙没有关系。
[machangwei@mcw7 ~]$ kubectl apply -f mm.yml
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
[machangwei@mcw7 ~]$
[machangwei@mcw7 ~]$ kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
真正的解决方法：
做法如下，重新用普通用户配置kubectl，以前的配置失效了
[machangwei@mcw7 ~]$ ls -a
. .. .bash_history .bash_logout .bash_profile .bashrc .kube mcw.txt mm.yml scripts tools .viminfo
[machangwei@mcw7 ~]$ mv .kube kubebak
[machangwei@mcw7 ~]$ mkdir -p $HOME/.kube
[machangwei@mcw7 ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[machangwei@mcw7 ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
[machangwei@mcw7 ~]$ kubectl get node
NAME STATUS ROLES AGE VERSION
mcw7 NotReady control-plane,master 10m v1.23.1
重新创建网络
[machangwei@mcw7 ~]$ ls
kubebak mcw.txt mm.yml scripts tools
[machangwei@mcw7 ~]$ kubectl apply -f mm.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
此时再次用-L查看防火墙
[root@mcw7 ~]$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes health check service ports */
KUBE-EXTERNAL-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL all -- anywhere anywhere Chain FORWARD (policy ACCEPT)
target prot opt source destination
KUBE-FORWARD all -- anywhere anywhere /* kubernetes forwarding rules */
KUBE-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ACCEPT all -- mcw7/16 anywhere
ACCEPT all -- anywhere mcw7/16 Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere Chain DOCKER (1 references)
target prot opt source destination Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere Chain KUBE-EXTERNAL-SERVICES (2 references)
target prot opt source destination Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
^C 看规则，应该用下面的才是合适的
[root@mcw7 ~]$ iptables-save
# Generated by iptables-save v1.4.21 on Sat Jan 8 07:35:11 2022
*nat
:PREROUTING ACCEPT [372:18270]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [239:14302]
:POSTROUTING ACCEPT [239:14302]
:DOCKER - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SEP-6E7XQMQ4RAYOWTTM - [0:0]
:KUBE-SEP-IT2ZTR26TO4XFPTO - [0:0]
:KUBE-SEP-N4G2XR5TDX7PQE7P - [0:0]
:KUBE-SEP-XOVE7RWZIDAMLO2S - [0:0]
:KUBE-SEP-YIL6JZP7A3QYXJU2 - [0:0]
:KUBE-SEP-ZP3FB6NMPNCO4VBJ - [0:0]
:KUBE-SEP-ZXMNUKOKXUTL2MK2 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE
-A KUBE-SEP-6E7XQMQ4RAYOWTTM -s 10.244.0.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-6E7XQMQ4RAYOWTTM -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.3:53
-A KUBE-SEP-IT2ZTR26TO4XFPTO -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-IT2ZTR26TO4XFPTO -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.2:53
-A KUBE-SEP-N4G2XR5TDX7PQE7P -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-N4G2XR5TDX7PQE7P -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.244.0.2:9153
-A KUBE-SEP-XOVE7RWZIDAMLO2S -s 10.0.0.137/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-XOVE7RWZIDAMLO2S -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 10.0.0.137:6443
-A KUBE-SEP-YIL6JZP7A3QYXJU2 -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-YIL6JZP7A3QYXJU2 -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.2:53
-A KUBE-SEP-ZP3FB6NMPNCO4VBJ -s 10.244.0.3/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZP3FB6NMPNCO4VBJ -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.244.0.3:9153
-A KUBE-SEP-ZXMNUKOKXUTL2MK2 -s 10.244.0.3/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZXMNUKOKXUTL2MK2 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.3:53
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 ! -s 10.244.0.0/24 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-IT2ZTR26TO4XFPTO
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-ZXMNUKOKXUTL2MK2
-A KUBE-SVC-JD5MR3NA4I4DYORP ! -s 10.244.0.0/24 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-N4G2XR5TDX7PQE7P
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-SEP-ZP3FB6NMPNCO4VBJ
-A KUBE-SVC-NPX46M4PTMTKRN6Y ! -s 10.244.0.0/24 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-XOVE7RWZIDAMLO2S
-A KUBE-SVC-TCOU7JCQXEZGVUNU ! -s 10.244.0.0/24 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-YIL6JZP7A3QYXJU2
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-6E7XQMQ4RAYOWTTM
COMMIT
# Completed on Sat Jan 8 07:35:11 2022
# Generated by iptables-save v1.4.21 on Sat Jan 8 07:35:11 2022
*mangle
:PREROUTING ACCEPT [376111:67516258]
:INPUT ACCEPT [369347:67204288]
:FORWARD ACCEPT [6764:311970]
:OUTPUT ACCEPT [369958:67425919]
:POSTROUTING ACCEPT [371215:67488646]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_docker - [0:0]
:PRE_docker_allow - [0:0]
:PRE_docker_deny - [0:0]
:PRE_docker_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i docker0 -j PRE_docker
-A PREROUTING_ZONES -i ens37 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_docker -j PRE_docker_log
-A PRE_docker -j PRE_docker_deny
-A PRE_docker -j PRE_docker_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Sat Jan 8 07:35:11 2022
# Generated by iptables-save v1.4.21 on Sat Jan 8 07:35:11 2022
*security
:INPUT ACCEPT [591940:133664590]
:FORWARD ACCEPT [1257:62727]
:OUTPUT ACCEPT [596315:107591486]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Sat Jan 8 07:35:11 2022
# Generated by iptables-save v1.4.21 on Sat Jan 8 07:35:11 2022
*raw
:PREROUTING ACCEPT [376111:67516258]
:OUTPUT ACCEPT [369958:67425919]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_docker - [0:0]
:PRE_docker_allow - [0:0]
:PRE_docker_deny - [0:0]
:PRE_docker_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i docker0 -j PRE_docker
-A PREROUTING_ZONES -i ens37 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_docker -j PRE_docker_log
-A PRE_docker -j PRE_docker_deny
-A PRE_docker -j PRE_docker_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Sat Jan 8 07:35:11 2022
# Generated by iptables-save v1.4.21 on Sat Jan 8 07:35:11 2022
*filter
:INPUT ACCEPT [14882:2406600]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [15254:2447569]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Sat Jan 8 07:35:11 2022
[root@mcw7 ~]$ 回头研究忘记加入集群的命令，如何重新生成，以及是否对已加入集群的节点是否产生影响
/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1问题解决
重新加入节点，有警告信息，我们应该把警告信息注意起来，比如docker让它开机启动，如果我们虚拟机没有设置开机启动，那么万一重启了虚拟机，容器就挂了
[root@mcw8 ~]$ kubeadm join 10.0.0.137:6443 --token 1e2kkw.ivkth6zzkbx72z4u \
> --discovery-token-ca-cert-hash sha256:fb83146082fb33ca2bff56a525c1e575b5f2587ab1be566f9dd3d7e8d7845462
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Hostname]: hostname "mcw8" could not be reached
[WARNING Hostname]: hostname "mcw8": lookup mcw8 on 10.0.0.2:53: no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher 解决方法
[root@mcw8 ~]$ echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables
[root@mcw8 ~]$ kubeadm join 10.0.0.137:6443 --token 1e2kkw.ivkth6zzkbx72z4u --discovery-token-ca-cert-hash sha256:fb83146082fb33ca2bff56a525c1e575b5f2587ab1be566f9dd3d7e8d7845462[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Hostname]: hostname "mcw8" could not be reached
[WARNING Hostname]: hostname "mcw8": lookup mcw8 on 10.0.0.2:53: no such host
^C
[root@mcw8 ~]$ echo y|kubeadm reset 如上之后，还是不行，加不到mcw7 master节点，之前记得mcw8和mcw9两个node是没有部署k8s网络的，现在部署一下再试试。配置普通用户kubectl，然后 [machangwei@mcw7 ~]$ scp mm.yml 10.0.0.138:/home/machangwei/
machangwei@10.0.0.138's password:
mm.yml 100% 5412 8.5MB/s 00:00
[machangwei@mcw7 ~]$ scp mm.yml 10.0.0.139:/home/machangwei/
machangwei@10.0.0.139's password:
mm.yml 但是节点是不需要配置普通用户的kubectl的，因为缺少文件的
[root@mcw8 ~]$ su - machangwei
[machangwei@mcw8 ~]$ mkdir -p $HOME/.kube
[machangwei@mcw8 ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: cannot stat ‘/etc/kubernetes/admin.conf’: No such file or directory
[machangwei@mcw8 ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
chown: cannot access ‘/home/machangwei/.kube/config’: No such file or directory 加入集群一直卡住 ，加一个--V=2的参数，打印详情
[root@mcw8 ~]$ kubeadm join 10.0.0.137:6443 --token 1e2kkw.ivkth6zzkbx72z4u --discovery-token-ca-cert-hash sha256:fb83146082fb33ca2bff56a525c1e575b5f2587ab1be566f9dd3d7e8d7845462 --v=2
I0108 00:54:46.002913 32058 join.go:413] [preflight] found NodeName empty; using OS hostname as NodeName
I0108 00:54:46.068584 32058 initconfiguration.go:117] detected and using CRI socket: /var/run/dockershim.sock
[preflight] Running pre-flight checks
I0108 00:54:46.068919 32058 preflight.go:92] [preflight] Running general checks 发现报错信息
I0108 00:54:46.849380 32058 checks.go:620] validating kubelet version
I0108 00:54:46.927861 32058 checks.go:133] validating if the "kubelet" service is enabled and active
I0108 00:54:46.938910 32058 checks.go:206] validating availability of port 10250
I0108 00:54:46.960668 32058 checks.go:283] validating the existence of file /etc/kubernetes/pki/ca.crt
I0108 00:54:46.960707 32058 checks.go:433] validating if the connectivity type is via proxy or direct
I0108 00:54:46.960795 32058 join.go:530] [preflight] Discovering cluster-info
I0108 00:54:46.960846 32058 token.go:80] [discovery] Created cluster-info discovery client, requesting info from "10.0.0.137:6443"
I0108 00:54:46.997909 32058 token.go:118] [discovery] Requesting info from "10.0.0.137:6443" again to validate TLS against the pinned public key
I0108 00:54:47.003864 32058 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://10.0.0.137:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": x509: certificate has expired or is not yet valid: current time 2022-01-08T00:54:47+08:00 is before 2022-01-07T23:18:44Z 时间不一致，将mcw8改到错误时间前。mcw7也改了before 2022-01-07T23:18:44Z。然后错误已经变成别的了
[root@mcw8 ~]$ date -s "2022-1-7 23:10:00"
Fri Jan 7 23:10:00 CST 2022
根据上面可知，错误变成如下了net/http: request canceled (Client.Timeout exceeded while awaiting headers)
错误变成如下了
ter-info?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0108 01:27:42.577217 32662 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://10.0.0.137:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
k8s系统容器总是起不来，停掉，报错如下。然后把停掉的所有容器多删几次，就好了。重新添加进集群，报错：拒绝
rpc error: code = Unknown desc = failed to create a sandbox for pod \"coredns-6d8c4cb4d-8l99d\": Error response from daemon: Conflict. The container name \"/k8s_POD_coredns-6d8c4cb4d-8l99d_kube-system_e030f426-3e8e-46fe-9e05-6c42a332f650_2\" is already in use by container \"b2dbcdd338ab4b2c35d5386e50e7e116fd41f26a0053a84ec3f1329e09d454a4\". You have to remove (or rename) that container to be able to reuse that name." pod="kube-system/coredns-6d8c4cb4d-8l99d"
[root@mcw8 ~]$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2edd274fd7b5 e6ea68648f0c "/opt/bin/flanneld -…" 7 seconds ago Exited (1) 5 seconds ago k8s_kube-flannel_kube-flannel-ds-tvz9q_kube-system_e62fa7b1-1cce-42dc-91d8-cdbd2bfda0f3_2
5b1715be012d quay.io/coreos/flannel "cp -f /etc/kube-fla…" 28 seconds ago Exited (0) 27 seconds ago k8s_install-cni_kube-flannel-ds-tvz9q_kube-system_e62fa7b1-1cce-42dc-91d8-cdbd2bfda0f3_0
7beb96ed15be rancher/mirrored-flannelcni-flannel-cni-plugin "cp -f /flannel /opt…" About a minute ago Exited (0) About a minute ago k8s_install-cni-plugin_kube-flannel-ds-tvz9q_kube-system_e62fa7b1-1cce-42dc-91d8-cdbd2bfda0f3_0
4e998fdfce3e registry.aliyuncs.com/google_containers/kube-proxy "/usr/local/bin/kube…" 2 minutes ago Up 2 minutes k8s_kube-proxy_kube-proxy-5p7dn_kube-system_92b1b38a-f6fa-4308-93fb-8045d2bae63f_0
fed18476d9a3 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-flannel-ds-tvz9q_kube-system_e62fa7b1-1cce-42dc-91d8-cdbd2bfda0f3_0
ebc2403e3052 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-proxy-5p7dn_kube-system_92b1b38a-f6fa-4308-93fb-8045d2bae63f_0 已经好了
[machangwei@mcw7 ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
mcw7 Ready control-plane,master 7m22s v1.23.1
mcw8 Ready <none> 4m51s v1.23.1
mcw9 Ready <none> 3m45s v1.23.1
[machangwei@mcw7 ~]$ 每个加进集群部署好的节点，都有三个容器。加进集群的命令是访问主节点apiserver服务。然后就开始拉取镜像部署节点上的容器了
k8s_kube-proxy_kube-
k8s_POD_kube-proxy-n
k8s_POD_kube-flannel
pod状态：ContainerCreating，ErrImagePull，ImagePullBackOff
[machangwei@mcw7 ~]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
mcw01dep-nginx-5dd785954d-d2kwp 0/1 ContainerCreating 0 9m7s
mcw01dep-nginx-5dd785954d-szdjd 0/1 ErrImagePull 0 9m7s
mcw01dep-nginx-5dd785954d-v9x8j 0/1 ErrImagePull 0 9m7s
[machangwei@mcw7 ~]$
[machangwei@mcw7 ~]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
mcw01dep-nginx-5dd785954d-d2kwp 0/1 ContainerCreating 0 9m15s
mcw01dep-nginx-5dd785954d-szdjd 0/1 ImagePullBackOff 0 9m15s
mcw01dep-nginx-5dd785954d-v9x8j 0/1 ImagePullBackOff 0 9m15s node上的容器都删除，但是主节点pod还是删不掉了，强制删除
[machangwei@mcw7 ~]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
mcw01dep-nginx-5dd785954d-v9x8j 0/1 Terminating 0 33m
[machangwei@mcw7 ~]$ kubectl delete pod mcw01dep-nginx-5dd785954d-v9x8j --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "mcw01dep-nginx-5dd785954d-v9x8j" force deleted
[machangwei@mcw7 ~]$ kubectl get pod
No resources found in default namespace. 拉取镜像无效？？？容器都起来了
[machangwei@mcw7 ~]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
mcw01dep-nginx-5dd785954d-65zd4 0/1 ContainerCreating 0 118s
mcw01dep-nginx-5dd785954d-hfw2k 0/1 ContainerCreating 0 118s
mcw01dep-nginx-5dd785954d-qxzpl 0/1 ContainerCreating 0 118s Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 112s default-scheduler Successfully assigned default/mcw01dep-nginx-5dd785954d-65zd4 to mcw8
Normal Pulling <invalid> kubelet Pulling image "nginx" 去node节点查看，原来起的是k8s_POD_mcw01dep-nginx这个，不是k8s_mcw01dep-nginx
既然主节点查看pod信息，拉取Nginx的年龄是无效 ，那么去node节点mcw8上直接手动拉取镜像
[root@mcw8 ~]$ docker pull nginx #镜像手动拉取成功
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest 再次查看pod详情
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m21s default-scheduler Successfully assigned default/mcw01dep-nginx-5dd785954d-65zd4 to mcw8
Normal Pulling <invalid> kubelet Pulling image "nginx" 看到第一行显示调度，也就是每个容器都有个同名的POD容器，那是个调度。来自默认调度，消息里还能看到pod部署到哪个节点了，
多次查看，我已经将mcw8节点拉取了镜像，但是它没认出来，也没有重新拉取，既然如此，我删掉pod，让它自动重建pod，从mcw8节点本地拉取镜像 查看pod，带有命名空间的显示年龄是无效的，也就是mcw8和9的网络存在问题，这个是不是要重新生成呢？这个网络是节点加入到集群时创建的
[machangwei@mcw7 ~]$ kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system kube-flannel-ds-tvz9q 0/1 CrashLoopBackOff 102 (<invalid> ago) 8h 10.0.0.138 mcw8 <none> <none>
kube-system kube-flannel-ds-v28gj 1/1 Running 102 (<invalid> ago) 8h 10.0.0.139 mcw9 <none> <none>
删除k8s系统的pod要指定命名空间
[machangwei@mcw7 ~]$ kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system kube-flannel-ds-tvz9q 0/1 CrashLoopBackOff 103 (<invalid> ago) 8h 10.0.0.138 mcw8 <none> <none>
kube-system kube-flannel-ds-v28gj 0/1 CrashLoopBackOff 102 (<invalid> ago) 8h 10.0.0.139 mcw9 <none> <none>
kube-system kube-flannel-ds-vjfkz 1/1 Running 0 8h 10.0.0.137 mcw7 <none> <none>
[machangwei@mcw7 ~]$ kubectl delete pod kube-flannel-ds-tvz9q
Error from server (NotFound): pods "kube-flannel-ds-tvz9q" not found
[machangwei@mcw7 ~]$ kubectl delete pod kube-flannel-ds-tvz9q --namespace=kube-system
pod "kube-flannel-ds-tvz9q" deleted
[machangwei@mcw7 ~]$ kubectl delete pod kube-flannel-ds-v28gj --namespace=kube-system
pod "kube-flannel-ds-v28gj" deleted
[machangwei@mcw7 ~]$ kubectl get pod --all-namespaces -o wide #没啥变化，还是无效的
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system kube-flannel-ds-gr7ck 0/1 CrashLoopBackOff 1 (<invalid> ago) 21s 10.0.0.138 mcw8 <none> <none>
kube-system kube-flannel-ds-m6qgl 1/1 Running 1 (<invalid> ago) 6s 10.0.0.139 mcw9 <none> <none>
kube-system kube-flannel-ds-vjfkz 1/1 Running 0 8h 10.0.0.137 mcw7 <none> <non
克隆虚拟机容器出各种问题，如果是创建的虚拟机没有这方面问题。
重新创建三个虚拟机，部署过程中遇到如下问题：coredns一直是peding，
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d8c4cb4d-nsv4x 0/1 Pending 0 8m59s
kube-system coredns-6d8c4cb4d-t7hr6 0/1 Pending 0 8m59s
排查过程：
查看错误信息：
[machangwei@mcwk8s-master ~]$ kubectl describe pod coredns-6d8c4cb4d-nsv4x -namespace=kube-system
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 21s (x7 over 7m9s) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate. 解决方案：
默认 k8s 不允许往 master 节点装东西，强行设置下允许：kubectl taint nodes --all node-role.kubernetes.io/master- [machangwei@mcwk8s-master ~]$ kubectl get nodes #查看节点，主节点未准备。执行如下命令，让主节点也作为一个node
NAME STATUS ROLES AGE VERSION
mcwk8s-master NotReady control-plane,master 16m v1.23.1
[machangwei@mcwk8s-master ~]$ kubectl taint nodes --all node-role.kubernetes.io/master-
node/mcwk8s-master untainted
[machangwei@mcwk8s-master ~]$ pod描述里有；
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/control-plane:NoSchedule
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s 允许master节点部署pod，使用命令如下:
kubectl taint nodes --all node-role.kubernetes.io/master-
禁止master部署pod
kubectl taint nodes k8s node-role.kubernetes.io/master=true:NoSchedule Jan 9 11:51:52 mcw10 kubelet: I0109 11:51:52.636701 25612 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Jan 9 11:51:53 mcw10 kubelet: E0109 11:51:53.909336 25612 kubelet.go:2347] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Jan 9 11:51:57 mcw10 kubelet: I0109 11:51:57.637836 25612 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d" [machangwei@mcwk8s-master ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
mcwk8s-master NotReady control-plane,master 43m v1.23.1
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d8c4cb4d-t24gx 0/1 Pending 0 18m
kube-system coredns-6d8c4cb4d-t7hr6 0/1 Pending 0 42m
结果发现跟之前的解决貌似没有关系，这是因为没有部署网络的原因，我部署好网络，dns的两个pod就好了
如下：
[machangwei@mcwk8s-master ~]$ kubectl apply -f mm.yml #部署网络
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[machangwei@mcwk8s-master ~]$ kubectl get nodes #查看节点还没有好
NAME STATUS ROLES AGE VERSION
mcwk8s-master NotReady control-plane,master 45m v1.23.1
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces #查看dns pod没有好，查看flannel初始化还没有好
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d8c4cb4d-t24gx 0/1 Pending 0 20m
kube-system coredns-6d8c4cb4d-t7hr6 0/1 Pending 0 44m
kube-system kube-flannel-ds-w8v9s 0/1 Init:0/2 0 14s [machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces #再次查看拉取镜像失败
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d8c4cb4d-t24gx 0/1 Pending 0 20m
kube-system coredns-6d8c4cb4d-t7hr6 0/1 Pending 0 45m
kube-system kube-flannel-ds-w8v9s 0/1 Init:ErrImagePull 0 45s [machangwei@mcwk8s-master ~]$ kubectl describe pod kube-flannel-ds-w8v9s --namespace=kube-system #查看描述信息
Warning Failed 4m26s kubelet Error: ErrImagePull #一直是拉取镜像失败，查看网络没有问题的
Warning Failed 4m25s kubelet Error: ImagePullBackOff #三分钟才拉取镜像成功
Normal BackOff 4m25s kubelet Back-off pulling image "quay.io/coreos/flannel:v0.15.1"
Normal Pulling 4m15s (x2 over 4m45s) kubelet Pulling image "quay.io/coreos/flannel:v0.15.1"
Normal Pulled 3m36s kubelet Successfully pulled image "quay.io/coreos/flannel:v0.15.1" in 39.090145025s
Normal Created 3m35s kubelet Created container install-cni
Normal Started 3m35s kubelet Started container install-cni
Normal Pulled 3m35s kubelet Container image "quay.io/coreos/flannel:v0.15.1" already present on machine
Normal Created 3m35s kubelet Created container kube-flannel
Normal Started 3m34s kubelet Started container kube-flannel 再次查看节点，已经是ready了，也就是说部署好网络，coredns才好，master节点作为一个node才ready
[machangwei@mcwk8s-master ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
mcwk8s-master Ready control-plane,master 57m v1.23.1
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d8c4cb4d-t24gx 1/1 Running 0 32m
kube-system coredns-6d8c4cb4d-t7hr6 1/1 Running 0 56m
kube-system etcd-mcwk8s-master 1/1 Running 0 57m
kube-system kube-apiserver-mcwk8s-master 1/1 Running 0 57m
kube-system kube-controller-manager-mcwk8s-master 1/1 Running 0 57m
kube-system kube-flannel-ds-w8v9s 1/1 Running 0 12m
kube-system kube-proxy-nvw6m 1/1 Running 0 56m
kube-system kube-scheduler-mcwk8s-master 1/1 Running 0 57m
node1上执行加入集群后，master上多出两个网络flannel没有ready的pod
是节点的网络，貌似不影响使用，暂时没有影响
[root@mcwk8s-node1 ~]$ kubeadm join 10.0.0.140:6443 --token 8yficm.352yz89c44mqk4y6 \
> --discovery-token-ca-cert-hash sha256:bcd36381d3de0adb7e05a12f688eee4043833290ebd39366fc47dd5233c552bf master上多出两个没有ready的pod,说明是node上的没有部署好这个网络pod呢
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-75npz 0/1 Init:1/2 0 99s
kube-system kube-flannel-ds-lpmxf 0/1 Init:1/2 0 111s
kube-system kube-flannel-ds-w8v9s 1/1 Running 0 16m [machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system kube-flannel-ds-75npz 0/1 CrashLoopBackOff 4 (50s ago) 4m37s 10.0.0.141 mcwk8s-node1 <none> <none>
kube-system kube-flannel-ds-lpmxf 0/1 Init:ImagePullBackOff 0 4m49s 10.0.0.142 mcwk8s-node2 <none> <none>
kube-system kube-flannel-ds-w8v9s 1/1 Running 0 19m 10.0.0.140 mcwk8s-master <none> <none> 查看nodes状态，现在已经有一个是ready了
[machangwei@mcwk8s-master ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
mcwk8s-master Ready control-plane,master 65m v1.23.1
mcwk8s-node1 Ready <none> 5m22s v1.23.1
mcwk8s-node2 NotReady <none> 5m35s v1.23.1 此时查看pod情况，虽然node已经ready了，但是网络的pod的状态，显示还是有点问题的
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system kube-flannel-ds-75npz 0/1 CrashLoopBackOff 5 (44s ago) 6m5s 10.0.0.141 mcwk8s-node1 <none> <none>
kube-system kube-flannel-ds-lpmxf 0/1 Init:ImagePullBackOff 0 6m17s 10.0.0.142 mcwk8s-node2 <none> <none>
kube-system kube-flannel-ds-w8v9s 1/1 Running 0 21m 10.0.0.140 mcwk8s-master <none> <none> 描述pod信息，查看CrashLoopBackOff这个状态，好像是重启容器失败，容器已经存在了
Normal Created 5m10s (x4 over 5m59s) kubelet Created container kube-flannel
Normal Started 5m10s (x4 over 5m58s) kubelet Started container kube-flannel
Warning BackOff 4m54s (x5 over 5m52s) kubelet Back-off restarting failed container
Normal Pulled 2m52s (x6 over 5m59s) kubelet Container image "quay.io/coreos/flannel:v0.15.1" already present on machine 描述pod信息，查看Init:ImagePullBackOff这个状态，是镜像拉取存在问题
Warning Failed 23s (x4 over 5m42s) kubelet Failed to pull image "quay.io/coreos/flannel:v0.15.1": rpc error: code = Unknown desc = context canceled
Warning Failed 23s (x4 over 5m42s) kubelet Error: ErrImagePull
镜像导入导出
建议：
可以依据具体使用场景来选择命令 若是只想备份images，使用save、load即可
若是在启动容器后，容器内容有变化，需要备份，则使用export、import 示例
docker save -o nginx.tar nginx:latest
或
docker save > nginx.tar nginx:latest
其中-o和>表示输出到文件，nginx.tar为目标文件，nginx:latest是源镜像名（name:tag） 示例
docker load -i nginx.tar
或
docker load < nginx.tar
其中-i和<表示从文件输入。会成功导入镜像及相关元数据，包括tag信息 示例
docker export -o nginx-test.tar nginx-test
其中-o表示输出到文件，nginx-test.tar为目标文件，nginx-test是源容器名（name） docker import nginx-test.tar nginx:imp
或
cat nginx-test.tar | docker import - nginx:imp 区别：
export命令导出的tar文件略小于save命令导出的 export命令是从容器（container）中导出tar文件，而save命令则是从镜像（images）中导出
基于第二点，export导出的文件再import回去时，无法保留镜像所有历史（即每一层layer信息，不熟悉的可以去看Dockerfile），不能进行回滚操作；而save是依据镜像来的，所以导入时可以完整保留下每一层layer信息。如下图所示，nginx:latest是save导出load导入的，nginx:imp是export导出import导入的。 原文链接：https://blog.csdn.net/ncdx111/article/details/79878098
Init:ImagePullBackOff这个状态的解决
查看node2上没有flannel镜像
[root@mcwk8s-node2 ~]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.23.1 b46c42588d51 3 weeks ago 112MB
rancher/mirrored-flannelcni-flannel-cni-plugin v1.0.0 cd5235cd7dc2 2 months ago 9.03MB
registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12 4 months ago 683kB 去主节点上导出一份镜像然后上传到node2上
[root@mcwk8s-master ~]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.15.1 e6ea68648f0c 8 weeks ago 69.5MB
[root@mcwk8s-master ~]$ docker save quay.io/coreos/flannel >mcwflanel-image.tar.gz
[root@mcwk8s-master ~]$ ls
anaconda-ks.cfg jiarujiqun.txt mcwflanel-image.tar.gz
[root@mcwk8s-master ~]$ scp mcwflanel-image.tar.gz 10.0.0.142:/root/ node2上导入镜像成功
[root@mcwk8s-node2 ~]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.23.1 b46c42588d51 3 weeks ago 112MB
rancher/mirrored-flannelcni-flannel-cni-plugin v1.0.0 cd5235cd7dc2 2 months ago 9.03MB
registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12 4 months ago 683kB
[root@mcwk8s-node2 ~]$ ls
anaconda-ks.cfg
[root@mcwk8s-node2 ~]$ ls
anaconda-ks.cfg mcwflanel-image.tar.gz
[root@mcwk8s-node2 ~]$ docker load < mcwflanel-image.tar.gz
ab9ef8fb7abb: Loading layer [==================================================>] 2.747MB/2.747MB
2ad3602f224f: Loading layer [==================================================>] 49.46MB/49.46MB
54089bc26b6b: Loading layer [==================================================>] 5.12kB/5.12kB
8c5368be4bdf: Loading layer [==================================================>] 9.216kB/9.216kB
5c32c759eea2: Loading layer [==================================================>] 7.68kB/7.68kB
Loaded image: quay.io/coreos/flannel:v0.15.1
[root@mcwk8s-node2 ~]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.23.1 b46c42588d51 3 weeks ago 112MB
quay.io/coreos/flannel v0.15.1 e6ea68648f0c 8 weeks ago 69.5MB
rancher/mirrored-flannelcni-flannel-cni-plugin v1.0.0 cd5235cd7dc2 2 months ago 9.03MB
registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12 4 months ago 683kB 主节点上查看pod状态，已经变化了，变成CrashLoopBackOff。重启了很多次
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-75npz 0/1 CrashLoopBackOff 9 (4m47s ago) 28m
kube-system kube-flannel-ds-lpmxf 0/1 CrashLoopBackOff 4 (74s ago) 28m
kube-system kube-flannel-ds-w8v9s 1/1 Running 0 43m
查看描述信息，重启失败。问题CrashLoopBackOff解决
[machangwei@mcwk8s-master ~]$ kubectl describe pod kube-flannel-ds-lpmxf --namespace=kube-system
Warning BackOff 3m25s (x20 over 7m48s) kubelet Back-off restarting failed container 虽然节点上的这两个一直不是ready，但是node状态已经是ready了，先不管了，部署一个应用验证一下
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-75npz 0/1 CrashLoopBackOff 12 (114s ago) 41m
kube-system kube-flannel-ds-lpmxf 0/1 CrashLoopBackOff 8 (3m46s ago) 41m
[machangwei@mcwk8s-master ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
mcwk8s-master Ready control-plane,master 100m v1.23.1
mcwk8s-node1 Ready <none> 40m v1.23.1
mcwk8s-node2 Ready <none> 41m v1.23.1 查看环境是否安装好了，已经没问题可以部署应用了
[machangwei@mcwk8s-master ~]$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
mcw01dep-nginx 1/1 1 1 5m58s
mcw02dep-nginx 1/2 2 1 71s
[machangwei@mcwk8s-master ~]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
mcw01dep-nginx-5dd785954d-z7s8m 1/1 Running 0 7m21s
mcw02dep-nginx-5b8b58857-7mlmh 1/1 Running 0 2m34s
mcw02dep-nginx-5b8b58857-pvwdd 1/1 Running 0 2m34s 把测试资源删掉，然后保存一份虚拟机快照，省的k8s环境变化，需要重新部署等，直接恢复快照就行。
[machangwei@mcwk8s-master ~]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
mcw01dep-nginx-5dd785954d-z7s8m 1/1 Running 0 7m21s
mcw02dep-nginx-5b8b58857-7mlmh 1/1 Running 0 2m34s
mcw02dep-nginx-5b8b58857-pvwdd 1/1 Running 0 2m34s
[machangwei@mcwk8s-master ~]$
[machangwei@mcwk8s-master ~]$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
mcw01dep-nginx 1/1 1 1 7m39s
mcw02dep-nginx 2/2 2 2 2m52s
[machangwei@mcwk8s-master ~]$ kubectl delete deployment mcw01dep-nginx mcw02dep-nginx
deployment.apps "mcw01dep-nginx" deleted
deployment.apps "mcw02dep-nginx" deleted
[machangwei@mcwk8s-master ~]$ kubectl get deployment
No resources found in default namespace.
[machangwei@mcwk8s-master ~]$
[machangwei@mcwk8s-master ~]$ kubectl get pod
No resources found in default namespace.
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
太卡，卡了哦半边天，虚拟机
系统或者网络占用过多CPU，造成内核软死锁（soft lockup）。Soft lockup名称解释：所谓，soft lockup就是说，这个bug没有让系统彻底死机，但是若干个进程（或者kernel thread）被锁死在了某个状态（一般在内核区域），很多情况下这个是由于内核锁的使用的问题。
虚拟机上k8s部署好的第二天用时总是出现的各种问题的更多相关文章
- Centos虚拟机上安装部署Tenginx，以及部署过程中遇到的问题
Tenginx下载网址: Tenginx 官网地址:http://tengine.taobao.org/ Tenginx的官方网址中可以阅读Nginx的文档,可以选择中文进行阅读.下载Tengine- ...
- mac虚拟机上(centos系统)设置联网第二种方式
这种方式简单,不容易出错,用的桥接的方式. 这样的安装的centos会得到ip地址 然后编辑一下网卡配置,使其变为静态得ip 输入命令 # vi /etc/sysconfig/network-scri ...
- 虚拟机上自动化部署(EFI网络安装)ESXi服务器遇到的问题
1,虚拟机的CPU为2核或以上. 2,虚拟机选项中不选择: 启用UEFI安全引导.3,引导延迟:设置3000毫秒或以上 4,#GP Exception 13 in world 1:unknown @ ...
- 在 Azure 虚拟机上部署反恶意软件解决方案
本博客文章由我和我的同事 Sung Hsueh 共同撰写,Sung Hsueh 是 Microsoft Azure 计算部负责安全事项的项目经理. 本博客文章为"虚拟机扩展程序"系 ...
- VM虚拟机上 实现CentOS 6.X下部署LVS(DR)+keepalived实现高性能高可用负载均衡
一.简介 LVS是Linux Virtual Server的简写,意即Linux虚拟服务器,是一个虚拟的服务器集群系统.本项目在1998年5月由章文嵩博士成立,是中国国内最早出现的自由软件项目之一. ...
- VM虚拟机上 实现CentOS 6.X下部署LVS(NAT)+keepalived实现高性能高可用负载均衡
一.简介 VS/NAT原理图: 二.系统环境 实验拓扑: 1 准备工作 a) 虚拟机上安装CentOS6.x操作系统,参考一下博客内容 http://www.cnblogs.com/seesea125 ...
- 【运维技术】VM虚拟机上使用centos7安装docker启动gogs服务教程【含B站视频教程】
VM虚拟机上使用centos7安装docker启动gogs服务视频教程 BiliBili视频教程链接飞机票,点我 使用VMware Workstation安装Centos7 MinMal系统 第一步: ...
- 在VM虚拟机上安装Microsoft Dynamics CRM 2016 步骤图解及安装注意事项
安装Dynamics CRM 2016环境配置要求: 系统版本:Windows Server 2012 R2(必须) SQL 版本: SQLServer2014SP1-FullSlipstream-x ...
- 在VirtualBox虚拟机上采集Fedora15系统
在VirtualBox虚拟机上采集Fedora15系统 第一部分:创建系统并磁盘分区 1.点击VirtualBox上的新建 2.添加名称,选择类型和版本,点下一步 3.写入内存(不要超过物理内存),点 ...
- 在虚拟机上配置linux lab的相关经验
最近一直在研究怎样在嵌入式开发板上移植linux嵌入式系统,但是不太想花费太多钱购买开发板.然后在网上搜索相关的arm模拟器.有qemu,skyeye,armulator等,在按照网上教程一步一步实践 ...
随机推荐
- SQL JOIN 子句：合并多个表中相关行的完整指南
SQL JOIN JOIN子句用于基于它们之间的相关列合并来自两个或更多表的行. 让我们看一下"Orders"表的一部分选择: OrderID CustomerID OrderDa ...
- Seaborn线性关系数据可视化
regplot() 绘制两个变量的线性拟合图. sns.regplot( x, y, data=None, x_estimator=None, x_bins=None, x_ci='ci', scat ...
- 设置 BCompare 打开文件时的默认字符编码
每次比对 .h .cpp 文件,BCompare总是默认以西欧字符编码打开文件,导致中文都变成乱码了,还需要手动的修改文件字符编码,这样才能正常的显示.非常的不方便 然后我们就需要设置默认的字符编码, ...
- node nvm使用
背景 node 经过了一次大的改变,直接从8到了10,差别很大,但是有的项目又需要用到8,这个时候不能完全升级. 所以我们需要一个管理虚拟环境的工具. 安装 https://github.com/co ...
- mysql 重新整理——性能下降的原因[四]
前言 什么是性能下降? 其实就是代码运行的环境变了,那么环境变化是什么? 比如cpu上升了,内存满了.有或者表中数量增加了,量变了. 其实这些是dba干的,但是呢,我们也需要去了解下,并且优化我们的c ...
- 国产GOWIN实现低成本实现CSI MIPI转换DVP
CSI MIPI转换DVP,要么就是通用IC操作,如龙讯芯片和索尼芯片,但是复杂的寄存器控制器实在开发太累.对于FPGA操作,大部分都是用xilinx的方案,xilinx方案成本太高,IP复杂. 而用 ...
- C#微服务必学清单
在 C# 领域,有一些不错的微服务书籍和开源框架,对于学习微服务相关知识非常有帮助.以下是一些建议您阅读的微服务书目和开源框架. 微服务书目: 1. <Building Microservice ...
- 力扣378(java&python)-有序矩阵中第 K 小的元素（中等）
题目: 给你一个 n x n 矩阵 matrix ,其中每行和每列元素均按升序排序,找到矩阵中第 k 小的元素.请注意,它是 排序后 的第 k 小元素,而不是第 k 个 不同 的元素. 你必须找到一个 ...
- 力扣599(java&python)- 两个列表的最小索引总和（简单）
题目: 假设 Andy 和 Doris 想在晚餐时选择一家餐厅,并且他们都有一个表示最喜爱餐厅的列表,每个餐厅的名字用字符串表示. 你需要帮助他们用最少的索引和找出他们共同喜爱的餐厅. 如果答案不止一 ...
- Dubbo-go-Mesh 开启新一代 Go 微服务形态
简介:Proxyless Service Mesh 能力将跟随 Dubbo-go 下一版本发布,稳定的性能需要社区成员们共同的关注与建设.在此基础之上,我们还会进一步探索轻量级 sdk + side ...