k8s集群证书过期（kubeadm 1.10.2 ）

1、k8s 集群架构描述

• kubeadm v1.10.2创建k8s集群。
• master节点高可用，三节点（10.18.60.3、10.18.60.4、10.18.60.5）。
• LVS实现master三节点代理。

2、K8S集群证书过期，日志报错如下

Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid

3、故障排定，查看证书

# openssl x509 -noout -text -in apiserver-kubelet-client.crt | grep Not
            Not Before: May 22 01:58:06 2018 GMT
            Not After : May 22 01:58:07 2019 GMT　　　　# 2019-5-22日过期

　　

4、kubeadm 命令介绍（v1.10.2）　　

注：本次只需用到以下两个参数命令，其它参数不做介绍

# kubeadm alpha phase certs -h　　　　　　　　# 创建证书
Usage:
  kubeadm alpha phase certs [command]
Available Commands:
  all                      Generates all PKI assets necessary to establish the control plane
  apiserver                Generates an API server serving certificate and key
  apiserver-etcd-client    Generates a client certificate for the API server to connect to etcd securely
  apiserver-kubelet-client Generates a client certificate for the API server to connect to the kubelets securely
  ca                       Generates a self-signed kubernetes CA to provision identities for components of the cluster
  etcd-ca                  Generates a self-signed CA to provision identities for etcd
  etcd-healthcheck-client  Generates a client certificate for liveness probes to healthcheck etcd
  etcd-peer                Generates an etcd peer certificate and key
  etcd-server              Generates an etcd serving certificate and key
  front-proxy-ca           Generates a front proxy CA certificate and key for a Kubernetes cluster
  front-proxy-client       Generates a front proxy CA client certificate and key for a Kubernetes cluster
  sa                       Generates a private key for signing service account tokens along with its public key

# kubeadm alpha phase kubeconfig -h　　　　# 生成配置文件（例如：admin.conf|controller-manager.conf|kubelet.conf|scheduler.conf）
Usage:
  kubeadm alpha phase kubeconfig [command]
Available Commands:
  admin              Generates a kubeconfig file for the admin to use and for kubeadm itself
  all                Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
  controller-manager Generates a kubeconfig file for the controller manager to use
  kubelet            Generates a kubeconfig file for the kubelet to use. Please note that this should be used *only* for bootstrapping purposes.
  scheduler          Generates a kubeconfig file for the scheduler to use
  user               Outputs a kubeconfig file for an additional user

# kubeadm alpha phase certs apiserver -h
      --apiserver-advertise-address string   填写本机apiserver ip。
      --apiserver-cert-extra-sans strings    多master节点，在创建apiserver证书时，需要指定每个节点的IP，代理IP、域名。
      --cert-dir string                      The path where to save the certificates (default "/etc/kubernetes/pki")
      --config string                        Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
      -h, --help                                 help for apiserver
      --service-cidr string                  Alternative range of IP address for service VIPs, from which derives the internal API server VIP that will be added to the API Server serving cert (default "10.96.0.0/12")
      --service-dns-domain string            Alternative domain for services, to use for the API server serving cert (default "cluster.local")　　

5、备份节点配置文件与证书

# cp -rfp /etc/kubernetes /etc/kubernetes.2019.5.23　　

6、创建证书

注：

    1、因为之前三个master节点的配置文件中全部填写的是LVS VIP（没做域名解析），为了今后切换方便给VIP配置了一个域名，而apiserver证书中没有配置该域名的认证，所以利用openssl对apiserver证书做了重签替换，并设置apiserver证书有效期10年，所以这次证书过期不涉及apiserver，只需要对apiserver-kubelet-client与front-proxy-client证书重新创建，而下边给出了对apiserver证书重签的命令。

　2、创建证书时需要配置VPN，kubeadm需要连接国外

# kubeadm alpha phase certs apiserver --apiserver-advertise-address 10.18.60.3 --apiserver-cert-extra-sans 10.18.60.4 --apiserver-cert-extra-sans 10.18.60.5 --apiserver-cert-extra-sans 10.16.60.6 --apiserver-cert-extra-sans k8s.m.api   # 创建apiserver证书
# kubeadm alpha phase certs apiserver-kubelet-client				  # 创建apiserver-kubelet-client证书
# kubeadm alpha phase certs front-proxy-client					  # 创建front-proxy-client证书

7、创建配置文件（admin.conf|controller-manager.conf|kubelet.conf|scheduler.conf）

# 创建完会看到/etc/kubernetes下面出现了配置文件
# kubeadm alpha phase kubeconfig all --apiserver-advertise-address 10.18.60.3　

　　

8、准备替换

因为我三节点的代理IP配置了域名，所以需要做下替换。

# sed -i 's/10\.18\.60\.3/k8s.m.api/g' admin.conf
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' controller-manager.conf
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' scheduler.conf
# sed -i 's/10\.5\.38\.39/k8s.m.api/g' kubelet.conf
# grep 'host:' /etc/kubernetes/manifests/kube-apiserver.yaml
        host: k8s.m.api

　

# 直接覆盖（注意文件权限）
# cp -rfp /etc/kubernetes/admin.conf ~/.kube/config

　　　　

# kubelet 客户端签发的不需要备份
# rm -rf /var/lib/kubelet/pki/*

　　

9、重启服务

# 重启本机所有docker容器
# docker restart $(docker ps -q)

# 重启kubelet
# systemctl restart kubelet.service

　　

10、验证

# 可以看到已经恢复
# kubectl get node

　　

11、恢复其它master节点

# 注意拷贝kubelet.conf文件到其它服务器（其它服务器自己生成）
# scp admin.conf  controller-manager.conf  scheduler.conf 10.18.60.4:/etc/kubernetes
# scp admin.conf  controller-manager.conf  scheduler.conf 10.18.60.5:/etc/kubernetes
# scp apiserver-kubelet-client.crt  apiserver-kubelet-client.key front-proxy-client.crt  front-proxy-client.key 10.18.60.4:/etc/kubernetes/pki
# scp apiserver-kubelet-client.crt  apiserver-kubelet-client.key front-proxy-client.crt  front-proxy-client.key 10.18.60.5:/etc/kubernetes/pki

# 其它两个节点只生成各自的kubelet配置文件
# kubeadm alpha phase kubeconfig kubelet
# grep 'server:' kubelet.conf
    server: https://k8s.m.api:6443

　　　

# 直接覆盖（注意文件权限）
# cp -rfp /etc/kubernetes/admin.conf ~/.kube/config

　　　　

# kubelet 客户端签发的不需要备份
# rm -rf /var/lib/kubelet/pki/*

# 重启本机所有docker容器
# docker restart $(docker ps -q)

# 重启kubelet
# systemctl restart kubelet.service

　　

# 可以看到已经恢复
# kubectl get node