配置安全证书的Etcd集群

不知在哪篇技术文档中看到，kubernetes master和etcd分开部署模式，因为集群的状态都保存在etcd中，这样当kubernetes master挂掉后，通过API Server交互的Scale等功能无法使用外，其他已经部署的Pod仍然能继续工作。

基于这种考虑，通过yum以及修改etcd.conf方式部署了一个三节点的etcd集群，但对于企业使用而言，虽然在局域网内访问，多数情况下还是需要配置安全证书，就好像很多政府部门因为三级等保的要求必须在weblogic中配置ssl一样，自己尝试在之前的环境中通过修改conf文件下配置，启动时遭遇各种问题失败，但同样的证书后修改为命令行方式配置后以及手工安装etcd后部署成功。记录如下:

• 安装cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

• 创建CA

# mkdir /root/ssl
# cd /root/ssl
# cfssl print-defaults config > ca-config.json
# cfssl print-defaults csr > ca-csr.json

修改ca-config.json

[root@etc0 ssl]# cat ca-config.json
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "kubernetes": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
          }
        }
    }
}

server auth表示client可以用该ca对server提供的证书进行验证

client auth表示server可以用该ca对client提供的证书进行验证

创建证书签名请求

[root@etc0 ssl]# cat ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size":
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

生成CA证书和私钥

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
# ls ca*
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem

• 创建kubernetes证书

[root@etc0 ssl]# cat kubernetes-csr.json
{
    "CN": "kubernetes",
    "hosts": [
        "127.0.0.1",
        "192.168.0.102",
        "192.168.0.103",
        "192.168.0.104",
        "192.168.0.105",
        "192.168.0.106",
        "10.254.0.1",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size":
    },
    "names": [
        {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "k8s",
        "OU": "System"
        }
    ]
}

可以看到该证书把etcd集群的所有ip,kubernetes master的所有ip以及kubernetes服务的ip(10.254.0.1)都加入进去了，这样他们都能使用同一个密钥

生成Kubernetes证书和密钥

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
# ls kuberntes*
kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem

• 分发证书文件

在每台etcd机器中运行

# mkdir -p /etc/kubernetes/ssl
# cp *.pem /etc/kubernetes/ssl

etcd集群配置

在/etc/hosts文件中加入地址

[root@etc0 ssl]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.102   etc0
192.168.0.103   etc1
192.168.0.104   etc2

• 手工下载和安装etcd

访问https://github.com/coreos/etcd/releases，我下载的是3.2.9版本

https://github.com/coreos/etcd/releases/download/v3.2.9/etcd-v3.2.9-linux-amd64.tar.gz

在每台机器上运行

tar -xvf etcd-v3.2.9-linux-amd64.tar.gz
mv etcd-v3.2.9-linux-amd64/etcd* /root/local/bin

mkdir -p /var/lib/etcd

建立一个etcd.service文件，内容如下（针对不同的etcd节点需要修改ip地址和name)

[root@etc0 ssl]# cat /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /root/local/bin/etcd --name=etc0 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls=https://192.168.0.102:2380 --listen-peer-urls=https://192.168.0.102:2380 --listen-client-urls=https://192.168.0.102:2379,https://127.0.0.1:2379 --advertise-client-urls=https://192.168.0.102:2379 --initial-cluster-token=etcd-cluster-0 --initial-cluster=\"etc0=https://192.168.0.102:2380,etc1=https://192.168.0.103:2380,etc2=https://192.168.0.104:2380\" --initial-cluster-state=new --data-dir=/var/lib/etcd"
Restart=on-failure
RestartSec=
LimitNOFILE=

[Install]
WantedBy=multi-user.target

　　 在每台etcd节点上启动etcd服务

# mv etcd.service /etc/systemd/system/
# systemctl daemon-reload
# systemctl enable etcd
# systemctl start etcd
# systemctl status etcd

[root@etc1 bin]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/etc/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu -- :: CST; 1h 20min ago
     Docs: https://github.com/coreos
 Main PID:  (etcd)
   CGroup: /system.slice/etcd.service
           └─ /root/local/bin/etcd --name=etc1 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem...

Oct  :: etc1 etcd[]: lost the TCP streaming connection with peer 70df68d0b37fcd43 (stream MsgApp v2 reader)
Oct  :: etc1 etcd[]: failed to dial 70df68d0b37fcd43 on stream Message (dial tcp 192.168.0.104:: getsockopt: connection refused)
Oct  :: etc1 etcd[]: peer 70df68d0b37fcd43 became inactive
Oct  :: etc1 etcd[]: peer 70df68d0b37fcd43 became active
Oct  :: etc1 etcd[]: closed an existing TCP streaming connection with peer 70df68d0b37fcd43 (stream MsgApp v2 writer)
Oct  :: etc1 etcd[]: established a TCP streaming connection with peer 70df68d0b37fcd43 (stream MsgApp v2 writer)
Oct  :: etc1 etcd[]: closed an existing TCP streaming connection with peer 70df68d0b37fcd43 (stream Message writer)
Oct  :: etc1 etcd[]: established a TCP streaming connection with peer 70df68d0b37fcd43 (stream Message writer)
Oct  :: etc1 etcd[]: established a TCP streaming connection with peer 70df68d0b37fcd43 (stream Message reader)
Oct  :: etc1 etcd[]: established a TCP streaming connection with peer 70df68d0b37fcd43 (stream MsgApp v2 reader)

如果出现任何错误，可以通过journalctl -xe去看到启动详情。

• 验证服务

[root@etc1 bin]# ./etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.0.102:2379,https://192.168.0.103:2379,https://192.168.0.104:2379 cluster-health
member 4701789a3a673ef5 is healthy: got healthy result from https://192.168.0.103:2379
member 70df68d0b37fcd43 is healthy: got healthy result from https://192.168.0.104:2379
member 90262c9df511cc4d is healthy: got healthy result from https://192.168.0.102:2379
cluster is healthy

如果不带证书访问，报错信息为

[root@etc1 bin]# ./etcdctl  --endpoints=https://192.168.0.102:2379,https://192.168.0.103:2379,https://192.168.0.104:2379 cluster-health
cluster may be unhealthy: failed to list members
Error:  client: etcd cluster is unavailable or misconfigured; error #: x509: certificate signed by unknown authority
; error #: x509: certificate signed by unknown authority
; error #: x509: certificate signed by unknown authority

error #: x509: certificate signed by unknown authority
error #: x509: certificate signed by unknown authority
error #: x509: certificate signed by unknown authority

• 目前的问题是：

为什么通过手工方式修改conf文件不成？