ETCD TLS 配置的坑

一、环境准备

环境总共 3 台虚拟机，系统为centos7，1个 master，2 个 etcd 节点，master 同时也作为 node 负载 pod，在分发证书等阶段将在另外一台主机上执行，该主机对集群内所有节点配置了 ssh 秘钥登录

IP	节点
192.168.0.153	master、node、etcd
192.168.0.154	master、node、etcd
192.168.0.164	master、node、etcd

由于 Etcd 和 Kubernetes 全部采用 TLS 通讯，所以先要生成 TLS 证书，证书生成工具采用 cfssl

二、证书相关处理

1、证书说明

证书名称	配置文件	用途
etcd-root-ca.pem	etcd-root-ca-csr.json	etcd 根 CA 证书
etcd.pem	etcd-gencert.json、etcd-csr.json	etcd 集群证书

2、CFSSL 工具安装

首先下载 cfssl，并给予可执行权限，然后扔到 PATH 目录下

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

Etcd 证书生成所需配置文件如下:

• etcd-root-ca-csr.json

• {
     "key": {
       "algo": "rsa",
       "size":
     },
     "names": [
       {
         "O": "etcd",
         "OU": "etcd Security",
         "L": "Beijing",
         "ST": "Beijing",
         "C": "CN"
       }
     ],
     "CN": "etcd-root-ca"
  }

• etcd-gencert.json

• {
    "signing": {
      "default": {
          "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
          ],
          "expiry": "87600h"
      }
    }
  }

• etcd-csr.json

• {
    "key": {
      "algo": "rsa",
      "size":
    },
    "names": [
      {
        "O": "etcd",
        "OU": "etcd Security",
        "L": "Beijing",
        "ST": "Beijing",
        "C": "CN"
      }
    ],
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "localhost",
      "192.168.0.153",
      "192.168.0.154",
      "192.168.0.164",
      "master",
      "node1",
      "node2"
    ]
  }

最后生成 Etcd 证书

cfssl gencert --initca=true etcd-root-ca-csr.json | cfssljson --bare etcd-root-ca
cfssl gencert --ca etcd-root-ca.pem --ca-key etcd-root-ca-key.pem --config etcd-gencert.json etcd-csr.json | cfssljson --bare etcd

生成的证书列表如下

三、部署 HA ETCD

安装前准备

关闭 selinux, setenforce 0

关闭防火墙， systemctl stop firewalld; iptables -F

ntpdate 时间同步

ntpdate time1.aliyun.com

1、安装 Etcd

以下操作都是在master节点上操作

ETCD 直接采用 rpm 安装，RPM 可以从 Fedora 官方仓库 获取 spec 文件自己 build，或者直接从 rpmFind 网站 搜索

#下载 rpm包
wget ftp://195.220.108.108/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/e/etcd-3.2.7-1.fc28.x86_64.rpm

#分发并安装
I="192.168.0.153 192.168.0.154 192.168.0.164"
for IP in $I; do
    scp etcd-3.2.-.fc28.x86_64.rpm root@$IP:~
    ssh root@$IP rpm -ivh etcd-3.2.-.fc28.x86_64.rpm
done

2、分发证书

I="192.168.0.153 192.168.0.154 192.168.0.164"

for IP in $I; do
    ssh root@$IP mkdir /etc/etcd/ssl/
    scp *.pem root@$IP:/etc/etcd/ssl/
    ssh root@$IP chown -R etcd:etcd /etc/etcd/ssl/
    ssh root@$IP chmod -R  /etc/etcd/
done

3、修改配置

rpm 安装好以后直接修改 /etc/etcd/etcd.conf 配置文件即可，其中单个节点配置如下(其他节点只是名字和 IP 不同)

# [member]
ETCD_NAME=etcd0
ETCD_DATA_DIR="/var/lib/etcd/etcd0.etcd"
ETCD_WAL_DIR="/var/lib/etcd/wal"
ETCD_SNAPSHOT_COUNT="100"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.0.153:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.153:2379,http://127.0.0.1:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
#ETCD_CORS=""

# [cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.153:2380"
# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd0=https://192.168.0.153:2380,etcd1=https://192.168.0.154:2380,etcd2=https://192.168.0.164:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.153:2379"

#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_STRICT_RECONFIG_CHECK="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"

# [proxy]
#ETCD_PROXY="off"/
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"

# [security]
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_PEER_AUTO_TLS="true"

# [logging]
#ETCD_DEBUG="false"
# examples for -log-package-levels etcdserver=WARNING,security=DEBUG
#ETCD_LOG_PACKAGE_LEVELS=""

node 节点要修改的地方：

ETCD_NAME
ETCD_LISTEN_PEER_URLS
ETCD_LISTEN_CLIENT_URLS
ETCD_INITIAL_ADVERTISE_PEER_URLS
ETCD_ADVERTISE_CLIENT_URLS

4、启动验证

配置修改后在每个节点进行启动即可，注意，Etcd 各个节点间必须保证时钟同步，否则会造成启动失败等错误

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

启动成功后验证节点状态

export ETCDCTL_API=3
etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.0.153:2379,https://192.168.0.154:2379,https://192.168.0.164:2379 endpoint health