分布式kv存储系统之Etcd集群
etcd是什么？
etcd是一个高可用的分布式键值数据库，可用于服务发现，etcd采用 raft 一致性算法，基于 Go 语言实现。其特点有简单易用，所谓简单易用是指安装配置简单，提供http/https接口；安全，安全是指etcd支持ssl证书认证，支持集群各节点间使用对等证书认证；客户端和服务端的双向证书认证；可靠，可靠是指etcd使用raft协议实现分布式系统数据的可用性和一致性；etcd主要有两个版本v2和v3；v2和v3的api是互不兼容的，所以我们在同一服务器上安装多个版本的etcd时，我们需要用ETCDCTL_API这个环境变量指定；
etcd集群部署
环境准备
| 主机名称 | ip地址 |
| master01.k8s.org | 192.168.0.41 |
| master02.k8s.org | 192.168.0.42 |
| master03.k8s.org | 192.168.0.43 |
各主机hosts文件解析
[root@master01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.99 time.test.org time-node
192.168.0.41 master01 master01.k8s.org etcd01 etcd01.k8s.org
192.168.0.42 master02 master02.k8s.org etcd02 etcd02.k8s.org
192.168.0.43 master03 master03.k8s.org etcd03 etcd03.k8s.org
192.168.0.44 node01 node01.k8s.org
192.168.0.45 node02 node02.k8s.org
192.168.0.46 node03 node03.k8s.org
[root@master01 ~]#
关闭各主机的firewalld服务
[root@master01 ~]# systemctl stop firewalld
[root@master01 ~]# systemctl disable firewalld
[root@master01 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@master01 ~]#
各主机间时间同步
[root@master01 ~]# grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server time.test.org iburst
# Serve time even if not synchronized to any NTP server.
[root@master01 ~]# systemctl restart chronyd.service
[root@master01 ~]# systemctl status chronyd.service
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-01-30 15:41:25 CST; 11s ago
Docs: man:chronyd(8)
man:chrony.conf(5)
Process: 1411 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 1407 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 1409 (chronyd)
CGroup: /system.slice/chronyd.service
└─1409 /usr/sbin/chronyd Jan 30 15:41:25 master01.k8s.org systemd[1]: Stopped NTP client/server.
Jan 30 15:41:25 master01.k8s.org systemd[1]: Starting NTP client/server...
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SI...+DEBUG)
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: commandkey directive is no longer supported
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: generatecommandkey directive is no longer supported
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: Frequency -25.600 +/- 2.450 ppm read from /var/lib/chrony/drift
Jan 30 15:41:25 master01.k8s.org systemd[1]: Started NTP client/server.
Jan 30 15:41:29 master01.k8s.org chronyd[1409]: Selected source 192.168.0.99
Hint: Some lines were ellipsized, use -l to show in full.
[root@master01 ~]#
提示：集群内部可以使用自己搭建的时间服务，把chrony.conf中的server 指向对应时间服务器，然后重启chronyd即可；当然也可以使用互联网上公有的时间服务器；总之一个服务以集群方式工作，其时间同步是非常重要；
各主机间ssh 互信
[root@master01 ~]# ssh master02
Last login: Sat Jan 30 15:34:33 2021 from master01
[root@master02 ~]# exit
logout
Connection to master02 closed.
[root@master01 ~]# ssh master03
Last login: Sat Jan 30 15:34:37 2021 from master01
[root@master03 ~]# exit
logout
Connection to master03 closed.
[root@master01 ~]#
提示：有关ssh互信的配置请参考本人博客：https://www.cnblogs.com/qiuhom-1874/p/11783371.html；各主机间实现ssh互信，其主要目的是方便各组件同步文件；做好以上准备以后，我们就可以下载etcd二进制包进行etcd集群部署；这里需要说明一下，在centos7上的extras仓库中有etcd的rpm包，我们可以使用yum来安装；但是extras仓库中的版本不是最新的，要想使用最新的就需要到官方github仓库中下载最新版本的etcd二进制包进行部署；两种部署方式没有什么特别的不同；如果对版本要求不是特别新的环境中，建议使用yum安装；
下载etcd二进制包
[root@master01 ~]#wget https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz
--2021-01-30 15:46:18-- https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz
Resolving github.com (github.com)... 52.192.72.89
Connecting to github.com (github.com)|52.192.72.89|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/11225014/ad6a1d80-2f1a-11eb-8cb8-2f1ae35d5487?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210130T074619Z&X-Amz-Expires=300&X-Amz-Signature=47569782ddb8a1f70fbd28350433d3a045d22f040dd95b7de1055c96e7b4c359&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=11225014&response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.14-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-01-30 15:46:19-- https://github-releases.githubusercontent.com/11225014/ad6a1d80-2f1a-11eb-8cb8-2f1ae35d5487?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210130T074619Z&X-Amz-Expires=300&X-Amz-Signature=47569782ddb8a1f70fbd28350433d3a045d22f040dd95b7de1055c96e7b4c359&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=11225014&response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.14-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.111.154, 185.199.109.154, 185.199.108.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.111.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17373058 (17M) [application/octet-stream]
Saving to: ‘etcd-v3.4.14-linux-amd64.tar.gz’ 100%[=================================================================================================>] 17,373,058 24.9MB/s in 0.7s 2021-01-30 15:46:20 (24.9 MB/s) - ‘etcd-v3.4.14-linux-amd64.tar.gz’ saved [17373058/17373058]
[root@master01 ~]#
解压etcd二进制包
[root@master01 ~]# ls
etcd-v3.4.14-linux-amd64.tar.gz
[root@master01 ~]# tar xf etcd-v3.4.14-linux-amd64.tar.gz -C /usr/local/src/
[root@master01 ~]# cd /usr/local/src/
[root@master01 src]# ls
etcd-v3.4.14-linux-amd64
[root@master01 src]# cd etcd-v3.4.14-linux-amd64/
[root@master01 etcd-v3.4.14-linux-amd64]# ls
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
[root@master01 etcd-v3.4.14-linux-amd64]#
把etcd和etcdctl软连接至path环境变量下
[root@master01 etcd-v3.4.14-linux-amd64]# ls
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
[root@master01 etcd-v3.4.14-linux-amd64]# ln -s /usr/local/src/etcd-v3.4.14-linux-amd64/etcd /usr/bin/
[root@master01 etcd-v3.4.14-linux-amd64]# ln -s /usr/local/src/etcd-v3.4.14-linux-amd64/etcdctl /usr/bin/
[root@master01 etcd-v3.4.14-linux-amd64]# ll /usr/bin/etcd
lrwxrwxrwx 1 root root 44 Jan 30 15:59 /usr/bin/etcd -> /usr/local/src/etcd-v3.4.14-linux-amd64/etcd
[root@master01 etcd-v3.4.14-linux-amd64]# ll /usr/bin/etcdctl
lrwxrwxrwx 1 root root 47 Jan 30 15:59 /usr/bin/etcdctl -> /usr/local/src/etcd-v3.4.14-linux-amd64/etcdctl
[root@master01 etcd-v3.4.14-linux-amd64]#
编写etcd.service unit文件
[root@master01 ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target [Service]
Type=simple
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"
Type=notify [Install]
WantedBy=multi-user.target
[root@master01 ~]#
提供etcd环境变量加载文件/etc/etcd/etcd.conf文件
[root@master01 ~]# mkdir /etc/etcd/
[root@master01 ~]# cd /etc/etcd/
[root@master01 etcd]# vim etcd.conf
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
#ETCD_LISTEN_PEER_URLS="http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="default"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
#ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_INITIAL_CLUSTER="default=http://localhost:2380"
#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
#
#[Proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[Security]
#ETCD_CERT_FILE=""
#ETCD_KEY_FILE=""
#ETCD_CLIENT_CERT_AUTH="false"
#ETCD_TRUSTED_CA_FILE=""
#ETCD_AUTO_TLS="false"
#ETCD_PEER_CERT_FILE=""
#ETCD_PEER_KEY_FILE=""
#ETCD_PEER_CLIENT_CERT_AUTH="false"
#ETCD_PEER_TRUSTED_CA_FILE=""
#ETCD_PEER_AUTO_TLS="false"
#
#[Logging]
#ETCD_DEBUG="false"
#ETCD_LOG_PACKAGE_LEVELS=""
#ETCD_LOG_OUTPUT="default"
#
#[Unsafe]
#ETCD_FORCE_NEW_CLUSTER="false"
#
#[Version]
#ETCD_VERSION="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
#
#[Profiling]
#ETCD_ENABLE_PPROF="false"
#ETCD_METRICS="basic"
#
#[Auth]
#ETCD_AUTH_TOKEN="simple"
"etcd.conf" [New] 69L, 1686C written
[root@master01 etcd]#
更改etcd.conf文件
提示：ETCD_DATA_DIR用于指定etcd数据目录；ETCD_LISTEN_PEER_URLS用于指定集群节点通信监听的url地址；ETCD_LISTEN_CLIENT_URLS用户指定客户端连接使用时的url地址；ETCD_NAME用户指定当前节点etcd实例的名称；ETCD_INITIAL_ADVERTISE_PEER_URLS用于指定集群事务通告的url地址；ETCD_ADVERTISE_CLIENT_URLS用户指定客户端事务通告url地址；ETCD_INITIAL_CLUSTER用户指定集群成员，一个成员由成员名称=对应集群间通信的url地址，多个成员用逗号隔开；
创建/var/lib/etcd目录
[root@master01 etcd]# mkdir /var/lib/etcd/
[root@master01 etcd]# ll -d /var/lib/etcd/
drwxr-xr-x 2 root root 6 Jan 30 16:20 /var/lib/etcd/
[root@master01 etcd]#
复制master01上的/usr/bin/etcd和etcdctl二进制文件到master02和master03的/usr/bin/目录下
[root@master01 etcd]# scp /usr/bin/etcd /usr/bin/etcdctl master02:/usr/bin/
etcd 100% 23MB 43.6MB/s 00:00
etcdctl 100% 17MB 49.1MB/s 00:00
[root@master01 etcd]# scp /usr/bin/etcd /usr/bin/etcdctl master03:/usr/bin/
etcd 100% 23MB 42.2MB/s 00:00
etcdctl 100% 17MB 56.8MB/s 00:00
[root@master01 etcd]#
复制master01上的etcd.service到master02和master03的/usr/lib/systemd/system目录下
[root@master01 etcd]# scp /usr/lib/systemd/system/etcd.service master02:/usr/lib/systemd/system/
etcd.service 100% 417 165.1KB/s 00:00
[root@master01 etcd]# scp /usr/lib/systemd/system/etcd.service master03:/usr/lib/systemd/system/
etcd.service 100% 417 175.7KB/s 00:00
[root@master01 etcd]#
在master02和master03上创建/etc/etcd/目录和/var/lib/etcd/目录
[root@master01 etcd]# ssh master02 'mkdir /etc/etcd/ && mkdir /var/lib/etcd'
[root@master01 etcd]# ssh master03 'mkdir /etc/etcd/ && mkdir /var/lib/etcd'
[root@master01 etcd]#
复制master01上的etcd.conf文件到master02和master03的/etc/etcd/目录下
[root@master01 etcd]# scp /etc/etcd/etcd.conf master02:/etc/etcd/
etcd.conf 100% 1749 743.3KB/s 00:00
[root@master01 etcd]# scp /etc/etcd/etcd.conf master03:/etc/etcd/
etcd.conf 100% 1749 824.2KB/s 00:00
[root@master01 etcd]#
在master01/02/03上创建etcd用户
[root@master01 etcd]# useradd -r etcd -s /sbin/nologin
[root@master01 etcd]# getent passwd etcd
etcd:x:997:995::/home/etcd:/sbin/nologin
[root@master01 etcd]# ssh master02 'useradd -r etcd -s /sbin/nologin && getent passwd etcd'
etcd:x:997:995::/home/etcd:/sbin/nologin
[root@master01 etcd]# ssh master03 'useradd -r etcd -s /sbin/nologin && getent passwd etcd'
etcd:x:997:995::/home/etcd:/sbin/nologin
[root@master01 etcd]#
更改master01/02/03上的/var/lib/etcd目录的属主和属组为etcd用户
[root@master01 etcd]# chown etcd.etcd /var/lib/etcd/
[root@master01 etcd]# ll -d /var/lib/etcd/
drwxr-xr-x 2 etcd etcd 6 Jan 30 17:36 /var/lib/etcd/
[root@master01 etcd]# ssh master02 ''chown etcd.etcd /var/lib/etcd/ && ls -ld /var/lib/etcd/
drwxr-xr-x 2 etcd etcd 6 Jan 30 17:36 /var/lib/etcd/
[root@master01 etcd]# ssh master03 ''chown etcd.etcd /var/lib/etcd/ && ls -ld /var/lib/etcd/
drwxr-xr-x 2 etcd etcd 6 Jan 30 17:36 /var/lib/etcd/
[root@master01 etcd]#
修改master02上的/etc/etcd/etcd.conf文件
修改master03上的/etc/etcd/etcd.conf文件
到此三个节点的配置文件和相关用户以及目录都准备就绪，接下来我们要重载systemd的配置文件，加载etcd.service文件
[root@master01 ~]# systemctl daemon-reload
[root@master01 ~]# ssh master02 'systemctl daemon-reload'
[root@master01 ~]# ssh master03 'systemctl daemon-reload'
[root@master01 ~]#
启动etcd
[root@master01 ~]# systemctl start etcd
[root@master01 ~]#
提示：在每个节点上运行上述命令启动etcd；第一个启动的节点将阻塞，原因是etcd以集群方式工作，它必须要有足够的得票才能正常工作，如果集群节点有3个，那么至少有两个节点正常启动etcd才能正常工作；
验证：查看各节点的2379和2380端口是否都处于监听？
[root@master01 ~]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.0.41:2379 *:*
LISTEN 0 128 192.168.0.41:2380 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
[root@master01 ~]# ssh master02 'ss -tnl'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.0.42:2379 *:*
LISTEN 0 128 192.168.0.42:2380 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
[root@master01 ~]# ssh master03 'ss -tnl'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.0.43:2379 *:*
LISTEN 0 128 192.168.0.43:2380 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
[root@master01 ~]#
验证：使用etcdctl查看集群状态
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 endpoint status
192.168.0.41:2379, b8b747c74aaea686, 3.4.14, 4.5 MB, true, false, 13, 2163, 2163,
192.168.0.42:2379, b3504381e8ba3cb, 3.4.14, 4.5 MB, false, false, 13, 2163, 2163,
192.168.0.43:2379, f572fdfc5cb68406, 3.4.14, 4.5 MB, false, false, 13, 2163, 2163,
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 member list
b3504381e8ba3cb, started, etcd02, http://etcd02:2380, http://etcd02:2379, false
b8b747c74aaea686, started, etcd01, http://etcd01:2380, http://etcd01:2379, false
f572fdfc5cb68406, started, etcd03, http://etcd03:2380, http://etcd03:2379, false
[root@master01 ~]#
提示：能够列出集群成员和查看集群成员状态，就表示etcd集群工作已经正常；
验证：向etcd任意节点写数据，看看是否能够正常写？
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 put name "test"
OK
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 get name
name
test
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 del name
1
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379 get name
[root@master01 ~]#
提示：使用etcdctl工具可以正常向etcd集群写入数据；
为etcd集群生成证书
在某一节点上安装git工具
[root@master01 ~]# yum install git -y
克隆生成证书的脚本工具
[root@master01 ~]# git clone https://github.com/iKubernetes/k8s-certs-generator.git
Cloning into 'k8s-certs-generator'...
remote: Enumerating objects: 58, done.
remote: Total 58 (delta 0), reused 0 (delta 0), pack-reused 58
Unpacking objects: 100% (58/58), done.
[root@master01 ~]# ls
etcd-v3.4.14-linux-amd64.tar.gz k8s-certs-generator
[root@master01 ~]# cd k8s-certs-generator/
[root@master01 k8s-certs-generator]# ls
etcd-certs-gen.sh gencerts.sh k8s-certs-gen.sh openssl.conf README.md
[root@master01 k8s-certs-generator]#
使用gencerts.sh脚本生成etcd所需证书
[root@master01 k8s-certs-generator]# sh gencerts.sh -h
Usage: ./gencerts.sh etcd|k8s
[root@master01 k8s-certs-generator]# sh gencerts.sh etcd
Enter Domain Name [ilinux.io]: k8s.org
Generating RSA private key, 4096 bit long modulus
.......++
.................................................................................................................................................................................................................................................................++
e is 65537 (0x10001)
Generating RSA private key, 2048 bit long modulus
.................................................+++
.........................+++
e is 65537 (0x10001)
Generating etcd/pki/peer.csr
Generating RSA private key, 2048 bit long modulus
...........................................................................................................................................+++
...............+++
e is 65537 (0x10001)
Generating etcd/pki/server.csr
Generating RSA private key, 2048 bit long modulus
..............................................................+++
............................+++
e is 65537 (0x10001)
Generating etcd/pki/apiserver-etcd-client.csr
Generating RSA private key, 2048 bit long modulus
............+++
.................................+++
e is 65537 (0x10001)
Generating etcd/pki/client.csr
Generating etcd/pki/peer.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4096 (0x1000)
Validity
Not Before: Jan 30 10:46:52 2021 GMT
Not After : Jan 28 10:46:52 2031 GMT
Subject:
commonName = etcd
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Server
Netscape Comment:
OpenSSL Generated Server Certificate
X509v3 Subject Key Identifier:
FC:BA:D7:73:4E:C7:1D:9D:73:12:E3:60:96:5B:69:58:CE:4F:14:FD
X509v3 Authority Key Identifier:
keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
DirName:/CN=etcd-ca
serial:BE:88:C0:B5:81:5D:6D:D6 X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name:
DNS:*.k8s.org
Certificate is to be certified until Jan 28 10:46:52 2031 GMT (3650 days) Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/server.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4097 (0x1001)
Validity
Not Before: Jan 30 10:46:53 2021 GMT
Not After : Jan 28 10:46:53 2031 GMT
Subject:
commonName = etcd
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Server
Netscape Comment:
OpenSSL Generated Server Certificate
X509v3 Subject Key Identifier:
1C:BE:22:C0:B7:5F:03:39:5C:E0:FC:47:88:8D:3A:FC:27:FA:0E:BC
X509v3 Authority Key Identifier:
keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
DirName:/CN=etcd-ca
serial:BE:88:C0:B5:81:5D:6D:D6 X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:*.k8s.org
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days) Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/apiserver-etcd-client.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4098 (0x1002)
Validity
Not Before: Jan 30 10:46:53 2021 GMT
Not After : Jan 28 10:46:53 2031 GMT
Subject:
commonName = etcd
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Client
Netscape Comment:
OpenSSL Generated Client Certificate
X509v3 Subject Key Identifier:
FD:52:EA:9F:84:72:35:46:9A:33:71:DE:D0:41:E6:8D:89:C0:62:AE
X509v3 Authority Key Identifier:
keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days) Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/client.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4099 (0x1003)
Validity
Not Before: Jan 30 10:46:53 2021 GMT
Not After : Jan 28 10:46:53 2031 GMT
Subject:
commonName = etcd
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Client
Netscape Comment:
OpenSSL Generated Client Certificate
X509v3 Subject Key Identifier:
6B:31:50:84:00:9E:0F:6E:B8:56:7A:C1:57:82:F4:BB:12:57:52:B2
X509v3 Authority Key Identifier:
keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days) Write out database with 1 new entries
Data Base Updated
[root@master01 k8s-certs-generator]# ls
etcd etcd-certs-gen.sh gencerts.sh k8s-certs-gen.sh openssl.conf README.md
[root@master01 k8s-certs-generator]# ls etcd
patches pki
[root@master01 k8s-certs-generator]# ls etcd/pki/
apiserver-etcd-client.crt apiserver-etcd-client.key ca.crt ca.key client.crt client.key peer.crt peer.key server.crt server.key
[root@master01 k8s-certs-generator]#
提示：server.crt和server.key用于etcd服务端的证书和密钥；peer.crt和peer.key用于集群内部各节间认证所需证书和密钥；client.crt和client.key用于客户端连接服务端所需的证书和密钥；ca.crt和ca.key是用于集群内部做认证和客户端连接服务端所信任的ca的证书和密钥；
复制证书文件到其他节点
[root@master01 k8s-certs-generator]# cp -a etcd/pki/ /etc/etcd/
[root@master01 k8s-certs-generator]# cd /etc/etcd/
[root@master01 etcd]# ls
etcd.conf pki
[root@master01 etcd]# scp -r pki/ master02:/etc/etcd/
ca.key 100% 3247 1.8MB/s 00:00
ca.crt 100% 1814 1.2MB/s 00:00
peer.key 100% 1679 1.1MB/s 00:00
server.key 100% 1679 1.2MB/s 00:00
apiserver-etcd-client.key 100% 1675 1.3MB/s 00:00
client.key 100% 1675 1.1MB/s 00:00
peer.crt 100% 1659 75.0KB/s 00:00
server.crt 100% 1647 917.8KB/s 00:00
apiserver-etcd-client.crt 100% 1570 1.2MB/s 00:00
client.crt 100% 1570 902.2KB/s 00:00
[root@master01 etcd]# scp -r pki/ master03:/etc/etcd/
ca.key 100% 3247 1.1MB/s 00:00
ca.crt 100% 1814 695.0KB/s 00:00
peer.key 100% 1679 621.6KB/s 00:00
server.key 100% 1679 657.1KB/s 00:00
apiserver-etcd-client.key 100% 1675 950.4KB/s 00:00
client.key 100% 1675 1.0MB/s 00:00
peer.crt 100% 1659 916.3KB/s 00:00
server.crt 100% 1647 1.0MB/s 00:00
apiserver-etcd-client.crt 100% 1570 850.8KB/s 00:00
client.crt 100% 1570 872.7KB/s 00:00
[root@master01 etcd]#
配置etcd基于https协议提供服务
配置master01上的etcd启用证书认证
[root@master01 etcd]# cat etcd.conf
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
#ETCD_WAL_DIR=""
ETCD_LISTEN_PEER_URLS="http://192.168.0.41:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.41:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="etcd01"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd01:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd01:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
#
#[Proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[Security]
ETCD_CERT_FILE="/etc/etcd/pki/server.crt"
ETCD_KEY_FILE="/etc/etcd/pki/server.key"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/ca.crt"
ETCD_AUTO_TLS="false"
ETCD_PEER_CERT_FILE="/etc/etcd/pki/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/ca.crt"
ETCD_PEER_AUTO_TLS="false"
#
#[Logging]
#ETCD_DEBUG="false"
#ETCD_LOG_PACKAGE_LEVELS=""
#ETCD_LOG_OUTPUT="default"
#
#[Unsafe]
#ETCD_FORCE_NEW_CLUSTER="false"
#
#[Version]
#ETCD_VERSION="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
#
#[Profiling]
#ETCD_ENABLE_PPROF="false"
#ETCD_METRICS="basic"
#
#[Auth]
#ETCD_AUTH_TOKEN="simple"
[root@master01 etcd]#
提示：ETCD_CERT_FILE用于指定etcd服务端证书文件路径；ETCD_KEY_FILE用户指定服务端证书文件所对应的密钥文件路径；ETCD_CLIENT_CERT_AUTH用户指定是否启用客户端证书认证；ETCD_TRUSTED_CA_FILE用户指定客户端认证信任的ca证书文件；
ETCD_AUTO_TLS用于指定是否自动生成证书文件；ETCD_PEER_CERT_FILE用于指定集群间对等证书文件路径；ETCD_PEER_KEY_FILE用于指定集群间对等证书对应的密钥文件；ETCD_PEER_CLIENT_CERT_AUTH用于指定是否启用对等证书认证；ETCD_PEER_TRUSTED_CA_FILE用于指定对等证书认证所信赖的ca证书；ETCD_PEER_AUTO_TLS用于指定是否自动生成对等证书；
修改/etc/etcd/etcd.conf文件，将etcd01修改成etcd01.k8s.org，将etcd02修改成etcd02.k8s.org，将etcd03修改成etcd03.k8s.org，将http修改成https
停止etcd服务，删除/var/lib/etcd/目录下的所有文件
[root@master01 etcd]# systemctl stop etcd
[root@master01 etcd]# rm -rf /var/lib/etcd/*
[root@master01 etcd]# ll /var/lib/etcd/
total 0
[root@master01 etcd]#
配置master02启用证书认证，并将对应http修改成很https，把对应短格式名称修改为类似etcd01.k8s.org名称
停止etcd服务，删除/var/lib/etcd/目录下的所有文件
[root@master02 ~]# systemctl stop etcd
[root@master02 ~]# rm -rf /var/lib/etcd/*
[root@master02 ~]# ll /var/lib/etcd/
total 0
[root@master02 ~]#
配置master03启用证书认证，并将对应http修改成很https，把对应短格式名称修改为长格式名称
停止etcd服务，删除/var/lib/etcd下的所有文件
[root@master03 ~]# systemctl stop etcd
[root@master03 ~]# rm -rf /var/lib/etcd/*
[root@master03 ~]# ll /var/lib/etcd/
total 0
[root@master03 ~]#
启动各节点上的etcd
[root@master01 etcd]# systemctl start etcd
[root@master01 etcd]#
提示：如果三个节点上的etcd都能正常启动，说明我们配置文件没有问题；
验证：查看所有节点的etcd服务是否都正常启动，并监听对应的的端口？
[root@master01 etcd]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.0.41:2379 *:*
LISTEN 0 128 192.168.0.41:2380 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
[root@master01 etcd]# ssh master02 'ss -tnl'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.0.42:2379 *:*
LISTEN 0 128 192.168.0.42:2380 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
[root@master01 etcd]# ssh master03 'ss -tnl'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.0.43:2379 *:*
LISTEN 0 128 192.168.0.43:2380 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
[root@master01 etcd]#
验证：查看集群成员
[root@master01 etcd]# etcdctl --endpoints="https://etcd01.k8s.org:2379,https://etcd02.k8s.org:2379,https://etcd03.k8s.org:2379" --cacert="/etc/etcd/pki/ca.crt" --cert="/etc/etcd/pki/client.crt" --key="/etc/etcd/pki/client.key" endpoint status
https://etcd01.k8s.org:2379, 61d91b7ed8f88f32, 3.4.14, 20 kB, true, false, 6, 9, 9,
https://etcd02.k8s.org:2379, ef13441fdfe8af38, 3.4.14, 20 kB, false, false, 6, 9, 9,
https://etcd03.k8s.org:2379, f11ed09b6567910f, 3.4.14, 20 kB, false, false, 6, 9, 9,
[root@master01 etcd]# etcdctl --endpoints="https://etcd01.k8s.org:2379" --cacert="/etc/etcd/pki/ca.crt" --cert="/etc/etcd/pki/client.crt" --key="/etc/etcd/pki/client.key" member list
61d91b7ed8f88f32, started, etcd01.k8s.org, https://etcd01.k8s.org:2380, https://etcd01.k8s.org:2379, false
ef13441fdfe8af38, started, etcd02.k8s.org, https://etcd02.k8s.org:2380, https://etcd02.k8s.org:2379, false
f11ed09b6567910f, started, etcd03.k8s.org, https://etcd03.k8s.org:2380, https://etcd03.k8s.org:2379, false
[root@master01 etcd]#
提示：现在etcd启用了ssl认证功能，客户端访问必须携带对应的客户端证书和私钥文件以及对应认证所信任的ca证书，才可以正常访问到etcd集群；这里需要注意指定的endpoints需要使用域名格式给出，给定ip地址是无法正常通过认证的；
分布式kv存储系统之Etcd集群的更多相关文章
- etcd集群部署
etcd是用于共享配置和服务发现的分布式KV存储系统,随着CoreOS和Kubernetes等项目在开源社区日益火热,它们都用到了etcd组件作为一个高可用.强一致性的服务发现存储仓库.操作系统版本: ...
- Kubernetes集群搭建之Etcd集群配置篇
介绍 etcd 是一个分布式一致性k-v存储系统,可用于服务注册发现与共享配置,具有以下优点. 简单 : 相比于晦涩难懂的paxos算法,etcd基于相对简单且易实现的raft算法实现一致性,并通过g ...
- k8s1.13.0二进制部署-ETCD集群（一）
Kubernetes集群中主要存在两种类型的节点:master.minion节点. Minion节点为运行 Docker容器的节点,负责和节点上运行的 Docker 进行交互,并且提供了代理功能.Ma ...
- etcd集群安装
etcd 是一个分布式一致性k-v存储系统,可用于服务注册发现与共享配置,具有以下优点:1.简单:相比于晦涩难懂的paxos算法,etcd基于相对简单且易实现的raft算法实现一致性,并通过gRPC提 ...
- Kubernetes后台数据库etcd：安装部署etcd集群，数据备份与恢复
目录 一.系统环境 二.前言 三.etcd数据库 3.1 概述 四.安装部署etcd单节点 4.1 环境介绍 4.2 配置节点的基本环境 4.3 安装部署etcd单节点 4.4 使用客户端访问etcd ...
- Centos7下Etcd集群搭建
一.简介 "A highly-available key value store for shared configuration and service discovery." ...
- Centos7环境下etcd集群的搭建
Centos7环境下etcd集群的搭建 一.简介 "A highly-available key value store for shared configuration and servi ...
- 【转】centos7 搭建etcd集群
转自http://www.cnblogs.com/zhenyuyaodidiao/p/6237019.html 一.简介 “A highly-available key value store for ...
- Kubernetes学习之路（二）之ETCD集群二进制部署
ETCD集群部署 所有持久化的状态信息以KV的形式存储在ETCD中.类似zookeeper,提供分布式协调服务.之所以说kubenetes各个组件是无状态的,就是因为其中把数据都存放在ETCD中.由于 ...
随机推荐
- sql模糊查询和根据日期筛选
<!-- 查询统计列表 --> >= 小于等于 <=大于等于 将字符类型转换成日期格式进行比较 select * from xy_platformMessage ...
- Lombok注解-@SneakyThrows
@SneakyThrows注解的用途得从java的异常设计体系说起. java中我们常见的2类异常. 1.普通Exception类,也就是我们常说的受检异常或者Checked Exception. 2 ...
- mysql存储过程定义者
1. 执行update mysql.proc set DEFINER='root@%' WHERE NAME='p_update_rim_batch_log' AND db='otherdataonl ...
- 线程上下文类加载器（Context ClassLoader）
1.线程上下文类加载器是从jdk1.2开始引入的,类Thread中的getContextClassLoader()与setContextClassLoader(ClassLoader c1),分别用来 ...
- Node项目模板管理脚手架ptm-cli开发
目录 一.ptm-cli 使用说明 1.特点 2.安装 3.使用 1)基础帮助命令 2)添加模板/项目 3)编辑模板/项目 4)查看模板/项目 5)删除模板/项目 6)基于模板新建/初始化项目 二 p ...
- Erlang那些事儿第3回之我是函数(fun),万物之源MFA
Erlang代码到处都是模式匹配,这把屠龙刀可是Erlang的看家本领.独家绝学,之前在<Erlang那些事儿第1回之我是变量,一次赋值永不改变>文章提到过,Erlang一切皆是模式匹配. ...
- 记一次Jquery学习引发的学习思考
学习廖雪峰Jquery的教程关于表单的操作时,被最后的习题给困住了,在一番思索后无奈地决定去看博客评论下的答案,却发现自己看不懂,遂以为是对Jquery的了解还不够深,于是便在网上疯狂搜索关于Jque ...
- linux系统修改不成功无法修改密码
一.问题描述 新上架的浪潮服务器使用装机平台进行统一安装,安装完成后修改用户密码时统一无法修改,使用root账户无法修改其他用户密码,自身根密码也无法修改成功,报错如下 Changing passwo ...
- VsCode/Pycharm配合python env 使用
前言 用惯了vscode,这几天试了一下pycharm,还是回来了. pycharm一个好处就是python env 环境支持的比较好, vscode虽然也支持但是要改一些东西 env的使用查看我的上 ...
- 【剑指 Offer】06.从尾到头打印链表
题目描述 输入一个链表的头节点,从尾到头反过来返回每个节点的值(用数组返回). 示例 1: 输入:head = [1,3,2] 输出:[2,3,1] 限制: 0 <= 链表长度 <= 10 ...