centos7下Etcd3集群搭建

一、环境介绍

etcd主要功能是分布式的存储键值，优点不多说了，分布是集群，自动选举等等，自行百度，主要说下配置方法，折腾了几天，终于优点眉目了，记录下操作方法，本文参考了如下链接

https://www.cnblogs.com/breg/p/5728237.html

http://www.361way.com/etcd-cluster/5468.html

https://n40lab.wordpress.com/2016/08/01/installing-coreos-etcd-server-on-centos-7/

创建了4个虚拟机，虚拟机中系统为centos7（虚拟机使用的docker）

for((i=1,i<5,i++))
do
    docker run -d --name etcd$i --hostname=etcd$i --privileged=true centos7 /usr/sbin/init
    docker exec -it etcd$i bash -c “yum update -y&& yum install net-tools -y”
done

hostname    ip
etcd1       192.168.0.5
etcd2       192.168.0.6
etcd3       192.168.0.7
etcd4       192.168.0.8
将上述ip表添加到/etc/hosts中

二、etcd安装与配置

2.1 yum安装etcd

yum install etcd -y

验证etcd是否可用

# systemctl enable etcd&&systemctl start etcd

# etcdctl set hello world

world

# etcdctl get hello

world

这里的集群模式是指完全集群模式，当然也可以在单机上通过不同的端口，部署伪集群模式，只是那样做只适合测试环境，生产环境考虑到可用性的话需要将etcd实例分布到不同的主机上，这里集群搭建有三种方式，分布是静态配置，etcd发现，dns发现。默认配置运行etcd，监听本地的2379端口，用于与client端交互，监听2380用于etcd内部交互。etcd启动时，集群模式下会用到的参数如下：

–name

etcd集群中的节点名，这里可以随意，可区分且不重复就行

–listen-peer-urls

监听的用于节点之间通信的url，可监听多个，集群内部将通过这些url进行数据交互(如选举，数据同步等)

–initial-advertise-peer-urls

建议用于节点之间通信的url，节点间将以该值进行通信。

–listen-client-urls

监听的用于客户端通信的url,同样可以监听多个。

–advertise-client-urls

建议使用的客户端通信url,该值用于etcd代理或etcd成员与etcd节点通信。

–initial-cluster-token etcd-cluster-1

节点的token值，设置该值后集群将生成唯一id,并为每个节点也生成唯一id,当使用相同配置文件再启动一个集群时，只要该token值不一样，etcd集群就不会相互影响。

–initial-cluster

也就是集群中所有的initial-advertise-peer-urls 的合集

–initial-cluster-state new

新建集群的标志，初始化状态使用 new，建立之后改此值为 existing

yum安装etcd需要修改两个配置文件

# vim /lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

WorkingDirectory=/var/lib/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

User=etcd

# set GOMAXPROCS to number of processors

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd \

--name=\"${ETCD_NAME}\" \

--data-dir=\"${ETCD_DATA_DIR}\" \

--listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \

--advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \

--initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \

--initial-cluster=\"${ETCD_INITIAL_CLUSTER}\"  \

--initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" \

--listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

# vim /etc/etcd/etcd.conf

ETCD_NAME=etcd1

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd1:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://etcd1:2380,etcd02=http://etcd2:2380,etcd03=http://etcd3:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="http://etcd1:2379"

配置好3台虚拟机之后同时启动etcd服务

# systemctl start etcd

正常情况可以查看到成员列表etcdctl member list

# etcdctl member list

3b05c2e4e4c104b4: name=etcd1 peerURLs=http://etcd1:2380 clientURLs=http://etcd1:2379 isLeader=false

bd388e7810915853: name=etcd2 peerURLs=http://etcd2:2380 clientURLs=http://etcd3:2379 isLeader=true

d282ac2ce600c1ce: name=etcd3 peerURLs=http://etcd3:2380 clientURLs=http://etcd2:2379 isLeader=false

2.2 二进制包安装etcd

yum虽然也能用，还是二进制包的方法操作简单些。

下载稳定版本的etcd二进制包

mkdir /var/lib/etcd;mkdir /etc/etcd; groupadd -r etcd; useradd -r -g etcd -d /var/lib/etcd -s /sbin/nologin -c "etcd user" etcd;chown -R etcd:etcd /var/lib/etcd

ETCD_VERSION=`curl -s -L https://github.com/coreos/etcd/releases/latest | grep linux-amd64\.tar\.gz | grep href | cut -f 6 -d '/' | sort -u`; ETCD_DIR=/opt/etcd-$ETCD_VERSION; mkdir $ETCD_DIR;curl -L https://github.com/coreos/etcd/releases/download/$ETCD_VERSION/etcd-$ETCD_VERSION-linux-amd64.tar.gz | tar xz --strip-components=1 -C $ETCD_DIR; ln -sf $ETCD_DIR/etcd /usr/bin/etcd && ln -sf $ETCD_DIR/etcdctl /usr/bin/etcdctl; etcd --version

四台虚拟机新建etcd服务启动脚本

[root@etcd1 /]# vim /lib/systemd/system/etcd.service

[Unit]

Description=etcd service

After=network.target

[Service]

Type=notify

WorkingDirectory=/var/lib/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

User=etcd

ExecStart=/usr/bin/etcd

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

etcd静态模式集群配置（即知道需要哪几台虚拟机需要建立集群）etcd1，etcd2与etcd3建立集群后etcd4加入集群。
修改etcd1配置文件

[root@etcd1 /]# cat /etc/etcd/etcd.conf

ETCD_NAME=etcd1

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd1:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://etcd1:2380,etcd02=http://etcd2:2380,etcd03=http://etcd3:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="http://etcd1:2379"

修改etcd2配置文件

[root@etcd2 /]# cat /etc/etcd/etcd.conf

ETCD_NAME=etcd2

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd2:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://etcd1:2380,etcd02=http://etcd2:2380,etcd03=http://etcd3:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="http://etcd2:2379"

修改etcd3配置文件

[root@etcd3 /]# cat /etc/etcd/etcd.conf

ETCD_NAME=etcd3

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd3:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://etcd1:2380,etcd02=http://etcd2:2380,etcd03=http://etcd3:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="http://etcd3:2379"

配置好3台虚拟机之后同时启动etcd服务

# systemctl start etcd

正常情况可以查看到成员列表etcdctl member list

[root@etcd1 /]# etcdctl member list

3b05c2e4e4c104b4: name=etcd01 peerURLs=http://etcd1:2380 clientURLs=http://etcd1:2379 isLeader=false

bd388e7810915853: name=etcd03 peerURLs=http://etcd3:2380 clientURLs=http://etcd3:2379 isLeader=true

d282ac2ce600c1ce: name=etcd02 peerURLs=http://etcd2:2380 clientURLs=http://etcd2:2379 isLeader=false

etcd配置过程中遇到的问题，添加节点，删除节点，备份数据
### 本地连接报错

[root@etcd2 /]# etcdctl member list

Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused

; error #1: dial tcp 127.0.0.1:4001: getsockopt: connection refused

error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused

error #1: dial tcp 127.0.0.1:4001: getsockopt: connection refused

如果出现如上的错误，是因为ETCD_LISTEN_CLIENT_URLS参数没有配置http://127.0.0.1:2379而导致的，所以这里我使用了0.0.0.0代表了监控所有地址。
除此之外仍上述报错，当配置的很混乱的时候可以删掉/var/lib/etcd/中的default.etcd文件,前提是数据不重要哈。还要注意default.etcd文件夹权限改为etcd：etcd

chown -R etcd:etcd default.etcd此步很重要，另外还需要细心检查/etc/etcd/etcd.conf和/lib/systemd/system/etcd.service是否有错误。改完之后再重启etcd服务（systemctl start etcd）就ok了。

etcd集群中添加一节点etcd04

在现有集群etcd三台虚拟机中任何一台操作添加新节点etcd04

[root@etcd1 /]# etcdctl member list

3b05c2e4e4c104b4: name=etcd01 peerURLs=http://etcd1:2380 clientURLs=http://etcd1:2379 isLeader=false

bd388e7810915853: name=etcd03 peerURLs=http://etcd3:2380 clientURLs=http://etcd3:2379 isLeader=true

d282ac2ce600c1ce: name=etcd02 peerURLs=http://etcd2:2380 clientURLs=http://etcd2:2379 isLeader=false

[root@etcd1 /]# etcdctl member add etcd04 http://etcd4:2380

Added member named etcd04 with ID ec8c708bb5b2fdc6 to cluster

ETCD_NAME="etcd04"

ETCD_INITIAL_CLUSTER="etcd01=http://etcd1:2380,etcd03=http://etcd3:2380,etcd02=http://etcd2:2380,etcd04=http://etcd4:2380"

ETCD_INITIAL_CLUSTER_STATE="existing"

注意此处三条必须要与etcd04配置文件（/etc/etcd/etcd.conf）相符合

[root@etcd1 /]# etcdctl member list

3b05c2e4e4c104b4: name=etcd01 peerURLs=http://etcd1:2380 clientURLs=http://etcd1:2379 isLeader=false

bd388e7810915853: name=etcd03 peerURLs=http://etcd3:2380 clientURLs=http://etcd3:2379 isLeader=true

d282ac2ce600c1ce: name=etcd02 peerURLs=http://etcd2:2380 clientURLs=http://etcd2:2379 isLeader=false

ec8c708bb5b2fdc6[unstarted]: peerURLs=http://etcd4:2380

启动etcd服务之前修改etcd04配置文件

[root@etcd4 /]# vim /etc/etcd/etcd.conf

ETCD_NAME=etcd04

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd4:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://etcd1:2380,etcd02=http://etcd2:2380,etcd03=http://etcd3:2380,etcd04=http://etcd4:2380"

ETCD_INITIAL_CLUSTER_STATE="existing"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="http://etcd4:2379"

如果/var/lib/etcd/中存在default.etcd一定要删除，否侧etcd服务启动失败（因为集群中给etcd04分配的物理id与现有库中的id不相同）

[root@etcd4 /]# rm -rf /var/lib/etcd/default.etcd/

[root@etcd4 /]# systemctl start etcd

[root@etcd4 /]# etcdctl member list         #启动服务后再查询etcd04已加入集群中

3b05c2e4e4c104b4: name=etcd01 peerURLs=http://etcd1:2380 clientURLs=http://etcd1:2379 isLeader=false

bd388e7810915853: name=etcd03 peerURLs=http://etcd3:2380 clientURLs=http://etcd3:2379 isLeader=true

d282ac2ce600c1ce: name=etcd02 peerURLs=http://etcd2:2380 clientURLs=http://etcd2:2379 isLeader=false

ec8c708bb5b2fdc6: name=etcd04 peerURLs=http://etcd4:2380 clientURLs=http://etcd4:2379 isLeader=false

etcd集群中删除etcd04节点

现有集群节点是4个

[root@etcd1 /]# etcdctl member list

3b05c2e4e4c104b4: name=etcd01 peerURLs=http://etcd1:2380 clientURLs=http://etcd1:2379 isLeader=false

b28ea7a9615a2c72: name=etcd04 peerURLs=http://etcd4:2380 clientURLs=http://etcd4:2379 isLeader=false

bd388e7810915853: name=etcd03 peerURLs=http://etcd3:2380 clientURLs=http://etcd3:2379 isLeader=true

d282ac2ce600c1ce: name=etcd02 peerURLs=http://etcd2:2380 clientURLs=http://etcd2:2379 isLeader=false

在除etcd04节点外的任意一台中操作删除etcd04节点

[root@etcd1 /]# etcdctl member remove b28ea7a9615a2c72

Removed member b28ea7a9615a2c72 from cluster

[root@etcd1 /]# etcdctl member list

3b05c2e4e4c104b4: name=etcd01 peerURLs=http://etcd1:2380 clientURLs=http://etcd1:2379 isLeader=false

bd388e7810915853: name=etcd03 peerURLs=http://etcd3:2380 clientURLs=http://etcd3:2379 isLeader=true

d282ac2ce600c1ce: name=etcd02 peerURLs=http://etcd2:2380 clientURLs=http://etcd2:2379 isLeader=false

etcd集群中备份数据

etcd的数据应该算是很安全的，即便出现单点故障，数据也不会丢失，前提是节点不在同一台服务器中
etcd的数据备份比较。。。网上的方法
etcdctl backup --data-dir /var/lib/etcd/default.etcd/ --backup-dir /home/etcd_backup
备份完之后，发现没有db文件，好尴尬啊，所有我直接cp了，
cp -r /var/lib/etcd/default.etcd/ /home/etcd_backup

chown -R etcd:etcd default.etcd嗯，就可以了。