在线tidb+tipd+tikv扩容，迁移，从UC到阿里云

集群现状：

共有五个节点，配置为16核32g内存，数据节点为1T ssd盘，非数据节点为100g ssd盘；

角色规划：

node1   tidb  tipd
node2   tidb  tipd
node3   tikv  tipd
node4   tikv
node5   tikv 

1.每次操作都需要更改配置文件inventory.ini，都是在tidb用户下进行；

2.初始化，打通新增节点与已有节点间免密码登录(tidb)用户；

3.扩容tidb节点两个

## TiDB Cluster Part
[tidb_servers]
10.15.xxx.xxx     ---旧的tidb---uc机器
10.15.xxx.xxx     ---旧的tidb---uc机器
10.80.xxx.xxx     ---新加tidb---阿里机器
10.80.xxx.xxx     ---新加tidb---阿里机器

3.1 部署

[tidb@tidb..tidb.prod.uc:~/tidb-ansible]$ ansible-playbook deploy.yml -l 10.80.xxx.xxx,10.80.xxx.xxx
　

。。。。。。。。。。。。。。。。。。。。。。。。。。。

。。。。。。。。。。。。。。。。。

。。。。。。。。。

PLAY RECAP ***********************************************************************************************************************************************************************************************
10.80.249.46 : ok=30 changed=12 unreachable=0 failed=0 
10.80.249.47 : ok=30 changed=12 unreachable=0 failed=0

Congrats! All goes well. :-)

3.2 启动新增tidb节点

[tidb@tidb..tidb.prod.uc:~/tidb-ansible]$ ansible-playbook rolling_update_monitor.yml --tags=prometheus
 
PLAY RECAP ***********************************************************************************************************************************************************************************************
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.80.xxx.xxx : ok= changed= unreachable= failed=
10.80.xxx.xxx : ok= changed= unreachable= failed=
localhost : ok= changed= unreachable= failed=
 
Congrats! All goes well. :-)

3.3更新监控

[tidb@tidb..tidb.prod.uc:~/tidb-ansible]$ ansible-playbook rolling_update_monitor.yml --tags=prometheus
 
PLAY RECAP ***********************************************************************************************************************************************************************************************
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.15.xxx.xxx : ok= changed= unreachable= failed=
10.80.xxx.xxx : ok= changed= unreachable= failed=
10.80.xxx.xxx : ok= changed= unreachable= failed=
localhost : ok= changed= unreachable= failed=
 
Congrats! All goes well. :-)

4.扩容tikv节点

[tikv_servers]
10.15.xxx.xxx
10.15.xxx.xxx
10.15.xxx.xxx
10.80.xxx.xxx
10.80.xxx.xxx
10.80.xxx.xxx

4.1.tikv为数据节点，扩容之前把ssd盘格式化分区挂载

root@tikv..tidb.prod.ali:~/.ssh# vi /etc/fstab
 
#
# /etc/fstab
# Created by anaconda on Sun Oct 15 15:19:00 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=eb448abb--4d8d-bcde-94434d586a31 / ext4 defaults
#/dev/vdb /data ext4 defaults,noatime 0 0
/dev/vdb /data ext4 defaults,nodelalloc,noatime  
 
root@tikv..tidb.prod.ali:~/.ssh#
root@tikv..tidb.prod.ali:~/.ssh# umount /data
root@tikv..tidb.prod.ali:~/.ssh# mount -a
root@tikv..tidb.prod.ali:~/.ssh# mount -t ext4
/dev/vda1 on / type ext4 (rw,relatime,data=ordered)
/dev/vdb on /data type ext4 (rw,noatime,nodelalloc,data=ordered)

4.2  加入tikv,启动服务，更新

[tidb@tidb.11.tidb.prod.uc:~/tidb-ansible]$ ansible-playbook bootstrap.yml -l 10.80.249.59,10.80.249.60,10.80.249.58 

[tidb@tidb.11.tidb.prod.uc:~/tidb-ansible]$ ansible-playbook start.yml -l 10.80.249.59,10.80.249.60,10.80.249.58

[tidb@tidb.11.tidb.prod.uc:~/tidb-ansible]$ ansible-playbook rolling_update_monitor.yml --tags=prometheus

5.加入tidb(加入tipd节点要一个一个加入)

[pd_servers]
10.15.xxx.xxx
10.15.xxx.xxx
10.15.xxx.xxx
10.80.xxx.xxx

5.1初始化新加入tipd

[tidb@tidb..tidb.prod.uc:~/tidb-ansible]$ ansible-playbook bootstrap.yml -l 10.80.xxx.xxx

2.安装目标服务器

[tidb@tidb.01.tidb.prod.uc:~/tidb-ansible]$ ansible-playbook deploy.yml -l 10.80.xxx.xxx

PLAY RECAP ***************************************************************************************************************************
10.80.249.46 : ok=34 changed=0 unreachable=0 failed=0

Congrats! All goes well. :-)

5.3登录到新加入的tipd节点，vi run_pd.sh

root@tidb..tidb.prod.ali:/data/tidb/deploy/scripts# vi run_pd.sh
--initial-cluster="pd1=http://10.15.xxx.xxx:2380,pd2=http://10.15.xxx.xxx:2380,pd3=http://10.15.xxx.xxx:2380,pd4=http://10.80.xxx.xxx:2380" \
#!/bin/bash
set -e
ulimit -n 
 
# WARNING: This file was auto-generated. Do not edit!
# All your edit might be overwritten!
DEPLOY_DIR=/data/tidb/deploy
 
cd "${DEPLOY_DIR}" || exit 
 
exec bin/pd-server \
--name="pd4" \
--client-urls="http://10.80.xxx.xxx:2379" \
--advertise-client-urls="http://10.80.xxx.xxx:2379" \
--peer-urls="http://10.80.xxx.xxx:2380" \
--advertise-peer-urls="http://10.80.xxx.xxx:2380" \
--data-dir="/data/tidb/deploy/data.pd" \
--config=conf/pd.toml \
--join="http://10.15.xxx.xxx:2380" \
--log-file="/data/tidb/deploy/log/pd.log" >> "/data/tidb/deploy/log/pd_stderr.log
 
在目标服务器手动启动pd服务:
 
tidb@tidb.11.tidb.prod.ali:/data/tidb/deploy/scripts$ sh -x start_pd.sh

5.4 在目标机器查看pd服务:

tidb@tidb.11.tidb.prod.ali:/data/tidb/deploy/scripts$ ps -ef | grep tidb
tidb 6922 1 0 14:29 ? 00:00:02 bin/pd-server --name=pd4 --client-urls=http://10.80.xxx.xxx:2379 --advertise-client-urls=http://10.80.xxx.xxx:2379 --peer-urls=http://10.80.xxx.xxx:2380 --advertise-peer-urls=http://10.80.xxx.xxx:2380 --data-dir=/data/tidb/deploy/data.pd --config=conf/pd.toml --join=http://10.15.xxx.xxx:2380 --log-file=/data/tidb/deploy/log/pd.log

5.5 滚动升级集群：

[tidb@tidb.11.tidb.prod.uc:~/tidb-ansible]$ ansible-playbook rolling_update.yml

5.6更新 Prometheus 配置并重启：

[tidb@tidb.11.tidb.prod.uc:~/tidb-ansible]$ ansible-playbook rolling_update_monitor.yml --tags=prometheus