KingbaseES V8R6集群运维案例之---repmgr standby promote应用案例

案例说明：

在容灾环境中，跨区域部署的异地备节点不会自主提升为主节点，在主节点发生故障或者人为需要切换时需要手动执行切换操作。若主节点已经失效，希望将异地备机提升为主节点。

$bin/repmgr standby promote

适用版本：

KingbaseES V8R6

集群节点信息：

 ID | Name    | Role    | Status    | Upstream | repmgrd | PID   | Paused? | Upstream last seen
----+---------+---------+-----------+----------+---------+-------+---------+--------------------
 1  | node101 | standby |   running | node102  | running | 19312 | no      | 1 second(s) ago
 2  | node102 | primary | * running |          | running | 20658 | no      | n/a

主备流复制状态信息：

test=# select * from sys_stat_replication;
  pid  | usesysid | usename | application_name |  client_addr  | client_hostname | client_port |         backend_start         | b
ackend_xmin |   state   |  sent_lsn  | write_lsn  | flush_lsn  | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority |
 sync_state |          reply_time
-------+----------+---------+------------------+---------------+-----------------+-------------+-------------------------------+--
 20165 |       10 | system  | node101          | 192.168.1.101 |                 |       10747 | 2022-09-08 11:12:59.798843+08 |
            | streaming | 4/4C0018A0 | 4/4C0018A0 | 4/4C0018A0 | 4/4C0018A0 |           |           |            |             1 |
 sync       | 2022-09-08 11:16:45.423742+08
(1 row)

关闭failover自动切换：

[kingbase@node102 bin]$ cat ../etc/repmgr.conf|grep failover
#failover='automatic'
failover='manual'

一、模拟主库数据库服务宕机

[kingbase@node102 bin]$ ./sys_ctl stop -D /data/kingbase/r6ha/data
waiting for server to shut down....... done

二、查看备库状态

1、数据库进程状态（仍为备库进程）

kingbase 19132     1  0 11:13 ?        00:00:00 /home/kingbase/cluster/R6HA/kha/kingbase/bin/kingbase -D /data/kingbase/r6ha/data
kingbase 19133 19132  0 11:13 ?        00:00:00 kingbase: logger
kingbase 19134 19132  0 11:13 ?        00:00:00 kingbase: startup   recovering 00000018000000040000004D
kingbase 19135 19132  0 11:13 ?        00:00:00 kingbase: checkpointer
kingbase 19136 19132  0 11:13 ?        00:00:00 kingbase: background writer
kingbase 19137 19132  0 11:13 ?        00:00:00 kingbase: stats collector
kingbase 19310 19132  0 11:13 ?        00:00:00 kingbase: system esrep 192.168.1.101(15211) idle

2、查看备库hamgr.log（获取切换过程）

# 备库repmgrd进程监控主库数据库服务状态（PQping()），当主库返回"PQPING_NO_RESPONSE"后，尝试
再次连接主库，超过阈值后，执行切换。

[2022-09-08 11:18:10] [INFO] sleeping 6 seconds until next reconnection attempt
[2022-09-08 11:18:16] [INFO] checking state of node 2, 1 of 10 attempts
[2022-09-08 11:18:16] [DEBUG] is_server_available_params(): ping status for "user=system connect_timeout=10 dbname=esrep host=192.168.1.102 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr" is PQPING_NO_RESPONSE
[2022-09-08 11:18:16] [WARNING] unable to ping "user=system connect_timeout=10 dbname=esrep host=192.168.1.102 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr"
[2022-09-08 11:18:16] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2022-09-08 11:18:16] [INFO] sleeping 6 seconds until next reconnection attempt
........

[2022-09-08 11:19:10] [INFO] checking state of node 2, 10 of 10 attempts
[2022-09-08 11:19:10] [DEBUG] is_server_available_params(): ping status for "user=system connect_timeout=10 dbname=esrep host=192.168.1.102 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr" is PQPING_NO_RESPONSE
.......

# 准备执行切换，但是在repmgr.conf中配置failover=‘manual’，将不会执行自动切换。

[2022-09-08 11:19:10] [DEBUG] do_election(): electoral term is 1
[2022-09-08 11:19:10] [NOTICE] this node is not configured for automatic failover so will not be considered as promotion candidate, and will not follow the new primary
[2022-09-08 11:19:10] [DETAIL] "failover" is set to "manual" in repmgr.conf
[2022-09-08 11:19:10] [HINT] manually execute "repmgr standby follow" to have this node follow the new primary
[2022-09-08 11:19:10] [DEBUG] election result: NOT CANDIDATE

三、在备库执行promote手工切换

1、执行手工切换

[kingbase@node101 bin]$ ./repmgr standby promote
DEBUG: connecting to: "user=system connect_timeout=10 dbname=esrep host=192.168.1.101 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr"
DEBUG: connecting to: "user=system connect_timeout=10 dbname=esrep host=192.168.1.102 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr"
DEBUG: connecting to: "user=system connect_timeout=10 dbname=esrep host=192.168.1.101 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr"
NOTICE: promoting standby to primary
DETAIL: promoting server "node101" (ID: 1) using sys_promote()
NOTICE: waiting up to 60 seconds (parameter "promote_check_timeout") for promotion to complete
DEBUG: setting node 1 as primary and marking existing primary as failed
NOTICE: STANDBY PROMOTE successful
DETAIL: server "node101" (ID: 1) was successfully promoted to primary

2、查看切换后数据库进程状态（切换为主库状态）

[kingbase@node101 bin]$ ps -ef |grep kingbase

kingbase 19132     1  0 11:13 ?        00:00:00 /home/kingbase/cluster/R6HA/kha/kingbase/bin/kingbase -D /data/kingbase/r6ha/data
kingbase 19133 19132  0 11:13 ?        00:00:00 kingbase: logger
kingbase 19135 19132  0 11:13 ?        00:00:00 kingbase: checkpointer
kingbase 19136 19132  0 11:13 ?        00:00:00 kingbase: background writer
kingbase 19137 19132  0 11:13 ?        00:00:00 kingbase: stats collector
kingbase 19780 19132  0 11:13 ?        00:00:00 kingbase: system test ::1(33354) idle
kingbase 19784 19132  0 11:13 ?        00:00:00 kingbase: system esrep 192.168.1.101(15243) idle
kingbase 20826 19132  0 11:20 ?        00:00:00 kingbase: walwriter
kingbase 20827 19132  0 11:20 ?        00:00:00 kingbase: autovacuum launcher
kingbase 20828 19132  1 11:20 ?        00:00:00 kingbase: archiver   archiving 0000001500000002000000FC

四、将原主库恢复为新备库加入集群

1、创建备库标识文件

[kingbase@node102 log]$ touch /data/kingbase/r6ha/data/standby.signal

2、启动数据库服务

[kingbase@node102 bin]$ ./sys_ctl start -D /data/kingbase/r6ha/data

3、注册备库节点

[kingbase@node102 bin]$ ./repmgr standby register --force
INFO: connecting to local node "node102" (ID: 2)
DEBUG: connecting to: "user=system connect_timeout=10 dbname=esrep host=192.168.1.102 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr"
INFO: connecting to primary database
DEBUG: connecting to: "user=system connect_timeout=10 dbname=esrep host=192.168.1.102 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr"
DEBUG: connecting to: "user=system connect_timeout=10 dbname=esrep host=192.168.1.101 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr"
DEBUG: remote_command():
  ssh -o Batchmode=yes -q -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=2 -o ServerAliveCountMax=5 -p 22 192.168.1.101 /home/kingbase/cluster/R6HA/kha/kingbase/bin/kbha -A updateinfo
INFO: standby registration complete
NOTICE: standby node "node102" (ID: 2) successfully registered

4、查看集群节点状态

[kingbase@node102 bin]$ ./repmgr cluster show

 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node101 | primary | * running |          | default  | 100      | 25       | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node102 | standby |   running | node101  | default  | 100      | 24       | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

五、总结

对于KingbaseES V8R6集群在failover切换时，如果备库不能自动切换为主库，或主库宕机后切换失败，都可以使用‘repmgr standby promote’强制手工切换备库为主库，恢复业务访问。