Redis集群模式之分布式集群模式

前言

Redis集群模式主要有2种:

• 主从集群
• 分布式集群。

前者主要是为了高可用或是读写分离，后者为了更好的存储数据，负载均衡。 
本文主要讲解主从集群。本章主要讲解后一半部分，Redis集群。

与本文相关的代码与配置文件都已经上传至github上: 
地址: https://github.com/SeanYanxml/bigdata

---

原理

Redis为了实现负载均衡，提供集群模式。以三个节点为例，集群模式相当于将1-15000片分片，分为1-5000、5000-10000、10000-15000。每个节点分一段数据片。这样的话，当一个节点宕机后，这个节点没有备份的话，此段分片将不再可以使用。所以，官方推荐，集群内的每个节点都应该配备一个从节点，作为冷备。部署原理图如下所示(暂略)。

---

部署

由于没有那么多的机器，所以我们一般单机部署6个节点(3主3从)，也就是官网推荐的模式。

主要步骤如下:

• 安装ruby，因为分配集群的代码时ruby脚本；
• 分配集群
• 查看集群结果

---

部署代码

# mkdir cluster-test
# cd cluster-test
# mkdir 7000 7001 7002 7003 7004 7005
# 在文件夹内 分别放置redis.conf文件 文件内容见下
# 启动6个节点
# nohup ../../src/redis-server  redis.conf > start.log 2>&1 &
＃ 使用ruby脚本分配集群资源
./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

# redis.conf
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

• 1
• 2
• 3
• 4
• 5
• 6

### 启动redis进程7000 7001 7002 7003 7004 7005

touch start-all.sh
###  内容如下
cd 7000
nohup ../../src/redis-server  redis.conf > start.log 2>&1 &
cd ../7001
nohup ../../src/redis-server  redis.conf > start.log 2>&1 &
cd ../7002
nohup ../../src/redis-server  redis.conf > start.log 2>&1 &
cd ../7003
nohup ../../src/redis-server  redis.conf > start.log 2>&1 &
cd ../7004
nohup ../../src/redis-server  redis.conf > start.log 2>&1 &
cd ../7005
nohup ../../src/redis-server  redis.conf > start.log 2>&1 &

chmod +x start-all.sh
./start-all.sh

ps -ef|grep redis 查看进程
[cpic@cpic-redis2-77 src]$ ps -ef|grep redis
cpic     19876     1  0 3月21 ?       05:14:19 ./redis-server *:6379
cpic     19886     1  0 3月21 ?       06:17:36 ./redis-sentinel *:26379 [sentinel]
cpic     25971     1  0 15:07 pts/0    00:00:00 ../../src/redis-server *:7000 [cluster]
cpic     25972     1  0 15:07 pts/0    00:00:00 ../../src/redis-server *:7001 [cluster]
cpic     25973     1  0 15:07 pts/0    00:00:00 ../../src/redis-server *:7002 [cluster]
cpic     25974     1  0 15:07 pts/0    00:00:00 ../../src/redis-server *:7003 [cluster]
cpic     25975     1  0 15:07 pts/0    00:00:00 ../../src/redis-server *:7004 [cluster]
cpic     25976     1  0 15:07 pts/0    00:00:00 ../../src/redis-server *:7005 [cluster]
cpic     26273 24815  0 15:15 pts/0    00:00:00 grep --color=auto redis

成功结果

＃ create 脚本初始化后脚本
[root@cpic-redis-76 src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
M: 562665aca5f7db25bbd81e7b971c2f4c9aa65f96 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
M: 6f93b84f7da7d6ea1b77bce1709432de472bbbe3 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
M: 6522a9ddec5b40cade141793351df44fe40a529f 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
S: f122aea6aa02be27f77454c0fcde9c65b891cd47 127.0.0.1:7003
   replicates 562665aca5f7db25bbd81e7b971c2f4c9aa65f96
S: e5b5bbd378d8b80250dca6fb5db80e755f88c53c 127.0.0.1:7004
   replicates 6f93b84f7da7d6ea1b77bce1709432de472bbbe3
S: 93ac017741ad4e65a4de3cb17ae295e6e5505b9a 127.0.0.1:7005
   replicates 6522a9ddec5b40cade141793351df44fe40a529f
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 562665aca5f7db25bbd81e7b971c2f4c9aa65f96 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 6f93b84f7da7d6ea1b77bce1709432de472bbbe3 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: e5b5bbd378d8b80250dca6fb5db80e755f88c53c 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 6f93b84f7da7d6ea1b77bce1709432de472bbbe3
M: 6522a9ddec5b40cade141793351df44fe40a529f 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 93ac017741ad4e65a4de3cb17ae295e6e5505b9a 127.0.0.1:7005
   slots: (0 slots) slave
   replicates 6522a9ddec5b40cade141793351df44fe40a529f
S: f122aea6aa02be27f77454c0fcde9c65b891cd47 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 562665aca5f7db25bbd81e7b971c2f4c9aa65f96
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

• 1

# 自动生成的node.conf文件
[root@cpic-redis-76 7005]# cat nodes.conf
1c7cbff1069612078d63329e5841c7430757a2cd 127.0.0.1:7003@17003 slave f5cc2de688640de572d16d72c2f174de027609a6 0 1516672833000 4 connected
dd09a06827e9bfeab3876f4464cd467f6d1279f0 127.0.0.1:7002@17002 master - 0 1516672833086 3 connected 10923-16383
5863f9ee11d3ee3beef466d76f48f221b48aa6b5 127.0.0.1:7004@17004 slave 3ed6531ec985e50e87ab8436d49f4c32ce82bdef 0 1516672832000 5 connected
3ed6531ec985e50e87ab8436d49f4c32ce82bdef 127.0.0.1:7001@17001 master - 0 1516672832000 2 connected 5461-10922
9013209020c2fe2322df8c420b1c251fe9c84955 127.0.0.1:7005@17005 myself,slave dd09a06827e9bfeab3876f4464cd467f6d1279f0 0 1516672831000 6 connected
f5cc2de688640de572d16d72c2f174de027609a6 127.0.0.1:7000@17000 master - 0 1516672832885 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9

# 启动的log日志
[root@cpic-redis-76 7005]# cat start.log
nohup: 忽略输入
5627:C 22 Jan 21:02:58.212 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
5627:C 22 Jan 21:02:58.212 # Redis version=4.0.6, bits=64, commit=00000000, modified=0, pid=5627, just started
5627:C 22 Jan 21:02:58.212 # Configuration loaded
5627:M 22 Jan 21:02:58.214 * Increased maximum number of open files to 10032 (it was originally set to 1024).
5627:M 22 Jan 21:02:58.215 * Node configuration loaded, I'm 9013209020c2fe2322df8c420b1c251fe9c84955
5627:M 22 Jan 21:02:58.216 * Running mode=cluster, port=7005.
5627:M 22 Jan 21:02:58.216 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
5627:M 22 Jan 21:02:58.216 # Server initialized
5627:M 22 Jan 21:02:58.216 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
5627:M 22 Jan 21:02:58.216 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
5627:M 22 Jan 21:02:58.216 * Ready to accept connections
5627:M 23 Jan 10:00:27.212 # configEpoch set to 6 via CLUSTER SET-CONFIG-EPOCH
5627:M 23 Jan 10:00:27.286 # IP address for this node updated to 127.0.0.1
5627:S 23 Jan 10:00:31.238 * Before turning into a slave, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
5627:S 23 Jan 10:00:31.239 # Cluster state changed: ok
5627:S 23 Jan 10:00:31.981 * Connecting to MASTER 127.0.0.1:7002
5627:S 23 Jan 10:00:31.981 * MASTER <-> SLAVE sync started
5627:S 23 Jan 10:00:31.981 * Non blocking connect for SYNC fired the event.
5627:S 23 Jan 10:00:31.981 * Master replied to PING, replication can continue...
5627:S 23 Jan 10:00:31.982 * Trying a partial resynchronization (request 033127a1144abc3683a83a6b7257c2b56d2bd0b8:1).
5627:S 23 Jan 10:00:31.983 * Full resync from master: e5e51569d1f49762952e6cc05e1f13ddff68719e:0
5627:S 23 Jan 10:00:31.983 * Discarding previously cached master state.
5627:S 23 Jan 10:00:32.084 * MASTER <-> SLAVE sync: receiving 175 bytes from master
5627:S 23 Jan 10:00:32.084 * MASTER <-> SLAVE sync: Flushing old data
5627:S 23 Jan 10:00:32.084 * MASTER <-> SLAVE sync: Loading DB in memory
5627:S 23 Jan 10:00:32.084 * MASTER <-> SLAVE sync: Finished with success
5627:S 23 Jan 10:00:32.085 * Background append only file rewriting started by pid 10164
5627:S 23 Jan 10:00:32.108 * AOF rewrite child asks to stop sending diffs.
10164:C 23 Jan 10:00:32.108 * Parent agreed to stop sending diffs. Finalizing AOF...
10164:C 23 Jan 10:00:32.108 * Concatenating 0.00 MB of AOF diff received from parent.
10164:C 23 Jan 10:00:32.108 * SYNC append only file rewrite performed
10164:C 23 Jan 10:00:32.109 * AOF rewrite: 0 MB of memory used by copy-on-write
5627:S 23 Jan 10:00:32.181 * Background AOF rewrite terminated with success
5627:S 23 Jan 10:00:32.181 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
5627:S 23 Jan 10:00:32.181 * Background AOF rewrite finished successfully

---

BUG

• BUG1

[root@cpic-redis-76 src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
[ERR] Node 127.0.0.1:7005 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
解决办法: 删掉原有数据dump.rdb/appendonly.aof/nodes.conf文件。

• 1
• 2
• 3
• 4

• BUG2

ruby安装一些列困难。

• 1

详见Redis Cluster 运维环境安装记录。

---

Reference

见Redis Cluster 运维环境安装记录