Redis Cluster的搭建与部署，实现redis的分布式方案

前言

　　上篇Redis Sentinel安装与部署，实现redis的高可用实现了redis的高可用，针对的主要是master宕机的情况，我们发现所有节点的数据都是一样的，那么一旦数据量过大，redis也会效率下降的问题。redis3.0版本正式推出后，有效地解决了Redis分布式方面的需求，当遇到单机内存、并发、流量等瓶颈时，可以采用Cluster架构方法达到负载均衡的目的。

　　而此篇将带领大家实现Redis Cluster的搭建， 并进行简单的客户端操作。

　　github地址：https://github.com/youzhibing/redis

环境准备　

　　redis版本：redis-3.0.0

　　linux：centos6.7

　　ip：192.168.11.202，不同的端口实现不同的redis实例

　　客户端jedis，基于spring-boot

redis cluster环境搭建

节点准备

　　　　192.168.11.202:6382,192.168.11.202:6383,192.168.11.202:6384,192.168.11.202:6385,192.168.11.202:6386,192.168.11.202:6387搭建初始集群

　　　　192.168.11.202:6388，192.168.11.202:6389扩容时用到

　　　　redis-6382.conf

port 6382
bind 192.168.11.202
requirepass "myredis"
daemonize yes
logfile "6382.log"
dbfilename "dump-6382.rdb"
dir "/opt/soft/redis/cluster_data"

masterauth "myredis"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6382.conf"

　　　　redis-6383.conf

port 6383
bind 192.168.11.202
requirepass "myredis"
daemonize yes
logfile "6383.log"
dbfilename "dump-6383.rdb"
dir "/opt/soft/redis/cluster_data"

masterauth "myredis"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6383.conf"

　　　　redis-6384.conf

port 6384
bind 192.168.11.202
requirepass "myredis"
daemonize yes
logfile "6384.log"
dbfilename "dump-6384.rdb"
dir "/opt/soft/redis/cluster_data"

masterauth "myredis"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6384.conf"

　　　　redis-6385.conf

port 6385
bind 192.168.11.202
requirepass "myredis"
daemonize yes
logfile "6385.log"
dbfilename "dump-6385.rdb"
dir "/opt/soft/redis/cluster_data"

masterauth "myredis"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6385.conf"

　　　　redis-6386.conf

port 6386
bind 192.168.11.202
requirepass "myredis"
daemonize yes
logfile "6386.log"
dbfilename "dump-6386.rdb"
dir "/opt/soft/redis/cluster_data"

masterauth "myredis"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6386.conf"

　　　　redis-6387.conf

port 6387
bind 192.168.11.202
requirepass "myredis"
daemonize yes
logfile "6387.log"
dbfilename "dump-6387.rdb"
dir "/opt/soft/redis/cluster_data"

masterauth "myredis"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6387.conf"

　　启动全部节点

[root@slave1 redis_cluster]# cd /opt/redis-3.0.0/redis_cluster/
[root@slave1 redis_cluster]# ./../src/redis-server redis-6382.conf
[root@slave1 redis_cluster]# ./../src/redis-server redis-6383.conf
[root@slave1 redis_cluster]# ./../src/redis-server redis-6384.conf
[root@slave1 redis_cluster]# ./../src/redis-server redis-6385.conf
[root@slave1 redis_cluster]# ./../src/redis-server redis-6386.conf
[root@slave1 redis_cluster]# ./../src/redis-server redis-6387.conf

　　 创建集群

　　　　节点全部启动后，每个节点目前只能识别出自己的节点信息，彼此之间并不知道对方的存在；

　　　　采用redis-trib.rb来实现集群的快速搭建，redis-trib.rb是采用Rudy实现的集群管理工具，内部通过Cluster相关命令帮我们简化集群创建、检查、槽迁移和均衡等常见运维操作。

　　　　有兴趣的朋友可以采用cluster 命令一步一步的手动实现redis cluster的搭建，就可以明白redis-trib.rb是如何快速实现redis cluster的搭建的。

　　　　搭建命令如下，其中--replicas 1表示每个主节点配置1个从节点

[root@slave1 src]# cd /opt/redis-3.0.0/src/
[root@slave1 src]# ./redis-trib.rb create --replicas 1 192.168.11.202:6382 192.168.11.202:6383 192.168.11.202:6384 192.168.11.202:6385 192.168.11.202:6386 192.168.11.202:6387　 

　　　　创建过程中会给出主从节点角色分配的计划，如下所示

>>> Creating cluster
Connecting to node 192.168.11.202:6382: OK
Connecting to node 192.168.11.202:6383: OK
Connecting to node 192.168.11.202:6384: OK
Connecting to node 192.168.11.202:6385: OK
Connecting to node 192.168.11.202:6386: OK
Connecting to node 192.168.11.202:6387: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.11.202:6382
192.168.11.202:6383
192.168.11.202:6384
Adding replica 192.168.11.202:6385 to 192.168.11.202:6382
Adding replica 192.168.11.202:6386 to 192.168.11.202:6383
Adding replica 192.168.11.202:6387 to 192.168.11.202:6384
M: 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 192.168.11.202:6382
   slots:0-5460 (5461 slots) master
M: 3771e67edab547deff6bd290e1a07b23646906ee 192.168.11.202:6383
   slots:5461-10922 (5462 slots) master
M: 10b3789bb30889b5e6f67175620feddcd496d19e 192.168.11.202:6384
   slots:10923-16383 (5461 slots) master
S: 7649466ec006e0902a7f1578417247a6d5540c47 192.168.11.202:6385
   replicates 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe
S: 4f36b08d8067a003af45dbe96a5363f348643509 192.168.11.202:6386
   replicates 3771e67edab547deff6bd290e1a07b23646906ee
S: a583def1e6a059e4fdb3592557fd6ab691fd61ec 192.168.11.202:6387
   replicates 10b3789bb30889b5e6f67175620feddcd496d19e
Can I set the above configuration? (type 'yes' to accept): 

　　　　为什么192.168.11.202:6382 192.168.11.202:6383 192.168.11.202:6384是主节点，请看注意点中第1点；当我们同意这份计划之后，输入yes，redis-trib.rb开始执行节点握手和槽分配操作，输出如下

>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 192.168.11.202:6382)
M: 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 192.168.11.202:6382
   slots:0-5460 (5461 slots) master
M: 3771e67edab547deff6bd290e1a07b23646906ee 192.168.11.202:6383
   slots:5461-10922 (5462 slots) master
M: 10b3789bb30889b5e6f67175620feddcd496d19e 192.168.11.202:6384
   slots:10923-16383 (5461 slots) master
M: 7649466ec006e0902a7f1578417247a6d5540c47 192.168.11.202:6385
   slots: (0 slots) master
   replicates 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe
M: 4f36b08d8067a003af45dbe96a5363f348643509 192.168.11.202:6386
   slots: (0 slots) master
   replicates 3771e67edab547deff6bd290e1a07b23646906ee
M: a583def1e6a059e4fdb3592557fd6ab691fd61ec 192.168.11.202:6387
   slots: (0 slots) master
   replicates 10b3789bb30889b5e6f67175620feddcd496d19e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

　　　　16384个槽全部被分配，集群创建成功。

　　集群完整性检查

　　　　redis-trib.rb check命令可以完成检查工作，check命令只需给出集群中任意一个节点地址就可以完成整个集群的检查工作，如下

　　　　redis-trib.rb check 192.168.11.202:6382， 输出结果如下

Connecting to node 192.168.11.202:6382: OK
Connecting to node 192.168.11.202:6385: OK
Connecting to node 192.168.11.202:6383: OK
Connecting to node 192.168.11.202:6384: OK
Connecting to node 192.168.11.202:6387: OK
Connecting to node 192.168.11.202:6386: OK
>>> Performing Cluster Check (using node 192.168.11.202:6382)
M: 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 192.168.11.202:6382
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 7649466ec006e0902a7f1578417247a6d5540c47 192.168.11.202:6385
   slots: (0 slots) slave
   replicates 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe
M: 3771e67edab547deff6bd290e1a07b23646906ee 192.168.11.202:6383
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
M: 10b3789bb30889b5e6f67175620feddcd496d19e 192.168.11.202:6384
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: a583def1e6a059e4fdb3592557fd6ab691fd61ec 192.168.11.202:6387
   slots: (0 slots) slave
   replicates 10b3789bb30889b5e6f67175620feddcd496d19e
S: 4f36b08d8067a003af45dbe96a5363f348643509 192.168.11.202:6386
   slots: (0 slots) slave
   replicates 3771e67edab547deff6bd290e1a07b23646906ee
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

　　　　[OK] All 16384 slots covered.表示集群所有的槽都已分配到节点。

　　　　引入了槽之后，整个数据流向如下图所示：

　　　　至于为什么引入槽，请看注意点中第3点

　　redis cluster简单操作

　　　　连接集群，随便连接某个节点都可以；-c 集群支持，支持自动重定向

　　　　[root@slave1 redis_cluster]# ./../src/redis-cli -h 192.168.11.202 -p 6382 -a myredis -c

　　　　连接上redis cluster后就可以执行相关redis命令了，如下

192.168.11.202:6382> get name
-> Redirected to slot [5798] located at 192.168.11.202:6388
"youzhibing"
192.168.11.202:6388> set weight 112
-> Redirected to slot [16280] located at 192.168.11.202:6384
OK
192.168.11.202:6384> get weight
"112"
192.168.11.202:6384> 

客户端（Jedis）连接与操作

　　redis-cluster.properties

#cluster
redis.cluster.host=192.168.11.202
redis.cluster.port=6382,6383,6384,6385,6386,6387
#redis读写超时时间（毫秒）
redis.cluster.socketTimeout=1000
#redis连接超时时间（毫秒）
redis.cluster.connectionTimeOut=3000
#最大尝试连接次数
redis.cluster.maxAttempts=10
#最大重定向次数
redis.cluster.maxRedirects=5
#master连接密码
redis.password=myredis

# 连接池
# 连接池最大连接数（使用负值表示没有限制）
redis.pool.maxActive=150
# 连接池中的最大空闲连接
redis.pool.maxIdle=10
# 连接池中的最小空闲连接
redis.pool.minIdle=1
# 获取连接时的最大等待毫秒数,小于零:阻塞不确定的时间,默认-1
redis.pool.maxWaitMillis=3000
# 每次释放连接的最大数目
redis.pool.numTestsPerEvictionRun=50
# 释放连接的扫描间隔（毫秒）
redis.pool.timeBetweenEvictionRunsMillis=3000
# 连接最小空闲时间（毫秒）
redis.pool.minEvictableIdleTimeMillis=1800000
# 连接空闲多久后释放, 当空闲时间>该值 且 空闲连接>最大空闲连接数 时直接释放（毫秒）
redis.pool.softMinEvictableIdleTimeMillis=10000
# 在获取连接的时候检查有效性, 默认false
redis.pool.testOnBorrow=true
# 在空闲时检查有效性, 默认false
redis.pool.testWhileIdle=true
# 在归还给pool时，是否提前进行validate操作
redis.pool.testOnReturn=true
# 连接耗尽时是否阻塞, false报异常,ture阻塞直到超时, 默认true
redis.pool.blockWhenExhausted=true

　　RedisClusterConfig.java

package com.lee.redis.config.cluster;

import java.util.HashSet;
import java.util.Set;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.util.StringUtils;

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
import redis.clients.jedis.JedisPoolConfig;

import com.alibaba.fastjson.JSON;
import com.lee.redis.exception.LocalException;

@Configuration
@PropertySource("redis/redis-cluster.properties")
public class RedisClusterConfig {

    private static final Logger LOGGER = LoggerFactory.getLogger(RedisClusterConfig.class);

    // pool
    @Value("${redis.pool.maxActive}")
    private int maxTotal;
    @Value("${redis.pool.maxIdle}")
    private int maxIdle;
    @Value("${redis.pool.minIdle}")
    private int minIdle;
    @Value("${redis.pool.maxWaitMillis}")
    private long maxWaitMillis;
    @Value("${redis.pool.numTestsPerEvictionRun}")
    private int numTestsPerEvictionRun;
    @Value("${redis.pool.timeBetweenEvictionRunsMillis}")
    private long timeBetweenEvictionRunsMillis;
    @Value("${redis.pool.minEvictableIdleTimeMillis}")
    private long minEvictableIdleTimeMillis;
    @Value("${redis.pool.softMinEvictableIdleTimeMillis}")
    private long softMinEvictableIdleTimeMillis;
    @Value("${redis.pool.testOnBorrow}")
    private boolean testOnBorrow;
    @Value("${redis.pool.testWhileIdle}")
    private boolean testWhileIdle;
    @Value("${redis.pool.testOnReturn}")
    private boolean testOnReturn;
    @Value("${redis.pool.blockWhenExhausted}")
    private boolean blockWhenExhausted;

    // cluster
    @Value("${redis.cluster.host}")
    private String host;
    @Value("${redis.cluster.port}")
    private String port;
    @Value("${redis.cluster.socketTimeout}")
    private int socketTimeout;
    @Value("${redis.cluster.connectionTimeOut}")
    private int connectionTimeOut;
    @Value("${redis.cluster.maxAttempts}")
    private int maxAttempts;
    @Value("${redis.cluster.maxRedirects}")
    private int maxRedirects;
    @Value("${redis.password}")
    private String password;

    @Bean
    public JedisPoolConfig jedisPoolConfig() {
        JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
        jedisPoolConfig.setMaxTotal(maxTotal);
        jedisPoolConfig.setMaxIdle(maxIdle);
        jedisPoolConfig.setMinIdle(minIdle);
        jedisPoolConfig.setMaxWaitMillis(maxWaitMillis);
        jedisPoolConfig.setNumTestsPerEvictionRun(numTestsPerEvictionRun);
        jedisPoolConfig
                .setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis);
        jedisPoolConfig
                .setMinEvictableIdleTimeMillis(minEvictableIdleTimeMillis);
        jedisPoolConfig
                .setSoftMinEvictableIdleTimeMillis(softMinEvictableIdleTimeMillis);
        jedisPoolConfig.setTestOnBorrow(testOnBorrow);
        jedisPoolConfig.setTestWhileIdle(testWhileIdle);
        jedisPoolConfig.setTestOnReturn(testOnReturn);
        jedisPoolConfig.setBlockWhenExhausted(blockWhenExhausted);

        return jedisPoolConfig;
    }

    @Bean
    public JedisCluster jedisCluster(JedisPoolConfig jedisPoolConfig) {

        if (StringUtils.isEmpty(host)) {
            LOGGER.info("redis集群主机未配置");
            throw new LocalException("redis集群主机未配置");
        }
        if (StringUtils.isEmpty(port)) {
            LOGGER.info("redis集群端口未配置");
            throw new LocalException("redis集群端口未配置");
        }
        String[] hosts = host.split(",");
        String[] portArray = port.split(";");
        if (hosts.length != portArray.length) {
            LOGGER.info("redis集群主机数与端口数不匹配");
            throw new LocalException("redis集群主机数与端口数不匹配");
        }
        Set<HostAndPort> redisNodes = new HashSet<HostAndPort>();
        for (int i = 0; i < hosts.length; i++) {
            String ports = portArray[i];
            String[] hostPorts = ports.split(",");
            for (String port : hostPorts) {
                HostAndPort node = new HostAndPort(hosts[i], Integer.parseInt(port));
                redisNodes.add(node);
            }
        }
        LOGGER.info("Set<RedisNode> : {}", JSON.toJSONString(redisNodes), true);

        return new JedisCluster(redisNodes, connectionTimeOut, socketTimeout, maxAttempts, password, jedisPoolConfig);
    }
}

　　ApplicationCluster.java

package com.lee.redis;

import org.springframework.boot.Banner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableAutoConfiguration
@ComponentScan(basePackages={"com.lee.redis.config.cluster"})
public class ApplicationCluster {
    public static void main(String[] args) {

        SpringApplication app = new SpringApplication(ApplicationCluster.class);
        app.setBannerMode(Banner.Mode.OFF);            // 是否打印banner
        // app.setApplicationContextClass();        // 指定spring应用上下文启动类
        app.setWebEnvironment(false);
        app.run(args);
    }
}

　　RedisClusterTest.java

package com.lee.redis;

import java.util.List;
import java.util.Map;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

import com.alibaba.fastjson.JSON;

import redis.clients.jedis.JedisCluster;
import redis.clients.jedis.JedisPool;

@RunWith(SpringRunner.class)
@SpringBootTest(classes = ApplicationCluster.class)
public class RedisClusterTest {
    private static final Logger LOGGER = LoggerFactory.getLogger(RedisClusterTest.class);

    @Autowired
    private JedisCluster jedisCluster;

    @Test
    public void initTest() {
        String name = jedisCluster.get("name");
        LOGGER.info("name is {}", name);

        // list操作
        long count = jedisCluster.lpush("list:names", "陈芸");    // lpush的返回值是在 push操作后的 list 长度
        LOGGER.info("count = {}", count);
        long nameLen = jedisCluster.llen("list:names");
        LOGGER.info("list:names lens is {}", nameLen);
        List<String> nameList = jedisCluster.lrange("list:names", 0, nameLen);
        LOGGER.info("names : {}", JSON.toJSONString(nameList));
    }
}

　　执行RedisClusterTest.java中的initTest方法， 结果如下

......
2018-03-06 09:56:05|INFO|com.lee.redis.RedisClusterTest|name is youzhibing
2018-03-06 09:56:05|INFO|com.lee.redis.RedisClusterTest|count = 3
2018-03-06 09:56:05|INFO|com.lee.redis.RedisClusterTest|list:names lens is 3
2018-03-06 09:56:05|INFO|com.lee.redis.RedisClusterTest|names : ["陈芸","沈复","沈复"]
......

集群的伸缩与故障转移

　　cluster扩容

　　　　新增节点：192.168.11.202:6388, 192.168.11.202:6389， 配置文件与之前的基本一致

　　　　　　redis-6388.conf

port 6388
bind 192.168.11.202
requirepass "myredis"
daemonize yes
logfile "6388.log"
dbfilename "dump-6388.rdb"
dir "/opt/soft/redis/cluster_data"

masterauth "myredis"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6388.conf"

　　　　　　redis-6389.conf

port 6389
bind 192.168.11.202
requirepass "myredis"
daemonize yes
logfile "6389.log"
dbfilename "dump-6389.rdb"
dir "/opt/soft/redis/cluster_data"

masterauth "myredis"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6389.conf"

　　　　　　启动6388、6389节点

[root@slave1 redis_cluster]# cd /opt/redis-3.0.0/redis_cluster/
[root@slave1 redis_cluster]# ./../src/redis-server redis-6388.conf
[root@slave1 redis_cluster]# ./../src/redis-server redis-6389.conf

　　　　加入集群：

[root@slave1 redis_cluster]# ./../src/redis-trib.rb add-node 192.168.11.202:6388 192.168.11.202:6382
#将6389添加成6388的从节点
[root@slave1 redis_cluster]# ./../src/redis-trib.rb add-node --slave --master-id e073db09e7aaed3c20d133726a26c8994932262c 192.168.11.202:6389 192.168.11.202:6382

　　迁移槽和数据

　　　　　　采用redis-trib-rb reshard命令执行槽重分片：[root@slave1 redis_cluster]# ./../src/redis-trib.rb reshard 192.168.11.202:6382

　　　　　　当出现 How many slots do you want to move (from 1 to 16384)? 提示我们想移动多少个槽，我们输入4096

　　　　　　当出现What is the receiving node ID? 提示我们哪个主节点接收新移动的槽， 我们输入6388的节点id：e073db09e7aaed3c20d133726a26c8994932262c，目标节点只能指定一个（节点id可以拷贝的哦）

　　　　　　之后输入源节点的id，用done结束，这里我用的all，就是从之前的全部主节点中移动4096个槽到6388

　　　　　　数据迁移之前会打印出所有的槽从源节点到目标节点的计划，确认无误后输入yes执行迁移工作

　　　　　　若迁移过程没有出错，那么迁移则顺利完成

　　cluster故障转移

　　　　6388上的所有key

192.168.11.202:6388> keys *
1) "list:names"
2) "name"
192.168.11.202:6388>

　　　　杀掉6388进程

[root@slave1 redis_cluster]# ps -ef | grep redis-server | grep 6388
root      8280     1  0 Mar05 ?        00:05:07 ./../src/redis-server 192.168.11.202:6388 [cluster]
[root@slave1 redis_cluster]# kill -9 8280

#集群节点查看
[root@slave1 redis_cluster]# ./../src/redis-cli -h 192.168.11.202 -p 6382 -a myredis -c
192.168.11.202:6382> cluster nodes
4f36b08d8067a003af45dbe96a5363f348643509 192.168.11.202:6386 slave 3771e67edab547deff6bd290e1a07b23646906ee 0 1520304517911 5 connected
0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 192.168.11.202:6382 myself,master - 0 0 1 connected 1394-5460
a583def1e6a059e4fdb3592557fd6ab691fd61ec 192.168.11.202:6387 slave 10b3789bb30889b5e6f67175620feddcd496d19e 0 1520304514886 6 connected
10b3789bb30889b5e6f67175620feddcd496d19e 192.168.11.202:6384 master - 0 1520304516904 3 connected 12318-16383
3771e67edab547deff6bd290e1a07b23646906ee 192.168.11.202:6383 master - 0 1520304513879 2 connected 7106-10922
e073db09e7aaed3c20d133726a26c8994932262c 192.168.11.202:6388 master,fail - 1520304485678 1520304484473 10 disconnected 0-1393 5461-7105 10923-12317
37de0d2dc1c267760156d4230502fa96a6bba64d 192.168.11.202:6389 slave e073db09e7aaed3c20d133726a26c8994932262c 0 1520304515895 10 connected
7649466ec006e0902a7f1578417247a6d5540c47 192.168.11.202:6385 slave 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 0 1520304518923 4 connected
192.168.11.202:6382> 

#查询key name
192.168.11.202:6382> get name
-> Redirected to slot [5798] located at 192.168.11.202:6389
"youzhibing"
192.168.11.202:6389> keys *
1) "list:names"
2) "name"
192.168.11.202:6389> 

　　　　　　6389已经成为主节点，承担着之前6388的角色，集群状态还是ok的， 对外提供的服务不受任何影响

　　　　重新启动6388服务

[root@slave1 redis_cluster]# ./../src/redis-server redis-6388.conf

#查看集群节点
192.168.11.202:6389> cluster nodes
0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 192.168.11.202:6382 master - 0 1520304789567 1 connected 1394-5460
37de0d2dc1c267760156d4230502fa96a6bba64d 192.168.11.202:6389 myself,master - 0 0 12 connected 0-1393 5461-7105 10923-12317
7649466ec006e0902a7f1578417247a6d5540c47 192.168.11.202:6385 slave 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 0 1520304788061 1 connected
a583def1e6a059e4fdb3592557fd6ab691fd61ec 192.168.11.202:6387 slave 10b3789bb30889b5e6f67175620feddcd496d19e 0 1520304787556 3 connected
3771e67edab547deff6bd290e1a07b23646906ee 192.168.11.202:6383 master - 0 1520304786550 2 connected 7106-10922
e073db09e7aaed3c20d133726a26c8994932262c 192.168.11.202:6388 slave 37de0d2dc1c267760156d4230502fa96a6bba64d 0 1520304786047 12 connected
10b3789bb30889b5e6f67175620feddcd496d19e 192.168.11.202:6384 master - 0 1520304785542 3 connected 12318-16383
4f36b08d8067a003af45dbe96a5363f348643509 192.168.11.202:6386 slave 3771e67edab547deff6bd290e1a07b23646906ee 0 1520304788561 2 connected
192.168.11.202:6389>

　　　　可以看到6388启动成功后，仍在集群中，只是是作为6389的从节点了

　　cluster收缩

　　　　1、我们下线6389和6388节点

　　　　通过集群节点信息我们知道6389负责槽：0-1393 5461-7105 10923-12317， 现在将0-1393迁移到6382，5461-7105迁移到6383， 10923-12317迁移到6384

How many slots do you want to move (from 1 to 16384)?1394
What is the receiving node ID? 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:37de0d2dc1c267760156d4230502fa96a6bba64d
Source node #2:done
......
Do you want to proceed with the proposed reshard plan (yes/no)? yes

How many slots do you want to move (from 1 to 16384)? 1645
What is the receiving node ID? 3771e67edab547deff6bd290e1a07b23646906ee
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:37de0d2dc1c267760156d4230502fa96a6bba64d
Source node #2:done
......
Do you want to proceed with the proposed reshard plan (yes/no)? yes

How many slots do you want to move (from 1 to 16384)?1395
What is the receiving node ID? 10b3789bb30889b5e6f67175620feddcd496d19e
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:37de0d2dc1c267760156d4230502fa96a6bba64d
Source node #2:done
......
Do you want to proceed with the proposed reshard plan (yes/no)? yes

　　　　槽节点迁移完之后，集群节点信息，发现6388已经没有分配槽了

192.168.11.202:6382> cluster nodes
3771e67edab547deff6bd290e1a07b23646906ee 192.168.11.202:6383 master - 0 1520333368013 16 connected 5461-10922
0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 192.168.11.202:6382 myself,master - 0 0 13 connected 0-5460
10b3789bb30889b5e6f67175620feddcd496d19e 192.168.11.202:6384 master - 0 1520333372037 17 connected 10923-16383
4f36b08d8067a003af45dbe96a5363f348643509 192.168.11.202:6386 slave 3771e67edab547deff6bd290e1a07b23646906ee 0 1520333370024 16 connected
a583def1e6a059e4fdb3592557fd6ab691fd61ec 192.168.11.202:6387 slave 10b3789bb30889b5e6f67175620feddcd496d19e 0 1520333370525 17 connected
37de0d2dc1c267760156d4230502fa96a6bba64d 192.168.11.202:6389 slave e073db09e7aaed3c20d133726a26c8994932262c 0 1520333369017 15 connected
7649466ec006e0902a7f1578417247a6d5540c47 192.168.11.202:6385 slave 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 0 1520333367008 13 connected
e073db09e7aaed3c20d133726a26c8994932262c 192.168.11.202:6388 master - 0 1520333371031 15 connected

　　　　2、忘记节点

　　　　由于集群内的节点不停地通过Gossip消息彼此交换节点信息，因此需要通过一种健壮的机制让集群内所有节点忘记下线的节点。也就是说让其他节点不再与要下线的节点进行Gossip消息交换。

　　　　利用redis-trib.rb del-node命令实现节点下线，先下线从节点再下线主节点，避免不必要的全量复制。命令如下

[root@slave1 redis_cluster]# ./../src/redis-trib.rb del-node 192.168.11.202:6389 37de0d2dc1c267760156d4230502fa96a6bba64d
[root@slave1 redis_cluster]# ./../src/redis-trib.rb del-node 192.168.11.202:6388 e073db09e7aaed3c20d133726a26c8994932262c

　　　　集群节点信息如下

192.168.11.202:6382> cluster nodes
3771e67edab547deff6bd290e1a07b23646906ee 192.168.11.202:6383 master - 0 1520333828887 16 connected 5461-10922
0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 192.168.11.202:6382 myself,master - 0 0 13 connected 0-5460
10b3789bb30889b5e6f67175620feddcd496d19e 192.168.11.202:6384 master - 0 1520333827377 17 connected 10923-16383
4f36b08d8067a003af45dbe96a5363f348643509 192.168.11.202:6386 slave 3771e67edab547deff6bd290e1a07b23646906ee 0 1520333826880 16 connected
a583def1e6a059e4fdb3592557fd6ab691fd61ec 192.168.11.202:6387 slave 10b3789bb30889b5e6f67175620feddcd496d19e 0 1520333829892 17 connected
7649466ec006e0902a7f1578417247a6d5540c47 192.168.11.202:6385 slave 0ec055f9daa5b4f570e6a4c4d46e5285d16e0afe 0 1520333827879 13 connected

　　　　16384个槽节点都有分布，集群状态ok， 节点6389和6388下线成功

注意点

　　1、创建集群的时候，redis-trib.rb会尽可能保证主从节点不分配在同一机器下，因此会重新排序节点顺序；节点列表顺序用于确定主从角色，先主节点之后是从节点

　　2、redis-trib.rb创建集群的时候，节点地址必须是不包含任何槽 / 数据的节点，否则会拒绝创建集群

　　3、虚拟槽的采用主要是针对一致性哈希分区的不足而提出的，一致性哈希分区不适用少量节点的情况，而虚拟槽的范围（redis cluster槽范围是0 ~ 16383）一般远大于节点数的，然后每个节点负责一定数量的槽，这样就规避掉了少量节点的问题，因为在数据与节点之间多了一层虚拟槽的映射

　　4、Jedis连接redis cluster的时候，配置redis-cluster节点的时候只需要配置任意一个可达的节点即可，不一定全部节点都配置上，因为每个节点都有整个集群的信息；Jedis键命令执行流程如下图， 有兴趣的朋友可以查看Jedis源码； 当然，全部节点都配置是更全面的做法

　　5、集群的伸缩与故障转移对客户端没有影响，只要整个集群状态是ok，那么客户端的请求都是能够得到正常响应的

　　6、只有保证16384个槽节点都能分配到节点上，那么集群状态就是ok，才能正常对外提供服务；所以 无论是集群扩容还是收缩，都必须保证16384个槽能正确的分配到节点上

参考

　　《Redis开发与运维》

　　Redis 集群教程