Redis4.0.14 迁槽失败

线上一个redis集群中主节点使用的内存达到了9.78g，按照redis单个实例最大内存不要超出10g的规范，扩容操作就放在了今天晚上进行。因为之前redis迁槽都是采用 redis-trib.rb reshard xxx.xxx.xxx.xxx:8001 的方式进行，今晚准备采用 redis-trib.rb reshard --from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3 --to 139ccdd0dfdf757950cb9a45b812bad898eb6b6e --slots 2730 --yes --timeout 10000 --pipeline 100 xxx.xxx.xxx.xxx:8001 的方式进行，就先在测试环境跑上一把，很遗憾，测试环境运行失败！！

在命令运行过程中，出现了以下错误:

Moving slot 8183 from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
Moving slot 8184 from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
Moving slot 8185 from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
Moving slot 8186 from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
Moving slot 8187 from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
Moving slot 8188 from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
Moving slot 8189 from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
Moving slot 8190 from 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
Moving slot 5461 from 192.168.96.14:8002@18002 to 192.168.96.14:8001:
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

OK，那就查看以下集群状态吧:

S: e68d2e6a96c154ac6927059fc0264da98fd0f73b 192.168.96.14:8006@18006
slots: (0 slots) slave
replicates d5a545398f54d03d9ac2289d37430f48f472d525
S: 1e020b85eb2ee077e0d2de3a9b70c9ea6a30c156 192.168.96.14:8004@18004
slots: (0 slots) slave
replicates 139ccdd0dfdf757950cb9a45b812bad898eb6b6e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 192.168.96.14:8001 has slots in importing state (5461).
[WARNING] Node 192.168.96.14:8002@18002 has slots in migrating state (5461).
[WARNING] The following slots are open: 5461
>>> Check slots coverage...
[OK] All 16384 slots covered.

好吧，集群状态报错，那就先把集群恢复原状：

1. redis-trib.rb fix xxx.xx.xx.xxx:8001

该命令不怎么好使啊:

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 192.168.96.14:8001 has slots in importing state (5461).
[WARNING] Node 192.168.96.14:8002@18002 has slots in migrating state (5461).
[WARNING] The following slots are open: 5461
>>> Fixing open slot 5461
Set as migrating in: 192.168.96.14:8002@18002
Set as importing in: 192.168.96.14:8001
Moving slot 5461 from 192.168.96.14:8002@18002 to 192.168.96.14:8001:
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

2. 没办法了，在 WARNING 的两个节点上分别执行 cluster setslot 5461 stable，然后再check集群状态，这次正常了，如果不行，就再fix一次。

集群状态虽然恢复了，但是扩容还是要做的，再在测试环境执行一次，还是同样的错误，这就有点尴尬了。

再尴尬问题还是要解决的，那就先看看这个报错的5461槽位中有哪些key，以及这些key的状态：

[root@localhost data]# redis-cli -p 8002
127.0.0.1:8002> CLUSTER GETKEYSINSLOT 5461 100
1) "key4290"
127.0.0.1:8002>
127.0.0.1:8002> DEBUG OBJECT key4290
Value at:0x7f0823b6f360 refcount:1 encoding:embstr serializedlength:11 lru:794344 lru_seconds_idle:3689

可以看到，这个key也没有什么不正常的，继续查原因，既然自动的不行，那就手动迁槽试试，看看到底那里出的问题

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

redis迁槽流程：

1. 对目标节点发送 cluster setslot {slot} importing {sourceNodeId}命令，让目标节点准备导入槽的数据

2. 对源节点发送  cluster setslot {slot} migrating {targetNodeId}命令，让源节点准备迁出槽的数据

3. 源节点循环执行cluster getkeysinslot {slot} {count}命令，获取count个属于槽 slot的键

4. 在源节点上执行 migrate {targetIp} {targetPort} "" 0 {timeout} keys {keys ....} 命令，把获取的键通过pipeline的机制批量迁移至目标节点

5. 重复3、4步直到槽下所有的键值数据都迁移至目标节点

6. 向集群中的所有主节点发送 cluster setslot {slot} node {targetNodeId}命令，通知槽分配给目标节点。

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

按照redis迁槽流程一步一步执行，终于在执行第4步的时候报了个错误：

[root@localhost logs]# redis-cli -p 8002
127.0.0.1:8002> cluster setslot 5461 migrating 139ccdd0dfdf757950cb9a45b812bad898eb6b6e
OK
127.0.0.1:8002> cluster getkeysinslot 5461 100
1) "key4290"
127.0.0.1:8002> migrate 192.168.96.14 8001 "" 0 5000 keys key4290
(error) ERR Target instance replied with error: NOREPLICAS Not enough good slaves to write.

貌似找到原因了，再去target节点上一看：

[root@localhost data]# redis-cli -p 8001
127.0.0.1:8001> cluster setslot 5461 importing 61e615f22e5e338ed8d0e107ac5d1f80b41cabc3
OK
127.0.0.1:8001> config get min-slaves-to-write
1) "min-slaves-to-write"
2) "2"

果然，min-slaves-to-write设置的是2，而测试环境的redis集群是3主3从的结构，调整 min-slaves-to-write的值，将其设置为0，再跑一遍，手动迁槽成功，但是redis-trib.rb操作还是报一样的错误，醉了

最后，同事说是不是redis-trib.rb有问题，好，换一台机器跑redis-trib.rb命令，成功。