豌豆夹Redis解决方案Codis安装使用

1.安装

1.1 Golang环境

Golang的安装非常简单，因为官网被墙，可以从国内镜像如studygolang.com下载。

[root@vm root]$ tar -C /usr/local -zxf go1.4.2.linux-amd64.tar.gz

[root@vm root]$ vim /etc/profile

export GOROOT=/usr/local/go

export PATH=$GOROOT/bin:$PATH

export GOPATH=/home/user/go

[root@vm root]$ source /etc/profile

[root@vm root]$ go version

注意：操作系统的软件仓库如apt-get或yum安装的Golang版本可能比较老，要卸载后重新手动安装。以我的Linux Mint为例，安装时apt-get install golang指定的其实是个meta，卸载时要指定apt-get remove golang-go才能卸载干净。

1.2 Codis源码包

GitHub上的官方安装指导是这样写的：

[root@vm root]$ go get -d github.com/wandoulabs/codis

[root@vm root]$ cd $GOPATH/src/github.com/wandoulabs/codis

[root@vm root]$ ./bootstrap.sh

[root@vm root]$ make gotest

这种方式会从GitHub上master下载最新的代码，但实践发现有时master代码可能会有问题。所以保险起见，还是下载稳定的release包吧。

1.3 编译安装

下载stable包后，我这里用的是最新的2.0.0版本包。手动创建出go的下载目录，将源码包解压后放进去，这样就手动完成了通过go get下载源码一步，避免用master分支代码的不稳定性。

[root@vm root] mkdir -p /home/root/go/src/github.com/wandoulabs/codis

[root@vm root] tar -xzvf ...

[root@vm root] cd codis

[root@vm codis] find . -name "configure" | xargs chmod +x

[root@vm codis] find . -name "*.sh" | xargs chmod +x

[root@vm codis] ./bootstrap.sh

    CC hyperloglog.o

    CC latency.o

    CC sparkline.o

    CC slots.o

    CC redis-cli.o

    CC redis-benchmark.o

    CC redis-check-dump.o

    CC redis-check-aof.o

    LINK redis-benchmark

    LINK redis-check-dump

    LINK redis-check-aof

    LINK redis-server

    INSTALL redis-sentinel

    LINK redis-cli

Hint: It's a good idea to run 'make test' ;)

make[2]: Leaving directory `/home/root/go/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src'

make[1]: Leaving directory `/home/root/go/src/github.com/wandoulabs/codis/extern/redis-2.8.21'

ln -s codis-server-2.8.21 bin/codis-server

go test ./pkg/... ./cmd/... -race

?       github.com/wandoulabs/codis/pkg/env     [no test files]

ok      github.com/wandoulabs/codis/pkg/models  5.057s

?       github.com/wandoulabs/codis/pkg/proxy/group     [no test files]

ok      github.com/wandoulabs/codis/pkg/proxy/redis     3.061s

ok      github.com/wandoulabs/codis/pkg/proxy/router    17.916s

?       github.com/wandoulabs/codis/pkg/proxy/router/topology   [no test files]

?       github.com/wandoulabs/codis/pkg/utils   [no test files]

?       github.com/wandoulabs/codis/pkg/utils/assert    [no test files]

?       github.com/wandoulabs/codis/pkg/utils/atomic2   [no test files]

ok      github.com/wandoulabs/codis/pkg/utils/bytesize  0.025s

?       github.com/wandoulabs/codis/pkg/utils/errors    [no test files]

?       github.com/wandoulabs/codis/pkg/utils/log       [no test files]

?       github.com/wandoulabs/codis/pkg/utils/trace     [no test files]

?       github.com/wandoulabs/codis/cmd/cconfig [no test files]

?       github.com/wandoulabs/codis/cmd/proxy   [no test files]

2.启动

之前参照官网上的“流程”一节，被“坑”苦了…鼓捣了老半天也不行，要么就是Dashboard端口不对了，要么就是Slot初始化不了了，都要放弃了。发现正确的步骤在codis/sample/usage.md里，作者都写好了启动脚本。

[root@vm sample] ./start_dashboard.sh

[root@vm sample] ./start_redis.sh

[root@vm sample] ./add_group.sh

[root@vm sample] ./initslot.sh

[root@vm sample] ./start_proxy.sh

[root@vm sample] ./set_proxy_online.sh

首先启动Dashboard，然后启动四个Redis实例，为这四个Redis实例创建四个Group，初始化Slot，启动一个Proxy实例，将Proxy置为Online状态。其中像新建Group、置为Online这种操作其实在Dashboard上都能做，但作者造福大家都给准备好了。一气呵成啊！刚才还自己瞎鼓捣了半天…

3.观察ZooKeeper

Codis使用ZooKeeper做协调，下面就看一下ZooKeeper里存的到底都是些什么。

Codis会根据我们在config.ini中配置的product_name作为ZooKeeper中的路径。

[zk: localhost:2181(CONNECTED) 34] ls /zk/codis/db_test

[fence, slots, servers, proxy, migrate_tasks, dashboard, LOCK, actions, ActionResponse]

3.1 Slot信息

前面initslots初始化的所有Slot都保存在slots路径下，每个Slot是一个结点。

[zk: localhost:2181(CONNECTED) 35] ls /zk/codis/db_test/slots

[slot_240, slot_241, slot_242, slot_244, slot_243, slot_246, slot_245, slot_248, slot_247, slot_249, slot_804, slot_803, slot_802, slot_801, slot_808, slot_807, slot_806, slot_805, slot_800, slot_230, slot_231,

                        ...     ...     ...

 slot_832, slot_286, slot_672, slot_675, slot_674, slot_670, slot_671, slot_4, slot_3, slot_2, slot_1, slot_8, slot_7, slot_6, slot_5, slot_9, slot_0, slot_600, slot_602, slot_601, slot_608, slot_607, slot_609, slot_604, slot_603, slot_606, slot_605, slot_205, slot_206, slot_203, slot_204, slot_201, slot_202, slot_200, slot_209, slot_207, slot_208]

随便选取一个Slot结点，用get命令能够查看结点上附着的数据：

[zk: localhost:2181(CONNECTED) 8] get /zk/codis/db_test/slots/slot_208

{"product_name":"test","id":208,"group_id":5,"state":{"status":"online","migrate_status":{"from":-1,"to":-1},"last_op_ts":"0"}}

3.2 Proxy信息

[zk: localhost:2181(CONNECTED) 40] ls /zk/codis/db_test/proxy

[proxy_1]

[zk: localhost:2181(CONNECTED) 12] get /zk/codis/db_test/proxy/proxy_1

{"id":"proxy_1","addr":"BC-VM-edce4ac67d304079868c0bb265337bd4:19000","last_event":"","last_event_ts":0,"state":"online","description":"","debug_var_addr":"BC-VM-edce4ac67d304079868c0bb265337bd4:11000","pid":2946,"start_at":"2015-07-03 16:13:53.277706979 +0800 CST"}

3.3 Server信息

Server的信息保存的只是Redis实例的IP地址和端口，所以并没有以JSON的形式附着在路径结点上，而是直接作为路径的一部分保存了。

[zk: localhost:2181(CONNECTED) 43] ls /zk/codis/db_test/servers

[group_1, group_2, group_3, group_4]

[zk: localhost:2181(CONNECTED) 44] ls /zk/codis/db_test/servers/group_1

[localhost:6380]

[zk: localhost:2181(CONNECTED) 45] ls /zk/codis/db_test/servers/group_2

[localhost:6381]

[zk: localhost:2181(CONNECTED) 46] ls /zk/codis/db_test/servers/group_3

[localhost:6382]

[zk: localhost:2181(CONNECTED) 47] ls /zk/codis/db_test/servers/group_4

[localhost:6383]

3.4 其他信息

ZooKeeper中还保存了很多其他信息，这里就不一一解释了。

[zk: localhost:2181(CONNECTED) 53] ls /zk/codis/db_test/fence

[BC-VM-edce4ac67d304079868c0bb265337bd4:19000]

[zk: localhost:2181(CONNECTED) 24] get /zk/codis/db_test/dashboard

{"addr": "localhost:18087", "pid": 2871}

[zk: localhost:2181(CONNECTED) 33] get /zk/codis/db_test/actions/0000003698

{"type":"slot_changed","desc":"","target":{"product_name":"test","id":941,"group_id":5,"state":{"status":"online","migrate_status":{"from":-1,"to":-1},"last_op_ts":"0"}},"ts":"1435914788","receivers":["{\"id\":\"proxy_1\",\"addr\":\"BC-VM-edce4ac67d304079868c0bb265337bd4:19000\",\"last_event\":\"\",\"last_event_ts\":0,\"state\":\"online\",\"description\":\"\",\"debug_var_addr\":\"BC-VM-edce4ac67d304079868c0bb265337bd4:11000\",\"pid\":2946,\"start_at\":\"2015-07-03 16:13:53.277706979 +0800 CST\"}"]}

4.测试

4.1 简单测试

因为Codis兼容大部分Redis操作，所以这里用原版Redis客户端连接到Proxy，测试一下保存和查询。

[root@vm redis-2.8.19] src/redis-cli -p 19000

127.0.0.1:19000> set person1 cdai

OK

127.0.0.1:19000> set student2 hank

OK

127.0.0.1:19000> set teacher3 carter

OK

127.0.0.1:19000> set manager4 rachel

OK

127.0.0.1:19000> get person1

"cdai"

127.0.0.1:19000> get student2

"hank"

127.0.0.1:19000> get teacher3

"carter"

127.0.0.1:19000> get manager4

"rachel"

4.2 自动Rebalance

新添加一个Redis实例6379，并在Dashboard中为其新建Group 5，将新实例添加进去。

[root@vm sample] cp redis_conf/6380.conf redis_conf/6379.conf

[root@vm redis_conf] diff redis_conf/6379.conf redis_conf/6380.conf

41c41

< pidfile /var/run/redis_6379.pid

---

> pidfile /var/run/redis.pid

45c45

< port 6379

---

> port 6380

177c177

< dbfilename 6379.rdb

---

> dbfilename 6380.rdb

[root@vm sample] nohup ../bin/codis-server ./redis_conf/6379.conf &> ./log/redis_6379.log &

重新balance之前先看一下目前四个key的分布情况：

6380保存manager4和student2
6381什么都没保存
6382保存teacher3
6383保存person1

[root@vm redis-2.8.19] src/redis-cli

127.0.0.1:6379> keys *

(empty list or set)

127.0.0.1:6379> quit

[root@vm redis-2.8.19] src/redis-cli -p 6380

127.0.0.1:6380> keys *

1) "manager4"

2) "student2"

127.0.0.1:6380> quit

[root@vm redis-2.8.19] src/redis-cli -p 6381

127.0.0.1:6381> keys *

(empty list or set)

127.0.0.1:6381> quit

[root@vm redis-2.8.19] src/redis-cli -p 6382

127.0.0.1:6382> keys *

1) "teacher3"

127.0.0.1:6382> quit

[root@vm redis-2.8.19] src/redis-cli -p 6383

127.0.0.1:6383> keys *

1) "person1"

127.0.0.1:6383> quit

现在点”Auto Balance”看刚才保存的四个key会发生什么。

在Migration过程中，ZooKeeper中的内容也有变化。/zk/codis/db_test/migrate_tasks保存了正在migrating的Slot：

[zk: localhost:2181(CONNECTED) 61] ls /zk/codis/db_test/migrate_tasks

[0000000079, 0000000078, 0000000082, 0000000083, 0000000080, 0000000081, 0000000086, 0000000087, 0000000084, 0000000085, 0000000118, 0000000119, 0000000116, 0000000117, 0000000114, 0000000115, 0000000112, 0000000113, 0000000110, 0000000111, 0000000120, 0000000200, 0000000201, 0000000202, 0000000107, 0000000203, 0000000108, 0000000109, 0000000103, 0000000104, 0000000105, 0000000106, 0000000100, 0000000101, 0000000102, 0000000152, 0000000153, 0000000150, 0000000151, 0000000146, 0000000145, 0000000144, 0000000143, 0000000149, 0000000148, 0000000147, 0000000160, 0000000161, 0000000162, 0000000163, 0000000164, 0000000155, 0000000154, 0000000157, 0000000156, 0000000159, 0000000158, 0000000130, 0000000131, 0000000128, 0000000127, 0000000126, 0000000125, 0000000124, 0000000123, 0000000122, 0000000121, 0000000129, 0000000140, 0000000141, 0000000142, 0000000137, 0000000136, 0000000139, 0000000138, 0000000133, 0000000132, 0000000135, 0000000134, 0000000193, 0000000192, 0000000191, 0000000190, 0000000197, 0000000196, 0000000195, 0000000194, 0000000189, 0000000187, 0000000188, 0000000198, 0000000199, 0000000175, 0000000174, 0000000173, 0000000172, 0000000171, 0000000170, 0000000169, 0000000099, 0000000167, 0000000168, 0000000165, 0000000166, 0000000098, 0000000184, 0000000097, 0000000183, 0000000096, 0000000186, 0000000095, 0000000185, 0000000094, 0000000180, 0000000093, 0000000092, 0000000182, 0000000091, 0000000181, 0000000090, 0000000176, 0000000177, 0000000088, 0000000178, 0000000089, 0000000179]

迁移后的key分配是：

[root@vm redis-2.8.19] src/redis-cli

127.0.0.1:6379> keys *

1) "student2"

127.0.0.1:6379> quit

[root@vm redis-2.8.19] src/redis-cli -p 6380

127.0.0.1:6380> keys *

1) "manager4"

127.0.0.1:6380> quit

[root@vm redis-2.8.19] src/redis-cli -p 6381

127.0.0.1:6381> keys *

(empty list or set)

127.0.0.1:6381> quit

[root@vm redis-2.8.19] src/redis-cli -p 6382

127.0.0.1:6382> keys *

1) "teacher3"

127.0.0.1:6382> quit

[root@vm redis-2.8.19] src/redis-cli -p 6383

127.0.0.1:6383> keys *

1) "person1"

127.0.0.1:6383> quit