mongodb 分片群集(sharding cluster)

实际环境架构

分别在3台机器运行一个mongod实例（称为mongod shard11，mongod shard12，mongod shard13）组织replica set1，作为cluster的shard1

分别在3台机器运行一个mongod实例（称为mongod shard21，mongod shard22，mongod shard23）组织replica set2，作为cluster的shard2

每台机器运行一个mongod实例，作为3个config server

每台机器运行一个mongs进程，用于客户端连接

-------------------------------------------------------------------------

Server1  10.3.0.100

Mongod shard11:27017

Mongod shard21:17017

Mongod config1:20000

Mongos1:20001

-------------------------------------------------------------------------

Server1 10.3.0.101

Mongod shard12:27017

Mongod shard22:17017

Mongod config1:20000

Mongos1:20001

--------------------------------------------------------------------------

Server3 10.3.0.102

Mongod shard13:27017

Mongod shard23:17017

Mongod config1:20000

Mongos1:20001

-------------------------------------------------------------------------

1. 创建用户 mongodb (如果用root启用mongodb,可略此步)

useradd -u 1001 mongodb

2. 下载 mongodb-linux-x86_64-2.2.0

tar zxvf mongodb-linux-x86_64-2.2.0.tgz

mv mongodb-linux-x86_64-2.2.0 /opt/local/mongodb

3. 创建 数据目录 日志目录 配置目录 (server 1)

mkdir -p /opt/local/mongodb/data/shard

mkdir -p /opt/local/mongodb/data/logs

mkdir -p /opt/local/mongodb/data/config

4. 创建 数据目录 日志目录 配置目录 (server 2)

mkdir -p /opt/local/mongodb/data/shard12

mkdir -p /opt/local/mongodb/data/shard22

mkdir -p /opt/local/mongodb/data/logs

mkdir -p /opt/local/mongodb/data/config

5. 创建 数据目录 日志目录 配置目录 (server 3)

mkdir -p /opt/local/mongodb/data/shard13

mkdir -p /opt/local/mongodb/data/shard23

mkdir -p /opt/local/mongodb/data/logs

mkdir -p /opt/local/mongodb/data/config

6. 修改 目录 所有者

chown -R mongodb:mongodb /opt/local/mongodb

7. 配置relica sets

7.1.1 配置shard1所用到的replica sets:

--------------------------------------------------------------------------------------

port=27017                                                  #端口号

fork=true                                                   #以守护进程的方式运行，创建服务器进程

logpath=/opt/local/mongodb/data/logs/shard.log              #日志输出文件路径

logappend=true                                              #日志输出方式

dbpath=/opt/local/mongodb/data/shard/                       #数据库路径

shardsvr=true                                               #设置是否分片

maxConns=10000                                              #数据库的最大连接数

replSet=shard1                                              #设置副本集名称

oplogSize=5000                                              #设置oplog的大小(MB)

----------------------------------------------------------------------------------------

server 1

/opt/local/mongodb/bin/mongod --shardsvr --maxConns 10000 --replSet shard1 --port 27017 --dbpath /opt/local/mongodb/data/shard11/ --oplogSize 500 --logpath=/opt/local/mongodb/data/logs/shard11.log --logappend --fork

server 2

/opt/local/mongodb/bin/mongod --shardsvr --maxConns 10000 --replSet shard1 --port 27017 --dbpath /opt/local/mongodb/data/shard12/ --oplogSize 500 --logpath=/opt/local/mongodb/data/logs/shard12.log --logappend --fork

server 3

/opt/local/mongodb/bin/mongod --shardsvr --maxConns 10000 --replSet shard1 --port 27017 --dbpath /opt/local/mongodb/data/shard13/ --oplogSize 500 --logpath=/opt/local/mongodb/data/logs/shard13.log --logappend --fork

7.1.2 初始化replica set

用mongo连接其中一个mongod，执行:

/opt/local/mongodb/bin/mongo --host 10.3.0.100:27017

----------------------------------------------------------------------

> config= {_id: 'shard1', members: [ {_id:0,host:'10.3.0.100:27017'},

... {_id:1,host:'10.3.0.101:27017'},

... {_id:2,host:'10.3.0.102:27017'},]

... }

{

"_id" : "shard1",

"members" : [

{

"_id" : 0,

"host" : "10.3.0.100:27017"

},

{

"_id" : 1,

"host" : "10.3.0.101:27017"

},

{

"_id" : 2,

"host" : "10.3.0.102:27017"

}

]

}

> rs.initiate(config);

{

"info" : "Config now saved locally. Should come online in about a minute.",

"ok" : 1

}

-------------------------------------------------------------------------

7.2.1 配置shard2所用到的replica sets:

/opt/local/mongodb/bin/mongod --shardsvr --maxConns 10000 --replSet shard2 --port 17017 --dbpath /opt/local/mongodb/data/shard21/ --oplogSize 100 --logpath=/opt/local/mongodb/data/logs/shard21.log --logappend --fork

server 2

/opt/local/mongodb/bin/mongod --shardsvr --maxConns 10000 --replSet shard2 --port 17017 --dbpath /opt/local/mongodb/data/shard22/ --oplogSize 100 --logpath=/opt/local/mongodb/data/logs/shard22.log --logappend --fork

server 3

/opt/local/mongodb/bin/mongod --shardsvr --maxConns 10000 --replSet shard2 --port 17017 --dbpath /opt/local/mongodb/data/shard23/ --oplogSize 100 --logpath=/opt/local/mongodb/data/logs/shard23.log --logappend --fork

7.2.2 初始化replica set

用mongo连接其中一个mongod，执行:

/opt/local/mongodb/bin/mongo --host 10.3.0.100:17017

-------------------------------------------------------------------------

> config= {_id: 'shard2', members: [ {_id:0,host:'10.3.0.100:17017'},

... {_id:1,host:'10.3.0.101:17017'},

... {_id:2,host:'10.3.0.102:17017'},]

... }

{

"_id" : "shard2",

"members" : [

{

"_id" : 0,

"host" : "10.3.0.100:17017"

},

{

"_id" : 1,

"host" : "10.3.0.101:17017"

},

{

"_id" : 2,

"host" : "10.3.0.102:17017"

}

]

}

> rs.initiate(config);

{

"errmsg" : "couldn't initiate : set name does not match the set name host 10.3.0.101:17017 expects",

"ok" : 0

}

>

-------------------------------------------------------------------------

8. 配置三台config server

server 1

/opt/local/mongodb/bin/mongod --configsvr --dbpath /opt/local/mongodb/data/config/ --port 20000 --logpath /opt/local/mongodb/data/logs/config.log --logappend --fork

server 2

/opt/local/mongodb/bin/mongod --configsvr --dbpath /opt/local/mongodb/data/config/ --port 20000 --logpath /opt/local/mongodb/data/logs/config.log --logappend --fork

server 3

/opt/local/mongodb/bin/mongod --configsvr --dbpath /opt/local/mongodb/data/config/ --port 20000 --logpath /opt/local/mongodb/data/logs/config.log --logappend --fork

9. 配置 mongs

在server1,server2,server3 上分别执行：

/opt/local/mongodb/bin/mongos --configdb 10.3.0.100:20000,10.3.0.101:20000,10.3.0.102:20000 --port 20001 --chunkSize 5 --logpath /opt/local/mongodb/data/logs/mongos.log --logappend --fork

#mongs不需要dbpath

10 Configuring the Shard Cluster

连接到其中一个mongos进程，并切换到admin数据库做以下配置

10.1  连接到mongs，并切换到admin

/opt/local/mongodb/bin/mongo 10.3.0.100:20001/admin

MongoDB shell version: 2.2.0

connecting to: 10.3.0.100:20001/admin

mongos> db

admin

10.2  加入shards

如里shard是单台服务器，

用>db.runCommand( { addshard : “<serverhostname>[:<port>]” } )加入

如果shard是replica sets，用replicaSetName/<serverhostname>[:port]

----------------------------------------------------------------------

shard 1

db.runCommand( { addshard : "shard1/10.3.0.100:27017,10.3.0.101:27017,10.3.0.102:27017", maxsize:204800});

{ "shardAdded" : "shard1", "ok" : 1 }

--------------------------------------------------------------------

shard 2

mongos> db.runCommand( { addshard : "shard2/10.3.0.100:17017,10.3.0.101:17017,10.3.0.102:17017", maxsize:204800});

{ "shardAdded" : "shard2", "ok" : 1 }

----------------------------------------------------------------------

查看是否添加成功

mongos> db.runCommand( { listshards : 1 } )

{

"shards" : [

{

"_id" : "shard1",

"host" : "shard1/10.3.0.100:27017,10.3.0.101:27017,10.3.0.102:27017"

},

{

"_id" : "shard2",

"host" : "shard2/10.3.0.100:17017,10.3.0.101:17017,10.3.0.102:17017"

}

],

"ok" : 1

}

删除 shards

db.runCommand( { removeshard : "shard1/10.3.0.100:27017,10.3.0.101:27017"} );

11. 激活分片

命令：

> db.runCommand( { enablesharding : “<dbname>” } );

通过执行以上命令，可以让数据库跨shard，如果不执行这步，数据库只会存放在一个shard，一旦激活数据库分片，数据库中不同的collection将被存放在不同的shard上，但一个collection仍旧存放在同一个shard上，要使单个collection也分片，

还需单独对collection作些操作

PS: 其他命令.....操作...

#增添节点:

以一个新建的mongodb服务为例,

连接 mongo

/opt/local/mongodb/bin/mongo --host 10.3.0.100:27017

MongoDB shell version: 2.2.0

connecting to: 10.3.0.100:27017/test

shard1:PRIMARY>rs.add(“IP:port”);

#删除节点：

必须在主节点上操作：

PRIMARY>rs.remove(“127.0.0.1:27020”)

查看同步状态

rs.status()

让从有读的权限：

在主控上执行:PRIMARY> db.getMongo().setSlaveOk();

SECONDARY> rs.slaveOk();

群集增加验证 auth  keyFile

群集如果加了 auth 验证，群集之前互相取不到 主 就无法验证...必须要增加 keyFile 验证才行...

先创建auth 验证

-----------------------------------------------------------------

> use admin

switched to db admin

> db.addUser('sa','sa')

{

"_id" : ObjectId("4e2914a585178da4e03a16c3"),

"user" : "sa",

"readOnly" : false,

"pwd" : "75692b1d11c072c6c79332e248c4f699"

}

>

----------------------------------------------------------------------------------

然后在每个server里创建 key 文件....

vi /opt/local/mongodb/data/config/key

输入随意字符....  1234567890111111111

授权 chown mongodb:mongodb key

chmod 600 key

-------------------------------------------------------------------------

然后分别启动

server 1

/opt/local/mongodb/bin/mongod --shardsvr --replSet shard1 --port 27017 --dbpath /opt/local/mongodb/data/shard11/ --oplogSize 100 --logpath=/opt/local/mongodb/data/logs/shard11.log --logappend --fork --keyFile /opt/local/mongodb/data/config/key

server 2

/opt/local/mongodb/bin/mongod --shardsvr --replSet shard1 --port 27017 --dbpath /opt/local/mongodb/data/shard12/ --oplogSize 100 --logpath=/opt/local/mongodb/data/logs/shard12.log --logappend --fork --keyFile /opt/local/mongodb/data/config/key

server 3

/opt/local/mongodb/bin/mongod --shardsvr --replSet shard1 --port 27017 --dbpath /opt/local/mongodb/data/shard13/ --oplogSize 100 --logpath=/opt/local/mongodb/data/logs/shard13.log --logappend --fork --keyFile /opt/local/mongodb/data/config/key

-------------------------------------------------------------------------

在启动config

server 1

/opt/local/mongodb/bin/mongod --configsvr --dbpath /opt/local/mongodb/data/config/ --port 20000 --logpath /opt/local/mongodb/data/logs/config.log --logappend --fork --keyFile /opt/local/mongodb/data/config/key

server 2

/opt/local/mongodb/bin/mongod --configsvr --dbpath /opt/local/mongodb/data/config/ --port 20000 --logpath /opt/local/mongodb/data/logs/config.log --logappend --fork --keyFile /opt/local/mongodb/data/config/key

server 3

/opt/local/mongodb/bin/mongod --configsvr --dbpath /opt/local/mongodb/data/config/ --port 20000 --logpath /opt/local/mongodb/data/logs/config.log --logappend --fork --keyFile /opt/local/mongodb/data/config/key

-------------------------------------------------------------------------

最后分别启动mongos

/opt/local/mongodb/bin/mongos --configdb 10.3.0.100:20000,10.3.0.101:20000,10.3.0.102:20000 --port 20001 --chunkSize 5 --logpath /opt/local/mongodb/data/logs/mongos.log --logappend --fork --keyFile /opt/local/mongodb/data/config/key

-------------------------------------------------------------------------

然后在查看是否能读取主....

./mongo

MongoDB shell version: 2.2.0

connecting to: test

> show dbs

Wed Sep 5 01:51:44 uncaught exception: listDatabases failed:{ "errmsg" : "need to login", "ok" : 0 }

> use admin

switched to db admin

> db.auth('sa','sa')

1

shard1:SECONDARY> db.serverStatus()

就可以查看....

注意:

添加auth 以后...导入数据 等一系列操作...都必须验证用户...

如：

/opt/local/mongodb/bin/mongorestore -u sa -p sa --drop /opt/1

其他都需要进行操作！否则！报错！！！

强制切换主

cfg = rs.conf()

cfg.members[0].priority = 0.5

cfg.members[1].priority = 0.5

cfg.members[2].priority = 1

rs.reconfig(cfg)

升级MongoDB分片集群到2.2版本

MongoDB 2.2版本与2.0版本在分片集群中共存存在问题，所以升级需要按照官方文档的步骤进行：

禁用Balancer

mongos> sh.setBalancerState(false)

首先升级所有的mongos

升级Config服务器,一次升级一台

升级一个分片的复制集

依次关闭并升级一个复制集的secondary

在Primary库上执行rs.stepDown()，强制下线。关闭Primary库并升级二进制程序。

当都升级完成后，连接mongos执行：sh.setBalancerState(true)恢复平衡器。

----------------------------------------------------------------------------------

WARNING: You are running on a NUMA machine.

We suggest launching mongod like this to avoid performance problems:

numactl --interleave=all mongod [other options]

sudo -u mongodb numactl --interleave=all /opt/local/mongodb/bin/mongod --shardsvr --maxConns 10000 --replSet shard1 --port 27017 --dbpath /opt/local/mongodb/data/shard16/ --oplogSize 1000 --logpath=/opt/local/mongodb/data/logs/shard16.log --logappend --fork

编辑内核  echo 0 > /proc/sys/vm/zone_reclaim_mode

----------------------------------------------------------------------------------

限制 mongodb 占用系统所有内存

ulimit -s 4096 && ulimit -m 31457280 && sudo -u mongodb numactl --interleave=all /opt/local/mongodb/bin/mongod --shardsvr --maxConns 10000 --replSet shard1 --port 27017 --dbpath /opt/local/mongodb/data/shard16/ --oplogSize 1000 --logpath=/opt/local/mongodb/data/logs/shard16.log --logappend --fork