perform-maintence-on-replica-set-members

https://docs.mongodb.com/v3.0/tutorial/perform-maintence-on-replica-set-members/

1 oplog 改变大小 --详见mongodb log
2 repl 成员维护
　　1 Stop a secondary
　　db.shutdownServer()
　　2 Restart the secondary as a standalone on a different port
　　mongod --port 37017 --dbpath /srv/mongodb
　　3 Perform maintenance operations on the secondary.While the member is a standalone, use the mongo shell to perform maintenance
　　mongo --port 37017
　　4 Restart mongod as a member of the replica set.
　　db.shutdownServer()
　　rs.status()
　　5 Perform maintenance on the primary last.
　　rs.stepDown(300)
3 Force a Member to Become Primary
　　Force a Member to be Primary by Setting its Priority High
　　This procedure assumes your current primary is m1.example.net and that you’d like to instead make m3.example.net primary. The procedure also assumes you have a three-member replica set with the configuration below.
　　This procedure assumes this configuration:
　　{
　　"_id" : "rs",
　　"version" : 7,
　　"members" : [
　　　　{
　　　　　　"_id" : 0,
　　　　　　"host" : "m1.example.net:27017"
　　　　},
　　　　{
　　　　　　"_id" : 1,
　　　　　　"host" : "m2.example.net:27017"
　　　　},
　　　　{
　　　　　　"_id" : 2,
　　　　　　"host" : "m3.example.net:27017"
　　　　}
　　　　]
　　}
　　1 In a mongo shell connected to the primary, use the following sequence of operations to make m3.example.net the primary:
　　cfg = rs.conf()
　　cfg.members[0].priority = 0.5
　　cfg.members[1].priority = 0.5
　　cfg.members[2].priority = 1
　　rs.reconfig(cfg)
　　The last statement calls rs.reconfig() with the modified configuration document to configure m3.example.net to have a higher replSetGetConfig.members[n].priority value than the other mongod instances.

db.adminCommand({replSetStepDown: 86400, force: 1})

　　rs.freeze()
　　rs.freeze() Makes the current replica set member ineligible to become primary for the period specified.使当前副本集成员不具有在指定期间成为主要成员的资格
　　rs.stepDown() Forces the primary of the replica set to become a secondary, triggering an election for primary.
　　Force a Member to be Primary Using Database Commands
　　mdb0.example.net - the current primary.
　　mdb1.example.net - a secondary.
　　mdb2.example.net - a secondary .
　　In a mongo shell, run rs.status() to ensure your replica set is running as expected.
　　In a mongo shell connected to the mongod instance running on mdb2.example.net, freeze mdb2.example.net so that it does not attempt to become primary for 120 seconds.
　　rs.freeze(120)
　　In a mongo shell connected the mongod running on mdb0.example.net, step down this instance that the mongod is not eligible to become primary for 120 seconds:
　　rs.stepDown(120)
　　mdb1.example.net becomes primary.

4 Resync a Member of a Replica Set//重新同步复制集群
　　MongoDB provides two options for performing an initial sync:// 2种方法，1全部重新同步 2 copy数据，然后同步
　　Restart the mongod with an empty data directory and let MongoDB’s normal initial syncing feature restore the data. This is the more simple option but may take longer to replace the data.See Automatically Sync a Member.
　　Restart the machine with a copy of a recent data directory from another member in the replica set. This procedure can replace the data more quickly but requires more manual steps.See Sync by Copying Data Files from Another Member.
　　1 Automatically Sync a Member
　　During initial sync, mongod removes the contents of the dbPath directory. //在初始同步的时候，会删除db路径下的内容
　　同步会影响primary节点，增加流量，如果复制集没有数据，可以使用添加节点方式，加入该节点
　　可以强制一个已经是复制集群的节点，进行重新同步
　　1 关闭节点，2 删除数据目录（最好备份一份）3重启这个节点 ---这时mongodb会initial sync
　　2 Sync by Copying Data Files from Another Member //cp数据文件进行同步
　　Copy the Data Files，您可以将数据文件捕获为快照或直接复制，但是，在大多数情况下，您不能将数据文件从运行的mongod实例复制到另一个实例中，因为数据文件在文件复制操作期间会发生变化。
　　If copying data files, ensure that your copy includes the content of the local database.
　　You cannot use a mongodump backup for the data files: only a snapshot backup.
　　Sync the Member

5 Configure Replica Set Tag Sets
　　Tag sets let you customize write concern and read preferences for a replica set.
　　Differences Between Read Preferences and Write Concerns
　　{
　　"_id" : "rs0",
　　"version" : 1,
　　"members" : [
　　{
　　　　"_id" : 0,
　　　　"host" : "mongodb0.example.net:27017"
　　},
　　{
　　　　"_id" : 1,
　　　　"host" : "mongodb1.example.net:27017"
　　},
　　{
　　　　"_id" : 2,
　　　　"host" : "mongodb2.example.net:27017"
　　}
　　]
}
　　You could add tag sets to the members of this replica set with the following command sequence in the mongo shell:
　　conf = rs.conf()
　　conf.members[0].tags = { "dc": "east", "use": "production" }
　　conf.members[1].tags = { "dc": "east", "use": "reporting" }
　　conf.members[2].tags = { "use": "production" }
　　rs.reconfig(conf)
　　IMPORTANT：In tag sets, all tag values must be strings.

　　Custom Multi-Datacenter Write Concerns
6 Reconfigure a Replica Set with Unavailable Members
　　This document provides steps for re-configuring a replica set when only a minority of members are accessible
　　Reconfigure by Forcing the Reconfiguration
　　Use this procedure only to recover from catastrophic interruptions. Do not use force every time you reconfigure. Also, do not use the force option in any automatic scripts and do not use force when there is still a primary.
　　To force reconfiguration:
　　1 Back up a surviving member.
　　2 Connect to a surviving member and save the current configuration. Consider the following example commands for saving the configuration:
　　cfg = rs.conf()
　　printjson(cfg)
　　cfg.members = [cfg.members[0] , cfg.members[4] , cfg.members[7]]
　　rs.reconfig(cfg, {force : true})

7 Manage Chained Replication
　　for example, if a secondary selects its replication target based on ping time and if the closest member is another secondary.
　　MongoDB enables chained replication by default.
　　1 Copy the configuration settings into the cfg object:
　　　　cfg = rs.config()
　　2 WARNING :To avoid data loss, skip this step if the configuration settings contain the settings embedded document.
　　　　If the current configuration settings do not contain the settings embedded document, create the embedded document by issuing the following command:
　　　　cfg.settings = { }
　　3 Issue the following sequence of commands to set settings.chainingAllowed to false:
　　　　cfg.settings.chainingAllowed = false
　　　　rs.reconfig(cfg)

　　Re-enable Chained Replication
　　cfg = rs.config()
　　cfg.settings.chainingAllowed = true
　　rs.reconfig(cfg)

8 Change Hostnames in a Replica Set
　　members[n].host
　　Change hostnames without disrupting availability.
　　1 For each secondary in the replica set, perform the following sequence of operations:
　　Stop the secondary.
　　Restart the secondary at the new location.
　　Open a mongo shell connected to the replica set’s primary. In our example, the primary runs on port 27017 so you would issue the following command:

　　mongo --port 27017
　　cfg = rs.conf()
　　cfg.members[1].host = "mongodb1.example.net:27017"
　　rs.reconfig(cfg)
　　Open a mongo shell connected to the primary and step down the primary using the rs.stepDown() method:
　　rs.stepDown()
　　The replica set elects another member to the become primary.
　　When the step down succeeds, shut down the old primary.
　　Start the mongod instance that will become the new primary in the new location.
　　Connect to the current primary, which was just elected, and update the replica set configuration document with the hostname of the node that is to become the new primary.

　　2 Stop all members running on the old hostnames at once.
　　Change All Hostnames at the Same Time
　　Stop all members in the replica set.
　　Restart each member on a different port and without using the --replSet run-time option.
　　mongod --dbpath /data/db1/ --port 37017 --bind_ip localhost,<ip address of the mongod host>
　　Consider the following sequence of commands to change the hostnames in a three-member set:
　　use local
　　cfg = db.system.replset.findOne( { "_id": "rs" } )
　　cfg.members[0].host = "mongodb0.example.net:27017"
　　cfg.members[1].host = "mongodb1.example.net:27017"
　　cfg.members[2].host = "mongodb2.example.net:27017"
　　db.system.replset.update( { "_id": "rs" } , cfg )
　　Stop the mongod process on the member.
　　After re-configuring all members of the set, start each mongod instance in the normal way: use the usual port number and use the --replSet option. For example:
　　mongod --dbpath /data/db1/ --port 27017 --replSet rs --bind_ip localhost,<ip address of the mongod host>
　　To confirm the new configuration, call rs.conf() in the mongo shell.

　　database0.example.com:27017 (the primary)
　　database1.example.com:27017
　　database2.example.com:27017
　　>>
　　mongodb0.example.net:27017 (the primary)
　　mongodb1.example.net:27017
　　mongodb2.example.net:27017

9 Configure a Secondary’s Sync Target

----------------
1 Adjust Priority for Replica Set Member
--调整复制集群的优先级，优先级越高的成员，将成为primary
　　To block a member from seeking election as primary, assign it a priority of 0. Hidden members and delayed members have priority set to 0.
　　Starting in MongoDB 3.6, arbiters have priority 0.
　　cfg = rs.conf()//赋值给一个变量
　　cfg.members[0].priority = 0.5//修改其优先级
　　cfg.members[1].priority = 2
　　cfg.members[2].priority = 2
　　rs.reconfig(cfg) //应用修改配置
2 Prevent Secondary from Becoming Primary
　　MongoDB does not permit the current primary to have a priority of 0. To prevent the current primary from again becoming a primary, you must first step down the current primary using rs.stepDown().
　　cfg = rs.conf()
　　To prevent a secondary member from becoming a primary, update the secondary member’s members[n].priority to 0
　　cfg.members[2].priority = 0
　　rs.reconfig(cfg)

3 Configure a Hidden Replica Set Member
　　Hidden members are part of a replica set but cannot become primary and are invisible to client applications.
　　Hidden members may vote in elections.
　　{
　　"_id" : <num>
　　"host" : <hostname:port>,
　　"priority" : 0,
　　"hidden" : true
　　}
　　cfg = rs.conf()
　　cfg.members[0].priority = 0
　　cfg.members[0].hidden = true
　　rs.reconfig(cfg)

4 Configure a Delayed Replica Set Member
　　When you configure a delayed member, the delay applies both to replication and to the member’s oplog.
　　cfg = rs.conf()
　　cfg.members[0].priority = 0
　　cfg.members[0].hidden = true
　　cfg.members[0].slaveDelay = 3600//The following example sets a 1-hour delay on a secondary member currently at the index 0 in the members array
　　rs.reconfig(cfg)

5 Configure Non-Voting Replica Set Member
　　Non-voting members allow you to add additional members for read distribution beyond the maximum seven voting members.
　　cfg = rs.conf();
　　cfg.members[3].votes = 0;
　　cfg.members[3].priority = 0;
　　cfg.members[4].votes = 0
　　cfg.members[4].priority = 0;
　　cfg.members[5].votes = 0
　　cfg.members[5].priority = 0;
　　rs.reconfig(cfg);

6 Convert a Secondary to an Arbiter¶
　　Convert Secondary to Arbiter and Reuse the Port Number
　　If your application is connecting directly to the secondary, modify the application so that MongoDB queries don’t reach the secondary.
　　Shut down the secondary
　　Remove the secondary from the replica set by calling the rs.remove() method. Perform this operation while connected to the current primary in the mongo shell:
　　rs.remove("<hostname><:port>")
　　rs.conf()//Verify that the replica set no longer includes the secondary by calling the rs.conf() method in the mongo shell:
　　Move the secondary’s data directory to an archive folder. For example:
　　mv /data/db /data/db-old
　　mkdir /data/db
　　Restart the mongod instance for the secondary, specifying the port number, the empty data directory, and the replica set.
　　mongod --port 27021 --dbpath /data/db --replSet rs --bind_ip localhost,<ip address of the mongod host>
　　In the mongo shell convert the secondary to an arbiter using the rs.addArb() method:
　　rs.addArb("<hostname><:port>")
　　rs.conf()
　　Convert Secondary to Arbiter Running on a New Port Number¶
　　Create a new, empty data directory to be used with the new port number. For example:
　　mkdir /data/db-temp
　　Start a new mongod instance on the new port number, specifying the new data directory and the existing replica set. Issue a command similar to the following:
　　mongod --port 27021 --dbpath /data/db-temp --replSet rs --bind_ip localhost,<ip address of the mongod host>
　　In the mongo shell connected to the current primary,
　　rs.addArb("<hostname><:port>")
　　rs.conf()
　　Shut down the secondary.
　　rs.remove("<hostname><:port>")
　　rs.conf()
　　mv /data/db /data/db-old
--------------------
　　1 Deploy a Replica Set
　　rs.initiate( {
　　_id : "rs0",
　　members: [
　　　　{ _id: 0, host: "mongodb0.example.net:27017" },
　　　　{ _id: 1, host: "mongodb1.example.net:27017" },
　　　　{ _id: 2, host: "mongodb2.example.net:27017" }
　　　　]
　　})
　　rs.conf()
2 Deploy a Replica Set for Testing and Development
　　Starting in MongoDB 3.6, MongoDB binaries, mongod and mongos, bind to localhost (127.0.0.1) by default.
　　IP Binding
　　For example, the following mongod instance binds to both the localhost and the sample ip address 198.51.100.1:
　　mongod --bind_ip localhost,198.51.100.1
　　In order to connect to this instance, remote clients must specify the ip address 198.51.100.1 or the hostname associated with the ip address:
　　mongo --host 198.51.100.1
　　mongo --host My-Example-Associated-Hostname
　　In this test deployment, the three members run on the same machine
　　mkdir -p /srv/mongodb/rs0-0 /srv/mongodb/rs0-1 /srv/mongodb/rs0-2
　　mongod --replSet rs0 --port 27017 --bind_ip localhost,<ip address of mongod host> --dbpath /srv/mongodb/rs0-0 --smallfiles --oplogSize 128
　　mongod --replSet rs0 --port 27018 --bind_ip localhost,<ip address of mongod host> --dbpath /srv/mongodb/rs0-1 --smallfiles --oplogSize 128
　　mongod --replSet rs0 --port 27019 --bind_ip localhost,<ip address of mongod host> --dbpath /srv/mongodb/rs0-2 --smallfiles --oplogSize 128
　　mongo --port 27017
　　rsconf = {
　　_id: "rs0",
　　members: [
　　{
　　　　_id: 0,
　　　　host: "<hostname>:27017"
　　},
　　{
　　　　_id: 1,
　　　　host: "<hostname>:27018"
　　},
　　{
　　　　_id: 2,
　　　　host: "<hostname>:27019"
　　}
　　]
}
　　rs.initiate( rsconf )
　　rs.conf()

3 Deploy a Geographically Redundant Replica Set
　　rs.initiate( {
　　_id : "rs0",
　　members: [
　　　　{ _id: 0, host: "mongodb0.example.net:27017" },
　　　　{ _id: 1, host: "mongodb1.example.net:27017" },
　　　　{ _id: 2, host: "mongodb2.example.net:27017" },
　　　　{ _id: 3, host: "mongodb3.example.net:27017" },
　　　　{ _id: 4, host: "mongodb4.example.net:27017", arbiterOnly: true }
　　]
　　})

4 Add an Arbiter to Replica Set
　　rs.addArb("m1.example.net:27017")
5 Convert a Standalone to a Replica Set
　　Shut down the standalone mongod instance.
　　Restart the instance. Use the --replSet option to specify the name of the new replica set.
6 Add Members to a Replica Set
　　mongod --dbpath /srv/mongodb/db0 --replSet rs0
　　mongod --config /etc/mongod.conf
　　rs.add( { host: "mongodb3.example.net:27017", priority: 0, votes: 0 } )
　　rs.status()

　　var cfg = rs.conf();
　　cfg.members[4].priority = 1
　　cfg.members[4].votes = 1
　　rs.reconfig(cfg)
7 Remove Members from Replica Set
　　Remove a Member Using rs.remove()
　　Shut down the mongod instance for the member you wish to remove.
　　use db.isMaster() while connected to any member of the replica set.
　　Use rs.remove() in either of the following forms to remove the member
　　rs.remove("mongod3.example.net:27017")
　　rs.remove("mongod3.example.net")
　　Remove a Member Using rs.reconfig()
　　cfg = rs.conf()
　　cfg.members.splice(2,1)
　　rs.reconfig(cfg)

8 Replace a Replica Set Member
　　cfg = rs.conf()
　　cfg.members[0].host = "mongo2.example.net"
　　rs.reconfig(cfg)
------------
1 Write Concern for Replica Sets
　　For example, the following method includes a write concern that specifies that the method return only after the write propagates to the primary and at least one secondary or the method times out after 5 seconds.
　　db.products.insert(
　　{ item: "envelopes", qty : 100, type: "Clasp" },
　　{ writeConcern: { w: 2, wtimeout: 5000 } } //w=2 至少一个second返回给primary，超时时间5秒
　　)
　　Modify Default Write Concern
　　cfg = rs.conf()
　　cfg.settings.getLastErrorDefaults = { w: "majority", wtimeout: 5000 }//waits for the write operation to complete on a majority of the voting members before returning:
　　rs.reconfig(cfg)
2 Read Preference//读写分离
　　By default, an application directs its read operations to the primary member in a replica set.
　　In general, do not use secondary and secondaryPreferred to provide extra capacity for reads, because:
　　All members of a replica have roughly equivalent write traffic; as a result, secondaries will service reads at roughly the same rate as the primary.
　　Replication is asynchronous and there is some amount of delay between a successful write operation and its replication to secondaries.//复制异步
　　如果集合中的任何成员变得不可用，则将读操作分发到二级，就会影响可用性，因为该集合的其余成员需要能够处理所有的应用程序请求。
　　对于sharcollection集合的查询，对于具有平衡器的集群来说，由于不完整或终止的块迁移，次级值可能会由于缺少或重复的数据而返回陈旧的结果。
　　Sharding通过在一组机器上分布读和写操作来增加读写能力，并且通常是增加容量的更好策略。
　　primary Default mode. All operations read from the current replica set primary.
　　primaryPreferred In most situations, operations read from the primary but if it is unavailable, operations read from secondary members.
　　secondary All operations read from the secondary members of the replica set.
　　secondaryPreferred In most situations, operations read from secondary members but if no secondary members are available, operations read from the primary.
　　nearest Operations read from member of the replica set with the least network latency, irrespective of the member’s type.
　　In the mongo shell, the readPref() cursor method provides access to read preferences
　　maxStalenessSeconds
　　Replica set members can lag behind the primary due to network congestion, low disk throughput, long-running operations, etc.
　　The read preference maxStalenessSeconds option lets you specify a maximum replication lag, or “staleness”, for reads from secondaries.
　　When a secondary’s estimated staleness exceeds maxStalenessSeconds, the client stops using it for read operations.
3 Server Selection Algorithm

--------
Replica Set High Availability
Replica sets provide high availability using automatic failover. Failover allows a secondary member to become primary if the current primary becomes unavailable.
--------
　　Replica Set Deployment Architectures
　　Strategies//策略
　　Determine the Number of Members //确定成员的数量
　　Maximum Number of Voting Members//最大投票成员
　　A replica set can have up to 50 members, but only 7 voting members. [1] If the replica set already has 7 voting members, additional members must be non-voting members.
　　Deploy an Odd Number of Members //部署奇数个成员
　　In general, avoid deploying more than one arbiter per replica set.// 避免在一个复制集中部署多个仲裁节点
　　Consider Fault Tolerance //考虑容错
　　Number of Members Majority Required to Elect a New Primary Fault Tolerance
　　3 2 1
　　4 3 1
　　5 3 2
　　6 4 2
　　Use Hidden and Delayed Members for Dedicated Functions //使用隐藏的和延迟,Add hidden or delayed members to support dedicated functions, such as backup or reporting.
　　Load Balance on Read-Heavy Deployments// 在一个读比较高的系统，可以使用分散读取second
　　Add Capacity Ahead of Demand//在需求之前增加容量
　　Distribute Members Geographically//分配成员的地理位置
　　Target Operations with Tag Sets
　　Use Journaling to Protect Against Power Failures//使用日志记录来防止电源故障MongoDB enables journaling by default.
　　Replica Set Naming
　　Deployment Patterns //部署模式

Three Member Replica Sets
　　Three-member replica sets provide the minimum recommended architecture for a replica set.
　　One primary.
　　Two secondary members. Both secondaries can become the primary in an election.
　　A three member replica set with a two members that store data has:
　　One primary.
　　One secondary member. The secondary can become primary in an election.
　　One arbiter. The arbiter only votes in elections
　　Since the arbiter does not hold a copy of the data, these deployments provides only one complete copy of the data.
　　Replica Sets Distributed Across Two or More Data Centers

------------
Replica Set Data Synchronization
　　Initial Sync
　　Initial sync copies all the data from one member of the replica set to another member.
　　When you perform an initial sync, MongoDB:
　　1 Clones all databases except the local database. To clone, the mongod scans every collection in each source database and inserts all data into its own copies of these collections.
　　3.4版本mongodb Initial sync builds all collection indexes as the documents are copied for each collection.先前的版本只会同步only the _id indexes are built during this stage.
　　3.4版本，Initial sync pulls newly added oplog records during the data copy. 要有足够的oplog空间
　　2 Applies all changes to the data set. Using the oplog from the source, the mongod updates its data set to reflect the current state of the replica set.
　　Fault Tolerance 容错
　　To recover from transient network or operation failures, initial sync has built-in retry logic. 同步异常，再次同步会根据logic再次同步
　　Replication
　　Secondary members replicate data continuously after the initial sync. Secondary members copy the oplog from their sync from source and apply these operations in an asynchronous process.
　　Secondaries may automatically change their sync from source as needed based on changes in the ping time and state of other members’ replication.
　　Secondaries avoid syncing from delayed members and hidden members.
　　If a secondary member has members[n].buildIndexes set to true, it can only sync from other members where buildIndexes is true.
　　Members where buildIndexes is false can sync from any other member, barring other sync restrictions. buildIndexes is true by default.
　　buildIndexes为true的只能从buildIndexes为true的节点上同步复制
　　Multithreaded Replication 多线程复制
　　MongoDB applies write operations in batches using multiple threads to improve concurrency. mongodb使用多线程来提高并发处理

　　Pre-Fetching Indexes to Improve Replication Throughput //预取所以可以提高吞吐量

--------
Replica Set Oplog
local.oplog.rs
Oplog Size
For Unix and Windows systems
The default oplog size depends on the storage engine:
Storage Engine Default Oplog Size Lower Bound Upper Bound
In-Memory Storage Engine 5% of physical memory 50 MB 50 GB
WiredTiger Storage Engine 5% of free disk space 990 MB 50 GB
MMAPv1 Storage Engine 5% of free disk space 990 MB 50 GB

The following workloads might require a larger oplog size. //大量的dml操作
Updates to Multiple Documents at Once
Deletions Equal the Same Amount of Data as Inserts
Significant Number of In-Place Updates