How do I reconfigure MongoDB replication?

Problem

How do I reconfigure MongoDB replication?

Solution

Load at the current configuration:

appdata:SECONDARY> cfg = rs.conf()
{
	"_id" : "appdata",
	"version" : 3,
	"members" : [
		{
			"_id" : 0,
			"host" : "appdata1.old.company.com:27017"
		},
		{
			"_id" : 1,
			"host" : "appdata2.old.company.com:27017"
		},
		{
			"_id" : 2,
			"host" : "appdata3.old.company.com:27017"
		}
	]
}
appdata:SECONDARY> 

Change the variable, using the new replica set members:

appdata:SECONDARY> cfg.members = [{"_id":0, "host":"appdata1.company.com:27017"},{"_id":1, "host":"appdata2.company.com:27017"},{"_id":2, "host":"appdata3.company.com:27017"}]
[
	{
		"_id" : 0,
		"host" : "appdata1.company.com:27017"
	},
	{
		"_id" : 1,
		"host" : "appdata2.company.com:27017"
	},
	{
		"_id" : 2,
		"host" : "appdata3.company.com:27017"
	}
]
appdata:SECONDARY> 

Load the new configuration:

appdata:SECONDARY> rs.reconfig(cfg , {force : true})
{ "ok" : 1 }
appdata:SECONDARY> 

Now repeat the same commands on each mongo slave.

Testing

Set a key on the primary:

appdata:PRIMARY> db.appdata.insert({test1:"test1"})
appdata:PRIMARY>

…and try to read it from a slave:

appdata:SECONDARY> db.appdata.find()
{ "_id" : ObjectId("54739f6d2b039af98a8bc075"), "test1" : "test1" }
appdata:SECONDARY> 

View the status:

appdata:PRIMARY>  rs.status()
{
	"set" : "userdata1",
	"date" : ISODate("2014-11-24T21:09:05Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "appdata1.company.com:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 7155,
			"optime" : Timestamp(1416862864000, 1),
			"optimeDate" : ISODate("2014-11-24T21:01:04Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "appdata2.company.com:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 144,
			"optime" : Timestamp(1416862864000, 1),
			"optimeDate" : ISODate("2014-11-24T21:01:04Z"),
			"lastHeartbeat" : ISODate("2014-11-24T21:09:05Z"),
			"pingMs" : 0
		},
		{
			"_id" : 2,
			"name" : "appdata3.company.com:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 481,
			"optime" : Timestamp(1416862864000, 1),
			"optimeDate" : ISODate("2014-11-24T21:01:04Z"),
			"lastHeartbeat" : ISODate("2014-11-24T21:09:04Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}
appdata:PRIMARY> 

Troubleshooting

If you see “exception: assertion src/mongo/db/repl/rs_initiate.cpp:119”:

appdata:SECONDARY> rs.reconfig(cfg , {force : true})
{
	"errmsg" : "exception: assertion src/mongo/db/repl/rs_initiate.cpp:119",
	"code" : 0,
	"ok" : 0
}
appdata:SECONDARY> 

You may see this while reconfiguring one of the slaves:

appdata:SECONDARY> rs.reconfig(cfg , {force : true})
{
	"errmsg" : "exception: member appdata1.company.com:27017 has a config version >= to the new cfg version; cannot change config",
	"code" : 13341,
	"ok" : 0
}
appdata:SECONDARY> 

..make sure that all the replica members are up and running, and their TCP ports are accessible. If the problem persists, reconfigure the primary node with only itself, then add the other slaves:

appdata:SECONDARY> cfg = rs.conf()
{
	"_id" : "appdata",
	"version" : 151288,
	"members" : [
		{
			"_id" : 0,
			"host" : "appdata1.company.com:27017"
		},
		{
			"_id" : 1,
			"host" : "appdata2.company.com:27017"
		},
		{
			"_id" : 2,
			"host" : "appdata3.company.com:27017"
		}
	]
}
appdata:SECONDARY> cfg.members = [{"_id":0, "host":"appdata1.company.com:27017"}]
[ { "_id" : 0, "host" : "appdata1.company.com:27017" } ]
appdata:SECONDARY> rs.reconfig(cfg , {force : true})
{ "ok" : 1 }
appdata:SECONDARY> cfg = rs.conf()
{
	"_id" : "appdata",
	"version" : 236238,
	"members" : [
		{
			"_id" : 0,
			"host" : "appdata1.company.com:27017"
		}
	]
}
appdata:PRIMARY> rs.add("appdata2.company.com:27017")
{ "ok" : 1 }
appdata:PRIMARY> rs.add("appdata3.company.com:27017")
{ "down" : [ "appdata3.company.com:27017" ], "ok" : 1 }
appdata:PRIMARY> 

I’m seeing “not master and slaveOk=false” when I try to read from a slave:

appdata:SECONDARY> db.appdata.find()
error: { "$err" : "not master and slaveOk=false", "code" : 13435 }
appdata:SECONDARY> 

On each of the slaves, turn on slave reads:

appdata:SECONDARY> rs.slaveOk()
appdata:SECONDARY>

Slaves are stuck “RECOVERING”, and status shows “still syncing, not yet to minValid optime”:

appdata:RECOVERING> db.appdata.find()
error: {
	"$err" : "not master or secondary; cannot currently read from this replSet member",
	"code" : 13436
}
appdata:RECOVERING> 

Status display:

appdata:PRIMARY> rs.status()
{
.
.
.
		{
			"_id" : 1,
			"name" : "appdata2.company.com:27017",
			"health" : 1,
			"state" : 3,
			"stateStr" : "RECOVERING",
			"uptime" : 445,
			"optime" : Timestamp(1415688615000, 4),
			"optimeDate" : ISODate("2014-11-11T06:50:15Z"),
			"lastHeartbeat" : ISODate("2014-11-26T01:51:24Z"),
			"pingMs" : 0,
			"errmsg" : "still syncing, not yet to minValid optime 54752053:1"
		},
.
.
.

You’ll need to Resync the replica set member.

Share: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Twitter
  • Facebook
  • Google Bookmarks
  • LinkedIn
  • Reddit
  • StumbleUpon

Leave a Reply

Your email address will not be published. Required fields are marked *