Reconfiguring a mongodb replicaset after a "loading local.system.replset config (LOADINGCONFIG)" error

Weird thing happen today when I came to work my MongoDB replicaset was not working anymore after a system update

There was a mongodb instance running but when I checked for the replicaset status I had this error message :

> rs.status()

{
        "startupStatus" : 1,
        "ok" : 0,
        "errmsg" : "loading local.system.replset config (LOADINGCONFIG)"
}

Now I have no idea what caused the problem but I could see (from the error message) that it was related to the replicaset configuration

The problem was actually pretty weird since when I printed the replicaset config it showed an old configuration

> rs.config()
{
        "_id" : "rps",
        "version" : 2,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.168.13.200:27017"
                },
                {
                        "_id" : 1,
                        "host" : "192.168.13.201:27017"
                }
        ]
}

Since this is a development environment I didn't have a database backup so I really needed the replicaset to start running again

Thankfully MongoDB (since version 2.0) offers a relatively easy way to reconfigure a replicaset that is the rs.reconfig() command

In my case what I had to to was to reconfigure my replicaset with the only surviving member (itself) since the second one was under heavy maintenance.

This operation can be done in 6 steps :

1. Get the current replicaset configuration into a variable

> var cfg = rs.config()

2. Overwrite the members property with the remaining replicaset nodes

> cfg.members = [{"_id" : 3, "host"  : "192.168.1.100"}]
[ { "_id" : 3, "host" : "192.168.1.100" } ]

3. Reconfigure the replicaset with the new configuration

> rs.reconfig(cfg , {force : true})
{
        "msg" : "will try this config momentarily, try running rs.conf() again in a few seconds",
        "ok" : 1
}

4. Wait for the configuration to be applied

Give it a few seconds and run again the conf command to see if the new configuration is properly applied

> rs.conf()
{
        "_id" : "rps",
        "version" : 77682,
        "members" : [
                {
                        "_id" : 3,
                        "host" : "192.168.1.100:27017"
                }
        ]
}

5. Restart your mondod instance

/etc/init.d/mongod restart

6. Check that everything is running properly

rps:PRIMARY> rs.status()
{
        "set" : "rps",
        "date" : ISODate("2013-05-23T07:54:47Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 3,
                        "name" : "192.168.1.100:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 8,
                        "optime" : {
                                "t" : 1352365425,
                                "i" : 1
                        },
                        "optimeDate" : ISODate("2012-11-08T09:03:45Z"),
                        "self" : true
                }
        ],
        "ok" : 1
}

Sources :

Reconfigure a Replica Set with Unavailable Members