Did you replace the Domino Directory using the wrong template ? Here is how to fix it

I few weeks ago I was working with one of my customers and suddenly some users couldn't access the servers. I opened the Domino Directory (names.nsf) and what I saw ? Yes, the Domino Directory was using the Mail template. Also, after a few seconds, at least 40 servers replicated the design, so, we had more than 40 servers around the world using the wrong template.

The administrators there wanted to shut down all the servers and start copying manually the names.nsf from the last backup (for some reason - 2 days ago), they called me and this is what I did. After everything was done, most of the users were not aware of what happened.

This is the procedure:

1. Stop the Replica in every server - The wrong design won't be replicated
2. Stop the Router in every server - Messages will be kept in the mail.box
3. At the console level, execute: load convert -d names.nsf * pubnames.ntf. It will replace the design using the right template, since Convert is actually a Mail Conversion Utility, the -d parameter will avoid the $Inbox folder creation.
4. Execute Step 3 in every server.
5. Start the router in every server
6. Start the replica in every server
7. Restart servers asap.

According to what I have seen, when a Domino Directory starts using the wrong template, this will happen:

• Users can't access. There is no $Users view - Users can't be authenticated. Users already authenticated will work with no problems for some time unless they are accessing the Domino Directory either directly or some applications is doing @DBColumns or anything like that to the names.nsf.
  
• Router can't deliver messages
  

Stop the servers and restore the last valid names.nsf is a valid process, however it takes some time and when you have more than 70 servers around the world you will need a faster solution.

As a final recommendation, review your Domino Directory replication settings. The Administration Server (or hub servers) shouldn't allow incoming design changes from any server.

Finally, I wanted to post this solution after one of Paul Mooney's session at Lotusphere 2008.