The Blight

Recently we were tasked with some implementation tasks which required that we do more stringent lab testing than normal. Since our lab was out of date, and basically out of order, we decided to try and pull some replicas of servers in our live environment into a virtual environment.

This would ensure that the lab was as up to date as possible as well as giving you the added benefit of the ease of backing up or snapshotting when doing a change.

We decided to use Vmware Server and Vmware Converter for this task. This decision was made because of the ease of using Vmware Converter. I won’t go into detail on the processes of conversion but Vmware Converter allows you to create a virtual machine from a live machine without any reboots or interference. Some colleagues have actually used this tool to virtualise a production MS SQL box while it was servicing requests. Not only did the virtual work, but there was no performance impact to the server during the process.

Now onto the main reason for this post – what needs to be done once the virtual servers are running?

1. Give the domain controllers static IP addresses.
2. Fix DNS

• Create a reverse zone for the lab IP range and update the PTR records for the domain controllers in this zone
• Point the servers to themselves as DNS servers or DNS will not start correctly.
• Fix any stubzones for child domains in the lab.

1. Create or modify a site in “Sites and Services” to include the new IP range in the lab.
2. Run a cleanup for all the other domain controllers which were not virtualised.

• This is done using ntdsutil scripted in the following fashion for each DC:

  Ntdsuti; “metadata cleanup” “remove selected server DNFoundUsingAdsiEdit” quit quit

• Repeat the above line in a batch file for each Domain Controller DN found under your Configuration container in AdsiEdit.

This brings me to the biggest problem experienced:

Obviously the process of virtualising Domain Controllers results in 1 or more (Depending on how many you virtualise) being out of date from a replication point of view. You might think that this would not be a problem and the domain controller should just replicate once they are able to communicate. The problem is that the domain controllers keep a set of USN (Update Sequence Numbers) which allow them to keep track of changes on domain controllers. Each DC stores a USN for each of the other DCs in the domain. So – when a DC tries to replicate with another DC and that DC believes that the USN number for the DC is out of date, it forces the DC with the older USN to go into a USN Rollback state and basically shuts it down from partaking in replication and other functions. This is actually a good thing and is there to protect your domain from bad restores etc.

A USN rollback is evident by the following errors in your DS log:

Message 1

Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 2095
Date:
Time:
User:
Computer:
Description: During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers. Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC. If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations. To determine if this misconfiguration exists, query this event ID using http://support.microsoft.com or contact your Microsoft product support. The most probable cause of this situation is the improper restore of Active Directory on the local domain controller. User Actions: If this situation occurred because of an improper or unintended restore, forcibly demote the DC. Remote DC: GUID Partition: CN=Configuration,DC=usn,DC=loc USN reported by Remote DC: 24707 USN reported by Local DC: 20485 For more information, see Help and Support Center at http://support.microsoft.com.

Message 2

Event Type: Warning
Event Source: NTDS General
Event Category: Replication
Event ID: 1113
Date:
Time:
User:
Computer:
Description: Inbound replication has been disabled by the user. For more information, see Help and Support Center at http://support.microsoft.com.

Message 3

Event Type: Warning
Event Source: NTDS General
Event Category: Replication
Event ID: 1115
Date: 3/10/2005
Time:
User:
Computer:
Description: Outbound replication has been disabled by the user. For more information, see Help and Support Center at http://support.microsoft.com

Message 4

Event Type: Error
Event Source: NTDS General
Event Category: Service Control
Event ID: 2103
Date: 3/10/2005
Time:
User:
Computer:
Description: The Active Directory database has been restored using an unsupported restoration procedure. Active Directory will be unable to log on users while this condition persists. As a result, the Net Logon service has paused. User Action See previous event logs for details. For more information, see Help and Support Center at http://support.microsoft.com.

So how to get around this in the lab when we know it is not a problem? Much reading and posting had me thinking that it wasn’t possible, when finally I stumbled on something which showed that if you did a restore, it would overwrite this feature and allow you to force replication to the out of date DC.

So basically what you do is to reboot the virtual DC into Directory Recovery Mode (by pressing F8 during the boot sequence) and then use NTBackup to restore an older system state backup to the server. Once you reboot, the server will again try to replicate.

You may still have another problem (Depending on the date of your backup) and this is lingering objects. The server won’t replicate as it believe it has some out of date information. Now because this is a lab, you can safely ignore this error and enable the replication of lingering objects using the following registry key on all of the domain controllers in the lab after restarting the netlogon service.

1. Locate and click the following registry key:
   

   HKEY_LOCAL_MACHINESystemCurrentControlSetServicesNTDSParameters
   

2. Click Add Value on the Edit menu.
   
3. Add the following value:
   

   Value Name: Strict Replication Consistency
   Data type: REG_DWORD
   Value data: If the value is 1, change it to 0.
   

You may also need to re-enable inbound and outbound replication using repadmin:

repadmin /options DC_Name -disable_inbound_repl -disable_outbound_repl

Now you should see that replication starts to take place as normal – it will obviously take some time to get all changes replicated.

Hope this helps somebody – lab testing is one of the most important things!