DCDIAG comes to the rescue, as usual, when diagnosing some strange ForestPrep problems occurring during an upward migration.
At a recent consulting project, I was troubleshooting a problem that I thought I'd share with you, where Exchange ForestPrep was failing. This was happening, even though the Domain Controller and would-be Exchange server were communicating fine.
The company was upgrading a Windows 2000 Active Directory domain to Windows Server 2003, as well as Exchange from 2000 to 2003. (To keep this column short, I won't discuss the Exchange migration.) The new pristine forest for the company was working flawlessly. The two Domain Controllers in the domain were replicating properly and everything in the new domain seemed to be working as expected.
The server that was to be the Exchange server was installed as a member server in the new domain. I downloaded the latest version of Exchange Server Deployment Tools and used Microsoft's recommended checklist. It’s tempting to skip some of the steps on the checklist when you run the Exchange Server Deployment Tools and go straight to the Exchange installation process. If you are installing Exchange on a Domain Controller and you are logged in as an account that is a member of the Enterprise Admins, Schema Admins, Domain Administrators, and local Administrators group, you don’t even need to run ForestPrep or DomainPrep. Those two processes are executed automatically by the set-up program. In our scenario, the Exchange server was a member server, not a Domain Controller.
Some administrators like to go through the checklist listed in the deployment tools, even if they are installing Exchange directly on the Domain Controller just to ensure that they haven’t missed any steps, or simply to discover any problems before they start the Exchange installation. The checklist asks you to run diagnostic tools such as DCDIAG, the Domain Controller diagnostic utility, which may help you if a failure occurs while you're going through later steps on the checklist. In our scenario, DCDIAG failed with the following error (here, I'm using a fictitious server and domain name). This is only a small portion of the results from the DCDIAG utility. (For complete sample output, click here.)
Tech Help—Just An
Got a Windows, Exchange or virtualization question
or need troubleshooting help? Or maybe you want a better
explanation than provided in the manuals? Describe
your dilemma in an e-mail to the MCPmag.com editors
the best questions get answered in this column and garner
the questioner with a nifty MCPmag.com baseball-style
When you send your questions, please include your
full first and last name, location, certifications (if
any) with your message. (If you prefer to remain anonymous,
specify this in your message, but submit the requested
information for verification purposes.)
C:\Support Tools>dcdiag /s:FirstDC
Domain Controller Diagnosis
Performing initial setup:
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\ FirstDC
Starting test: Connectivity
The host 35b7169d-a567-45fbe-8e77-b3f6ef46501a._msdcs.
example.com could not be resolved to an IP address.
Check the DNS server, DHCP, server name, etc
........................ FIRSTDC failed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\FIRSTDC
Skipping all tests, because server FIRSTDC is
not responding to directory service requests
The results indicate some connectivity problems. However, the Exchange server and the root Domain Controller were able to ping each other by name, connect to shares, and communicate normally. All the services were running as expected. The primary tests performed by DCDIAG revealed that the Domain Controller was not responding to directory service requests. At this time, running ForestPrep failed with an error that complained about "insufficient rights" and that Schema not being modifiable. Not a very helpful message when you are logged in as a member of the Enterprise Admins group.
Another thing that was unusual from the night before when everything was working flawlessly was the error message that popped up when running Active Directory Domains and Trusts. The error indicated that the PDC Emulator Flexible Single Master Operation (FSMO) role cannot be located, even though all five FSMO roles were available on the root DC (FIRSTDC), which was up and running.
FSMO versus Operations Master
Technically there’s no such thing as FSMO (pronounced "fizz-mo") in Windows Server 2003 anymore because Microsoft years ago replace the term FSMO with the new term "Operations Master." They also updated (or tried anyway) and replaced the term in most places in the operating system and their documentation. However, there are lots of remnants of FSMO still left in Windows Server 2003. Several utilities (DCDIAG, NTDSUTIL, etc.) still use the outdated term. Perhaps it would have been better to leave FSMO and not confuse the administrators with two different terms. Until Microsoft figures out a way to clean the remnants of FSMO from Active Directory and registry, we're stuck with both.
It was time to look at the Event Viewer before I did anything else, or modified any configuration on the network. The Directory Service log in the Event Viewer on the Domain Controller had a few warnings where the source was listed as NTDS Replication.
[Click on image for larger view.]
Figure 1. Event Viewer warnings related to directory replication.
Compared to Windows 2000 Server, Microsoft has done much better job in Windows Server 2003 to populate content in the description area of the events that are listed in the Event Viewer. As Figure 2 shows, the description was very helpful in leading me to the exact cause of the problem. The warning in Event ID 2092 clearly indicated that the Domain Controller has not replicated since it has been started and, therefore, it's unable to validate the FSMO role. Although the description pointed to Infrastructure Master role, rather than PDC Emulator, at least it pointed me in the right direction.
This server is the owner of the following FSMO role, but does not consider it valid. For the partition which contains the FSMO, this server has not replicated successfully with any of its partners since this server has been restarted. Replication errors are preventing validation of this role.
Operations which require contacting a FSMO operation master will fail until this condition is corrected.
FSMO Role: CN=Infrastructure,DC=example,DC=com
I was also directed to the Knowledge Base article 305476, "Initial synchronization requirements for Windows 2000 Server and Windows Server 2003 operations master role holders," which was very helpful. For a complete text of the description of Event ID 2092, click here.
Figure 2. Event ID 2092 warning about directory replication.
It turned out that someone shut down the second Domain Controller. You might think that since the Exchange server was able to contact the root DC which held all the FSMO roles, you should be able to update the Schema. However, the problem was that the second Domain Controller was down and the first Domain Controller wasn’t able to replicate with it and confirm its FSMO role. Although, you could seize the roles on the Domain Controller that is running, you should only do that as a last resort, especially if the roles that need to be seized are the two forest-wide roles: Schema Master and Domain Naming Master.
As soon as I started the second Domain Controller, verified that Active Directory replication has taken place, and ran DCDIAG again, everything was fine. The ForestPrep and the DomainPrep ran without a hitch.
Have you run into similar situations with Exchange ForestPrep failures or Active Directory replication that you could share with other readers? How did you solve your particular problem? Share them with me at email@example.com and I'll publish the best of them in this column.
Zubair Alexander, MCSE, MCT, MCSA and Microsoft MVP is the founder of SeattlePro Enterprises, an IT training and consulting business. His experience covers a wide range of spectrum: trainer, consultant, systems administrator, security architect, network engineer, author, technical editor, college instructor and public speaker. Zubair holds more than 25 technical certifications and Bachelor of Science degrees in Aeronautics & Astronautics Engineering, Mathematics and Computer Information Systems. His Web site, www.techgalaxy.net, is dedicated to technical resources for IT professionals. Zubair may be reached at firstname.lastname@example.org.