Boswell's Q&A

Slow Win2K Performance After the Crash

Performance dies on this unnamed admin's Dell server, so Bill helps him trace the problem to driver chaos.

• By Bill Boswell
• 06/22/2004

Bill: A Windows 2000 domain controller crashed with a blue screen of death earlier in the week and since then, it's been really slow. No, I didn't write down the error code.

Yesterday, the server started beeping constantly and there was an error message on the screen about a failed drive in the RAID array. This is a Dell PowerEdge with mirrored drives for the operating system and data. The RAID controller is a standard Dell PERC controller.

I called Dell tech support and they determined that the drive was functioning normally. They walked me through regenerating the mirrored set. Right now, the boot-time PERC controller management console shows a correct drive configuration with no errors.

But when I restarted after regenerating the mirror, the system hung for nearly a half hour. I restarted in Safe Mode and as the list of drivers appeared on the screen, the system hung at the Mup.sys driver. It's done this consistently ever since.

What's wrong? Do I have a bad drive? What is Mup.sys and why is it hanging?
—Anon

Get Help from Bill Got a Windows or Exchange question or need troubleshooting help? Or maybe you want a better explanation than provided in the manuals? Describe your dilemma in an e-mail to Bill at mailto:[email protected]; the best questions get answered in this column. When you send your questions, please include your full first and last name, location, certifications (if any) with your message. (If you prefer to remain anonymous, specify this in your message but submit the requested information for verification purposes.)

Readers: I called Anon and we spent quite a while troubleshooting this problem. Here's what we did:

First off, I didn't suspect any problem with Mup.sys. This is the driver for the Multiple UNC Provider, which determines which network client protocol to use when the target server is specified by a UNC path such as \\Server_Name. During safe mode boot, the system displays each driver as it is loaded, so the fact that Mup.sys was the last driver on the screen prior to the system hang indicated that the problem was with the next driver.

Ordinarily, you can see the drivers loaded during a Safe Mode boot by looking at a file called Ntbtlog.txt. Because the server was hanging during safe mode, we decided to boot to the Recovery Console using the Windows 2000 Setup CD. This would also help us determine if there were a problem with the file system or the mass storage device driver.

As part of the Recovery Console boot process, we loaded a current copy of the Dell PERC controller driver that we downloaded from Dell's Web site. We were able to get logged on in the Recovery Console once Anon figured out the local Administrator password. The Recovery Console uses the Administrator account from the local SAM, which on a domain controller is the same account used for Directory Services Restore Mode (DSRM).

In the Recovery Console, we were able to read from and write to the drive with no slowdown or glitches, so that part of the system seemed to be working fine. There was no Ntbtlog.txt file though, indicating that the system was hanging before the file could be written to disk.

At this point, because the PERC controller console showed no problems and we were able to mount the drive using the latest PERC driver, I wanted to replace the driver currently on the operating system drive with the latest driver that we had downloaded. Anon wanted to do some more research before making changes to the system.

So, we started scouring the Internet looking for other possible causes. We found quite a few instances of the "hung at Mup.sys" symptom, but with a variety of fixes. Several administrators solved the problem by replacing memory. Several others solved it by replacing drive controllers or by simply moving the controllers to a different slot. One administrator even replaced both processors.

Then we found a posting by Sean Branham at the Annoyances.org web site. See the full text of the thread at http://www.annoyances.org/exec/forum/
winxp/t1047532372. Sean correctly determined that the cause of all these disparate "hung at Mup.sys" failures were actually caused by problem with the Extended System Configuration Data (ESCD) stored in the system BIOS.

The ESCD maintains a static list of Plug-and-Play resource allocations. This avoids recalculating all the allocations at each restart. If the ESCD gets corrupted, then the operating system cannot assign resources correctly. Windows makes this resource decision just after it loads the Mup.sys driver because that's when it loads the Advanced Configuration and Power Interface (ACPI) drivers.

You can download the (mercifully short) ESCD specification from http://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/escd.rtf.

Once we knew that something in BIOS might be causing the problem, solving it was a snap. We downloaded the most current firmware revision from Dell's web site and flashed the BIOS and that was that. (Some motherboards come with an ESCD rebuild option in CMOS, so it would not be necessary to flash the BIOS.) The system booted without a hitch and performance was right back to where it had been before the problems started. If it hadn't been for Sean's insight, we would have spent time and money replacing the PERC controller, which unfortunately might well have solved the problem because replacing the board would have refreshed the ESCD.

It's difficult to determine whether the system crash earlier in the week caused the ESCD problem or vice-versa, or if some other problem caused both. At this point, Anon is going to keep an eye on the system and hope for the best.

I'd like to thank Sean both for solving this tricky problem and for taking the time to post a detailed account. This was the first time I'd visited the Annoyances.org web site, and it looks like a great resource.

Hope this helps.