Windows Advisor

Windows Update Troubles, Or What to Do When the Mother of All Patches Fails

Windows Update didn't update my Windows servers like it's supposed to, so here's the process I went through to get my servers back up to speed.

• By Zubair Alexander
• 12/23/2011

If you've encountered problems installing service packs or software updates on Windows Server 2008 R2, Windows 7 or Windows Vista, you are not alone. It's a common problem and lots of people -- including me -- have faced them. In this article, I share some of my recent experiences and how I solved them, and in turn I hope it helps you get a better understanding of the issues to make troubleshooting less painful.

I should point out that I my recent issues have involved the installation of Service Pack 1 on numerous servers (mostly new ones) as well as existing Windows 7 clients. The focus of this article is on Windows Server 2008 R2, but you can also apply the techniques documented in this article on Windows 7 and Windows Vista.

And also note that applying patches and updates is more art than science. Depending on your operating system, with the applications that you are running (especially antivirus software) and the specific updates you are currently running, you may experience varying degree of success installing Windows Updates.

So, to continue: It took me three full days to find a solution that worked for me. In that time, I had searched the Web and tried various solutions, with some that worked and some that failed. Naturally, my searches included TechNet, various forums, and I also looked closely at all the errors in the Event Viewer, read through a bunch of blog posts, I Googled, I Binged, etc.

Unlike the old Windows NT days when the patches were considered a risky business, for the past decade or so Microsoft has done a great job of making the updates and security patches fairly reliable. It's a daunting task to deal with a gazillion updates on various systems and gain the confidence of consumers, but Microsoft has gained enough of my confidence that I have been configuring all my computers -- including servers -- to download and install the Windows Update automatically. Even though I have occasionally encountered a few crashes, overall I have been fairly satisfied with the automatic Windows Update service.

But lately, things have not been so rosy. Windows Update has been causing more problems more frequently. So, just recently I decided to make it a practice to manually update my computers because of the fear of more frequent system crashes and other unexpected and unreliable results.

The Mother of All Patches? Maybe!
In my searches, I found one tool that Microsoft hasn't drawn much attention to: a patch called the Windows System Update Readiness Tool. With that tool, Microsoft confirmed my fears about Windows Update. In essence, it's a patch to fix the patches. In other words, if the updates start to cause nasty problems or if you have other corruption issues, this patch is supposed to fix the corruption. I guess you can call it the mother of all bug fixes -- well, sort of. These days vendors rarely use the term "bug fix," as if calling it that would be admitting that there was a problem with the software in the first place. Instead, vendors usually refer to bug fixes as "patches," "updates," "repairs," "remedies," "fixes," "solutions." Now, you can add "tool" to the mix. You tell me which one sounds better: Windows System Update Readiness Tool or Windows Update Bug Fix? Exactly!

From my research, though, I discovered that Microsoft is aware that even the Windows System Update Readiness Tool may not fix the Windows Update problems that some of us are encountering in Windows Server 2008 R2. So, Microsoft posted this article on TechNet for advanced diagnosing and fixing servicing corruption. The article is listed under the Troubleshooting section as "Known Issues with Windows Server 2008 R2."

So, here's what we know at this point: Microsoft is aware of this issue and has released a bug fix for the bug fixes. It's also admitting that the bug fix for the bug fixes may not work, so we may need to rely on some advance diagnostics to fix the problems with the corruption in Windows Servicing Store. In my experience, Microsoft is 100 percent right -- and that's exactly what I ended up doing to solve the issues on at least five different servers on two different networks. I had to use the advanced diagnostic techniques to install SP1 because the mother of all patches -- Windows System Update Readiness Tool -- didn't work for me in these particular cases.

Install Any Update, As Long As It's Not SP1
For the most part I have experienced the problem with installing SP1 on new installations. The problem is that the service pack hangs after a minute or so and the installation fails. Rebooting doesn't help.

After spending a lot of time I finally narrowed the problem down to one particular update, KB 2620704. I was installing Windows Server 2008 R2 on several servers at the same time. I installed all the updates on my new servers -- 92 updates, to be exact -- and then installed KB 2620704 that was causing problems. On some servers, KB 2620704 failed; on others I was able to install it successfully. However, even after I was able to install all the updates, including KB 2620704, I still was unable to install SP1.

In addition, I couldn't install SharePoint Server 2010 -- on one of the servers, when I tried to install the software prerequisites for SharePoint, it failed. In the last five years I have installed SharePoint probably over 100 times and for the most part I have never run into any major issues. This issue with installation has only surfaced recently. (I cover the SharePoint problem in more detail later.)

After the installation of KB 2620704 failed, Windows Update offered me a new update called System Update Readiness Tool for Windows Server 2008 R2 x64 Edition (KB 947821) [August 2011], as shown in Fig. 1.

Figure 1. When installation of KB 2620704 failed, I was sent to the System Update Readiness Tool. (Click image to view larger version.)

According to Microsoft:

"This tool is being offered because an inconsistency was found in the Windows servicing store which may prevent the successful installation of future updates, service packs, and software. This tool checks your computer for such inconsistencies and tries to resolve issues if found."

This update is only offered if there is a problem that the tool can resolve. In my situation the tool didn't fix the problem because I was not able to install SP1. In case you are wondering about the Windows Servicing Store that I mentioned earlier, it's a component that is required to successfully install the service packs.

The tool didn't do the job for me, so it was time to do more research. After some time I found this Microsoft TechNet article, Advanced guidelines for diagnosing and fixing servicing corruption. Microsoft recommends using these guidelines if the tool fails to fix your problem.

The System Update Readiness Tool creates log files that capture any issues that the tool has found or fixed. The log files are located at:

• %SYSTEMROOT%\Logs\CBS\CheckSUR.log
• %SYSTEMROOT%\Logs\CBS\CheckSUR.persist.log

I looked at the following log to see what's going on:

%windir%\logs\cbs\checksur.log

The result is in Fig. 2.

Figure 2. What appeared inside the checksur.log file. (Click image to view larger version.)

I noticed at the end of the log it pointed to KB 2620704, which I knew was a problem right from the start. On the server where I was able to install KB 2620704 everything was fine, but on the server where I wasn't able to install SP1 I knew I had to install KB 2620704 before I would be able to proceed. I was left with only two updates, KB 2620704 and the SP1 update KB 976932, so I unchecked SP1 update KB 976932 and tried to install KB 2620704. It failed with the error Code 800F0818 (Fig. 3).

Figure 3. Another attempt at installing KB 2620704 ended up with the system coming up with an error code 800F0818. (Click image to view larger version.)

Advanced Techniques for Fixing Servicing Corruption
At this point, some people get lucky and are able to install SP1 once they have installed KB 2620704; I wasn't so fortunate. So, here's what I did next: Following the instructions in the TechNet article, Advanced guidelines for diagnosing and fixing servicing corruption, I looked at the two files listed at the end of the checksur.log:

servicing\packages\Package_for_KB2620704_SP1~31bf3856ad364e35~amd64~~6.1.1.0.mum
servicing\packages\Package_for_KB2620704_SP1~31bf3856ad364e35~amd64~~6.1.1.0.cat

Then I created a folder called Temp at the root of drive C. Next I started cmd.exe as an administrator and backed up the two files listed above to C:\Temp as a precaution:

copy %windir%\servicing\packages\Package_for_KB2620704_SP1~31bf3856ad364e35~amd64~~6.1.1.0.mum c:\temp
copy %windir%\servicing\packages\Package_for_KB2620704_SP1~31bf3856ad364e35~amd64~~6.1.1.0.cat c:\temp

Note: There is a space after the word copy and before c:\temp.

I then took ownership of these files so I can copy these files from another server:

takeown /f %windir%\servicing\packages\Package_for_KB2620704_SP1~31bf3856ad364e35~amd64~~6.1.1.0.mum
takeown /f %windir%\servicing\packages\Package_for_KB2620704_SP1~31bf3856ad364e35~amd64~~6.1.1.0.cat

Next, I used icacls to grant administrators permissions to overwrite the files:

icacls %windir%\servicing\packages\Package_for_KB2620704_SP1~31bf3856ad364e35~amd64~~6.1.1.0.mum /grant administrators:F
icacls %windir%\servicing\packages\Package_for_KB2620704_SP1~31bf3856ad364e35~amd64~~6.1.1.0.cat /grant administrators:F

Finally, I logged out and then logged back in so I could copy the two files from another server where I was able to successfully install KB 2620704 to the server. Even though I was logged in with a domain account that was a member of the local administrators group, the permission to copy the files was denied.

I went to Windows\servicing\packages folder in Windows Explorer and gave my domain account full-control permissions to the packages folder. I removed this permission after I was able to copy the two files. I ran the update for KB 2620704 and it was finally successful.

I then tried to install SP1 (KB 976932) again. By that time I knew all these KB article numbers better than my address and phone number. Fortunately, this time it worked and I was able to install SP1 on my Windows Server 2008 R2. It only took about 40 hours in three days to install SP1. Piece of cake!

SP1 and Support for AVX
If you think you can skip installing SP1 to avoid all the hassles, think again. Here's a situation that I ran into that I would like to share with you: I recently purchased a new server, installed Windows Server 2008 R2 and Hyper-V, installed all the updates, copied my virtual machine from another server and tried to start the virtual machine. It failed. I realized I needed to enable hardware-assisted virtualization in the BIOS, which I did, but the virtual machine still didn't start. According to the error, the virtual machine failed to initialize.

I found this KB article that explained that Windows Server 2008 R2 doesn't support the Advanced Vector Extensions (AVX) feature that is available on certain Intel CPUs. I knew my server supported AVX. Upgrading to SP1 added the support for AVX on my server and I was able to successfully start the virtual machine in Hyper-V.

SharePoint Server 2010 Installation Issues
Now, here's the process I went through to install SharePoint Server 2010 on a new server. Once the service pack was installed I was expecting to successfully install SharePoint, but I was wrong. This time the software prerequisites tool was able to install a couple of prerequisites, including the Web Server (IIS) Role, but was unable to install the KB 976462 hot fix (see Fig. 4).

Figure 4. When I tried to update my SharePoint server with KB 976462, I got this nasty message. (Click image to view larger version.)

Okay, no problem. I downloaded KB 976462 for my x64 system (Windows6.1-KB976462-v2-x64.msu) from this link and tried to run it, but got the following error: "The update is not applicable to your computer." Here's how I worked around that hurdle: I followed the instructions in yet another KB article, KB 934307:

1. I created a folder C:\KB976462 and downloaded the file Windows6.1-KB976462-v2-x64.msu into that folder.
2. I created another folder C:\TEMP.
3. I opened command prompt as an administrator and ran the following command to expand the MSU file to the temp folder which resulted in four files in the TEMP folder:
   
   expand -f:* "C:\KB976462\Windows6.1-KB976462-v2-x64.msu" C:\TEMP
   
   
4. Then I ran the following command:
   
   pkgmgr.exe /n:c:\temp\Windows6.1-KB976462-v2-x64.xml
   
   You will not see any message after you run this command successfully. Just wait a minute or so until the command prompt is returned and that's how you will know if the command successfully executed. The only time you will see a message is if something goes wrong.
5. Next, I double-clicked the Windows6.1-KB976462-v2-x64.msu file in the C:\KB976462 folder where I originally downloaded it. I got a pop-up message stating, "The update is not applicable to your computer." This is the same message I received when I had double-clicked the file the first time. By the way, some people have reported that they get the message that the patch is already installed.
   
   Note: Regardless of what message you see, you should simply ignore it and go to the next step and you will be a happy camper.
   
   
6. I ignored the notice and ran the SharePoint software prerequisites tool again. This time everything worked and I was able to install the software prerequisites (see Fig. 5).
   
   

   Figure 5. Awesome! The installation worked this time! (Click image to view larger version.)

   
   
   
7. I deleted both the C:\KB976462 and the C:\TEMP folder.
8. I rebooted the server. Even though I had not received any notices or warnings that I need to restart the server, when I tried to install SharePoint the wizard indicated that I must reboot first before proceeding to install SharePoint.
9. After the reboot I continued on with SharePoint Server 2010 installation without a hitch.

If you follow my instructions exactly the way I have documented them, you will have better luck. I have tried these steps several times and they have worked each time. In fact, the day after I blogged about this issue, I ran into the same issue while working on a SharePoint project for another client. I was able to follow these instructions and get their server installed pretty quickly.