Your Testing Environment

Developers have a challenging task when it comes to setting up software to match client requirements. Here's how to stay sane with the shifting landscape of beta and release software.

• By Mike Gunderloy
• 12/01/2001

Here's the basic problem: If you're developing software for a wide customer base (which might be for half a dozen consulting clients or a commercial product that you sell to all comers), sooner or later you're going to get a call—"It doesn't work!" At that point, you need to play detective. In many cases, you'll end up determining that the software seems to have been installed correctly, but that the user is running it on different hardware or software than you use for development. Perhaps they're your first user with Windows ME, or they've downloaded a new beta version of Internet Explorer (which might well break your XML code, if you depend on the MSXML parser), or they have four IDE hard drives in their machine, or they're using the Hebrew version of Windows, or…the possibilities are endless.

It's Called Hardware for Good Reason
Hardware problems can be the most intractable ones to deal with, simply because of the difficulty of duplicating the hardware environment. True story: I once worked on a subcontract for a very large software corporation developing some software that was being tested internationally. The Japanese subsidiary reported that the program mysteriously bombed on most of their computers. After a lot of back-and-forth e-mail, we figured out that it was just trying to open a temporary file when it locked up. After a lot more e-mail, we finally managed to find a computer in our area that was a duplicate of the ones they were using. That was when I learned that NEC had shipped a bunch of computers where the hard drive was drive A. And our code was trying to save a temporary file to the root of the C drive! OK, boneheaded move (the correct answer, of course, is to ask Windows where the temporary directory lives), but half a dozen developers couldn't find the right answer without the hardware, no matter how long we stared at the source code.

One brutal fact of this business is that (unless you're working for a large corporation with a very large test lab) you will never be able to buy hardware to duplicate all of the cockamamie configurations that your customers are using. Despite this, your first line of defense in coping with software testing should be to build up your own network of computers. I recommend at least three test machines in addition to your development machine, all networked together. That should give you enough resiliency to deal with multiple problems, as well as provide backup for the inevitable hardware problems. Fortunately, you don't need to spend a whole bundle of money on this. First, even though you need the latest-and-greatest whiz-bang hardware for your own desk, your customers are likely using older computers that you can match without excessive spending. Second, you don't want three identical brand-new computers. I recommend buying three different used computers, just to triple your chance of finding odd hardware interactions. If you poke around on eBay, you'll find Pentium-class computers that are probably as good as those used by the majority of your customers starting at a few hundred dollars. Throw in a keyboard-mouse-video (KVM) switch, some cables, and a hub, and you can set up the nucleus of your test lab for a thousand dollars or so.

I strongly suggest that you not do your testing on your main development box. Why not? Because sooner or later you're going to have to install some beta software or obscure driver on the test machine. The best way to avoid having junk pile up to the point where you waste a couple of days reinstalling Windows and all of your development applications is to never install beta software on your development box. In fact, don't install anything that's not necessary to your productivity on that box. Keep a second computer by your desk if you can to hold betas and games and the latest unstable peer-to-peer Internet application—and be prepared to format that hard drive once every six months or so, when it starts crashing on a regular basis.

Of course, if you're like me, you'll never throw a computer away, and eventually end up with a dozen test boxes. This is a good thing, so long as you can convince your significant other that you really need all those computers.

Software Can Get Costly
After you've got hardware to work with, you need to start worrying about software. Here's a short list of the questions that I usually ask to try to track down pertinent software information:

• Did the application ever work? Did it just stop working? If so, what was the most recent thing you installed?
• Which operating system are you using? Which service packs? Which hotfixes?
• What version of Internet Explorer is installed? Any other web browser?
• What version of Office is installed? Visual Basic? Any service packs?

Take my word for it: You do not want to buy every bit of software you'll need piecemeal. Even apart from the nuisance value, you'll spend way too much money, particularly if you need versions in other languages (and you will need them if you sell any copies to other countries; it's not unusual for a bug to only surface on a particular version, say French Windows 98). I highly recommend that you sign up for a subscription to the Microsoft Developer Network, if you're not already a member. Subscriptions come in a variety of levels and prices, depending on which software you need; you'll find the details at http://msdn.microsoft.com/subscriptions/prodinfo/levels.asp. Personally, I've never regretted spending the money for MSDN Universal every year.

By the way, it's not too tough to debug on French Windows (or German, or even Chinese) even if you don't speak French, because all of the menu items remain in the same order.

Testing Environment Shortcuts
But getting all the software you need is only half the battle. You can waste a lot of time installing and reinstalling operating systems. Two types of tools can help you minimize this waste of time: disk imaging and virtual machines.

With disk imaging products, you can take a pristine copy of an operating system and save it to a CD (or tape or a file). Later, when you need that operating system again, it's much easier to restore the CD than it is to go through the whole reinstall. If you go this route, after a while you'll build up a set of CDs customized to your own hardware and the way you like to have your Windows set up, and be able to quickly bring up a test box with the desired configuration. Some products in this category include Norton Ghost (http://www.symantec.com/sabu/ghost/ghost_personal/) and PowerQuest Drive Image 5 (http://www.powerquest.com/driveimage/).

Useful though disk imaging is for building up a catalog of configurations, it's not the best answer for managing multiple operating systems. The idea of a virtual machine is to run one operating system inside of a window on another operating system. For years people said that the Intel x86 architecture didn't support virtual machines well enough for this to be a workable idea, and then along came VMware (http://www.vmware.com). VMware itself installs as an application on your Windows or Linux PC. Then you open the VMWare window, push the virtual power switch—and a second PC boots up! Plug in a CD-ROM, and it will install a new operating system inside the window. VMware offers many other features, including solid networking from inside the virtual machine, save to disk, and an "undo" facility that can remove changes made by errant software. I can't recommend it highly enough; right now on one of my boxes I have a Windows XP virtual machine and a FreeBSD virtual machine both running in windows on top of a Windows 2000 Server installation. The only drawback is that you need a lot of RAM for this to work well; I'd recommend at least 512MB if you're going to make heavy use of this strategy for testing and debugging.

Finally, if you do end up spending time duplicating a customer's environment to track down some obscure problem: take notes. You may want to look at a commercial bug-tracking system; there are dozens to choose from. Or you may want to simply keep a Word document or Access database of your own to store this information. But whatever you do, don't depend on memory to tell you what the fix was for running your software on Bulgarian Windows!

I hope you find some of that advice helpful—I know I wish someone had told me about disk imaging long before I found out about it myself. Did I miss a trick you find essential for development and debugging? Write me at [email protected] to let me know!