News

Free and Easy VM Performance Benchmarks

Need tools for comparing virtual machine performance across different platforms, but no money in your IT budget? Here are a few suggestions.

• By Chris Wolf
• 07/10/2007

Youssef, it's hard to go wrong with the spec.org benchmarks, as they are the gold standard and well embraced by the IT community. In fact, spec.org is currently working on a SPEC Virtualization Performance benchmark that will create an industry standard for measuring virtualization platform performance.

Today, the most widely recognized virtualization benchmark is VMware's VMmark. Both VMware and XenSource have published performance comparisons using spec.org standard benchmarks. You can view each vendor's performance comparison in the following documents:

• VMware: A Performance Comparison of Hypervisors (click here to download .PDF)
• XenSource: A Performance Comparison on Commercial Hypervisors (click here to download .PDF)

Performance numbers across virtualization platforms can vary, depending on a VM's installed guest operating system and applications. Since the spec.org benchmarks are designed for individual server roles such as file, web, mail, or database serving, they will offer the most complete picture of server performance.

If you are looking for a simple baseline comparison using free or inexpensive benchmarking tools, I can recommend the following:

• Netperf (network performance) (.zip download here)
• Iometer
• Passmark

I wrote about netperf in my Tech Line column (see "Bottleneck Battle," which includes a link to download the tool as well as instructions on how to use it. Iometer is an open-source benchmarking tool that has Windows, Linux and NetWare builds. Iometer allows you to benchmark a system's storage and network I/O. Passmark can be installed and used free for 30 days, and is available for purchase for $24. While Passmark is designed for benchmarking PC performance, it is still useful for collecting performance baselines for Windows server operating systems as well. Passmark will provide performance evaluations of CPU, memory, and disk I/O, and includes several other performance tests as well.

When evaluating virtualization platforms, there are two general test methodologies: virtual-to-native and head-to-head comparisons.

Comparing virtual machine performance against native performance is a popular test methodology. In order to perform this test, follow these general steps:

1. Use the test system's CMOS Setup program to restrict its available physical RAM to match the memory you plan to allocate to the test virtual machine. For example, if you plan to test the VM with 1GB of assigned RAM and the physical server has 4GB of physical RAM, you should limit the physical server's memory to 1GB so that the its physical characteristics will match that of the VM.
2. Install the test OS (Windows Server 2003 is the most popular) on the physical server and install any needed device drivers.
3. Use Netperf, Iometer or Passmark to collect performance data. Save the data to removable media or an alternate server.
4. Remove any memory restrictions previously configured in the CMOS Setup.
5. Install the virtualization application you wish to test.
6. Create a new VM and install the guest OS inside the VM. Once the OS installation completes, install the VM Tools or VM Additions, which will set up the best device drivers for the virtualization platform.
7. Repeat the tests from step 3 to collect performance data for the virtual machine.
8. Repeat steps 5-7 for each virtualization platform that you wish to evaluate.

If you just wish to test head-to-head virtualization platform performance, a physical server baseline is not necessary. Just follow these steps:

1. Install the virtualization application you wish to test.
2. Create a new VM and install the guest OS inside the VM. Once the OS installation completes, install the VM Tools or VM Additions, which will set up the best device drivers for the virtualization platform.
3. Use Netperf, Iometer or Passmark within the VM's guest OS to collect performance data. Save the data to removable media or an alternate server.
4. Repeat steps 1-3 for each virtualization platform that you wish to evaluate.

Again, the spec.org tests will provide the most complete and data center worthy results. However, if you're looking for a quick and dirty baseline performance comparison, the tools described in this article should provide what you need. Finally, if you are not happy with the numbers you see, I would recommend evaluating a product such as InovaWave's DXtreme, which can significantly improve I/O performance.

If any of you have time to benchmark different virtualization platforms, please share your results. If you email them to me, I will post them on my web site and add a hyperlink to the results as a comment to this article.

Happy benchmarking!