News

Microsoft Resubmits Withdrawn Benchmarks

• By Scott Bekker
• 07/27/2000

At issue were the benchmark results Microsoft chairman and chief software architect Bill Gates unveiled at the February launch of Windows 2000 that shattered all previous performance records on the Transaction Processing Performance Council's OLTP performance benchmark, the TPC-C.

Late last month, the TPC, an industry group that oversees the TPC-C and a handful of other performance benchmarks to ensure that they are fair and meaningful, disqualified the Microsoft numbers.

Saying the ruling was over a technicality, Microsoft (www.microsoft.com) vowed to resubmit audited results shortly.

Today, Microsoft announced that it had rerun the tests and once again had audited performance benchmarks posted on the TPC site (www.tpc.org). The announcement came at a meeting of financial analysts where Microsoft was discussing SQL Server. Sales of the relational database management system were a highlight of Microsoft's recent financial results, which were otherwise relatively flat.

The company achieved the record in February with Windows 2000 Advanced Server and pre-release code of SQL Server 2000 on a cluster of Compaq Computer Corp. (www.compaq.com) 8-processor ProLiant servers. Since then, IBM Corp. (www.ibm.com) reclaimed the top spot in raw performance with Microsoft's help using a cluster of 32 Netfinity servers running Windows 2000 Advanced Server on IBM's DB2 database software.

The resubmitted numbers from Microsoft and Compaq improve upon the February results, but remain far behind what IBM achieved earlier this month. (See chart below).

The SQL Server 2000 on Windows 2000 Advanced Server results were hugely important to Microsoft, which splashed the results up at every major presentation since the Windows 2000 launch. For the first time, a Microsoft database on a Microsoft operating system was at the top of the raw performance chart, a rank Microsoft products never came close to competing for before. The placement represented a huge jump in status for Microsoft, which had barely broken into the top 10 on raw performance with a Compaq result shortly before vaulting to the top spot.

To get there, Microsoft and Compaq built a complex cluster using a new feature of SQL Server 2000 called Distributed Partition Views. In that architecture, each server in the cluster controls its own piece of the larger database. Microsoft posted two record results for first and second place on TPC-C performance. One configuration used 12 server nodes, the other had 8 nodes. Some observers question whether the architecture is feasible for most IT departments to set up and support.

Microsoft got tripped up when a challenge was mounted within the TPC over a rule involving the primary key of a database.

The primary key is the field that uniquely identifies a record and often links records in different tables of a relational databases. In a personnel database, for example, the primary key might use an employee's social security number. While some database experts contend that a primary key should be something that never changes, the challenge came over the inability of SQL Server to update primary keys in the distributed configuration.

There has been some fallout since the TPC decision. IBM found the primary key interpretation surprising and needed to add the primary key update capability to DB2 to get its 32-node configuration submitted to the TPC. In the month since the Microsoft results came down, another 10 TPC-C benchmark results dating back to 1996 have been withdrawn. Those tests had been performed using Oracle and Sybase databases.

Some trade press reports put Oracle behind the TPC challenge. Microsoft officials declined to comment. But Microsoft did take pains to skewer its chief rival for database market share on the Windows server platform.

In a news release on the resubmission, Microsoft pointed out that no TPC-C result using an Oracle database ranks higher than seventh place on the TPC performance list, and that performance on Oracle systems has improved only 2 percent while the SQL Server system performance jumped 15 percent.

For the resubmission round of tests, Microsoft and Compaq chose to rerun the original systems with their 550 MHz Pentium III Xeon processors for a slight gain in performance over the original results. The companies then ran the benchmark on the systems using the 700 MHz Xeons Intel Corp. (www.intel.com) released recently. The four rerun benchmarks now hold positions two through five on the TPC raw performance list. - Scott Bekker

*Top non-Windows 2000 result on TPC-C raw performance

Back to article.