Boswell's Q&A

Without a Trace

Frame sizes? Fine. DS3 configuration? Check. So, what's causing slow-as-molasses file copying across this admin's WAN?

• By Bill Boswell
• 11/30/2004

Bill: I have a mix of Windows 2000 and Windows 2003 servers and NT servers in a network with three offices, one in Chicago, one in Seattle, and one in New York. I'm in the Chicago office, which is headquarters. I'm having a problem with poor network file transfer performance across our main WAN link to Seattle. This problem has been nagging us for quite a while.

Here's my benchmark. I copy a large file (250 MB) using Robocopy from a Windows 2003 server in Chicago to several other servers both locally and across the WAN. All of the servers have 100Mbps NICs. The WAN link is a DS3 frame relay connection.

When I copy the test file between Windows 2003 servers locally, Robocopy reports a throughput of 380 MB per minute. I calculate that to be about 51 Mbps, which is pretty good for a 100 Mbps Ethernet.

When I copy the same file across the network to a Windows 2003 server in Seattle, Robocopy reports 38 MB/min, which is only 5.1 Mbps. That seems really bad for a DS3 connection. But it gets worse. When I copy the same file to an NT server in Seattle, the throughput drops to 3.5 MB/min, which is only 0.5 Mbps.

I wanted to benchmark the file copy to an NT server here locally in Chicago, but I get different numbers depending on whether I copy the file to the NT server or copy it from the NT server. When I copy the file to the NT server, I get the equivalent of 15 Mbps. When I copy it from the NT server, I get 39 Mbps.

I captured a Network Monitor trace of the file transfer between the Windows 2003 server in Chicago and an NT 4 server in Seattle. It doesn't help me much. Here are a few sample frames that are typical of the entire file transfer:

#  Time     Source      Destination Prot Summary
37 0.468750 10.10.10.10 10.50.50.50 SMB  Write AndX Request, FID: 0x080f, 4292 bytes at offset 5906824
38 0.531250 10.50.50.50 10.10.10.10 TCP  netbios-ssn > 2958 [ACK] Seq=79369844 Ack=1230722740 Win=8760 Len=0
39 0.531250 10.50.50.50 10.10.10.10 SMB  Write AndX Response, FID: 0x080f, 4292 bytes
40 0.531250 10.10.10.10 10.50.50.50 SMB  Write AndX Request, FID: 0x080f, 4292 bytes at offset 5911116
41 0.593750 10.50.50.50 10.10.10.10 TCP  netbios-ssn > 2958 [ACK] Seq=79369895 Ack=1230727100 Win=8760 Len=0
42 0.593750 10.50.50.50 10.10.10.10 SMB  Write AndX Response, FID: 0x080f, 4292 bytes

I was confused about the frame size of 4292 bytes, because I thought that an Ethernet frame could only be 1510 bytes maximum. I am thinking that these might be jumbo frames, because I found a Microsoft article, "Microsoft Windows 2000 TCP/IP Implementation Details" http://www.microsoft.com/technet/itsolutions/network/deploy/
depovg/tcpip2k.mspx that has a feature comparison grid saying that NT and Windows 2000 (and Windows 2003, I'm assuming) all support jumbo frames. But, my network engineer says that he isn't seeing any jumbo frames arrive at the LAN interface of the router.

I think there's a problem with the DS3 configuration, but my network engineer and AT&T says that it's configured just right. They also say that there aren't any collisions or errors at the ports for the servers, so it doesn't look like a mismatched full/half duplex problem.

Here's one other weird thing that happened: One of the test files I used was a saved file from a Netmon capture. It was about 100 MB. When I used that file to do the transfer tests in Robocopy, all of my statistics improved considerably. In fact, I was able to get more than 100 Mbps throughput, which should be impossible.

So, what is the cause of my poor WAN file copy performance and why is the performance so much worse for NT than for Windows 2003? And why do I get different throughputs for different test files?
—Tad Nelson

Get Help from Bill Got a Windows or Exchange question or need troubleshooting help? Or maybe you want a better explanation than provided in the manuals? Describe your dilemma in an e-mail to Bill at mailto:[email protected]; the best questions get answered in this column. When you send your questions, please include your full first and last name, location, certifications (if any) with your message. (If you prefer to remain anonymous, specify this in your message but submit the requested information for verification purposes.)

Tad and I ended up troubleshooting this problem for quite a while. Frankly, I'll admit that I learned a few things along the way. We ended up isolating the cause of the problem, but before we reveal it next week (yes, you'll have to wait), I'd like you to try your hand at finding a solution. If you can figure out Tad's problems, send your solutions to me at [email protected]. We'll publish some of your answers next week and crown one winner randomly among you with an MCPmag.com baseball cap and a copy of my book, Learning Exchange Server 2003.

Here's a hint: There's only one actual "problem" in this scenario. Everything else turns out to be inherent in the system design. Have fun!