Without a Trace
Frame sizes? Fine. DS3 configuration? Check. So, what's causing slow-as-molasses file copying across this admin's WAN?
- By Bill Boswell
I have a mix of Windows 2000 and Windows 2003 servers
and NT servers in a network with three offices, one in Chicago, one in
Seattle, and one in New York. I'm in the Chicago office, which is headquarters.
I'm having a problem with poor network file transfer performance across
our main WAN link to Seattle. This problem has been nagging us for quite
Here's my benchmark. I copy a large file (250 MB) using Robocopy from
a Windows 2003 server in Chicago to several other servers both locally
and across the WAN. All of the servers have 100Mbps NICs. The WAN link
is a DS3 frame relay connection.
When I copy the test file between Windows 2003 servers locally, Robocopy
reports a throughput of 380 MB per minute. I calculate that to be about
51 Mbps, which is pretty good for a 100 Mbps Ethernet.
When I copy the same file across the network to a Windows 2003 server
in Seattle, Robocopy reports 38 MB/min, which is only 5.1 Mbps. That seems
really bad for a DS3 connection. But it gets worse. When I copy the same
file to an NT server in Seattle, the throughput drops to 3.5 MB/min, which
is only 0.5 Mbps.
I wanted to benchmark the file copy to an NT server here locally in Chicago,
but I get different numbers depending on whether I copy the file to
the NT server or copy it from the NT server. When I copy the file
to the NT server, I get the equivalent of 15 Mbps. When I copy
it from the NT server, I get 39 Mbps.
I captured a Network Monitor trace of the file transfer between the Windows
2003 server in Chicago and an NT 4 server in Seattle. It doesn't help
me much. Here are a few sample frames that are typical of the entire file
# Time Source Destination
37 0.468750 10.10.10.10 10.50.50.50 SMB Write AndX Request,
FID: 0x080f, 4292 bytes at offset 5906824
38 0.531250 10.50.50.50 10.10.10.10 TCP netbios-ssn > 2958 [ACK]
Seq=79369844 Ack=1230722740 Win=8760 Len=0
39 0.531250 10.50.50.50 10.10.10.10 SMB Write AndX Response, FID:
0x080f, 4292 bytes
40 0.531250 10.10.10.10 10.50.50.50 SMB Write AndX Request, FID:
0x080f, 4292 bytes at offset 5911116
41 0.593750 10.50.50.50 10.10.10.10 TCP netbios-ssn > 2958 [ACK]
Seq=79369895 Ack=1230727100 Win=8760 Len=0
42 0.593750 10.50.50.50 10.10.10.10 SMB Write AndX Response, FID:
0x080f, 4292 bytes
I was confused about the frame size of 4292 bytes, because I thought
that an Ethernet frame could only be 1510 bytes maximum. I am thinking
that these might be jumbo frames, because I found a Microsoft article,
"Microsoft Windows 2000 TCP/IP Implementation Details" http://www.microsoft.com/technet/itsolutions/network/deploy/
depovg/tcpip2k.mspx that has a feature comparison grid saying that
NT and Windows 2000 (and Windows 2003, I'm assuming) all support jumbo
frames. But, my network engineer says that he isn't seeing any jumbo frames
arrive at the LAN interface of the router.
I think there's a problem with the DS3 configuration, but my network
engineer and AT&T says that it's configured just right. They also
say that there aren't any collisions or errors at the ports for the servers,
so it doesn't look like a mismatched full/half duplex problem.
Here's one other weird thing that happened: One of the test files I used
was a saved file from a Netmon capture. It was about 100 MB. When I used
that file to do the transfer tests in Robocopy, all of my statistics improved
considerably. In fact, I was able to get more than 100 Mbps throughput,
which should be impossible.
So, what is the cause of my poor WAN file copy performance and why is
the performance so much worse for NT than for Windows 2003? And why do
I get different throughputs for different test files?
Help from Bill
Got a Windows or Exchange question or need troubleshooting
help? Or maybe you want a better explanation than provided
in the manuals? Describe your dilemma in an e-mail
to Bill at mailto:[email protected];
the best questions get answered in this column.
When you send your questions, please include your
full first and last name, location, certifications (if
any) with your message. (If you prefer to remain anonymous,
specify this in your message but submit the requested
information for verification purposes.)
Tad and I ended up troubleshooting this problem for quite a while. Frankly,
I'll admit that I learned a few things along the way. We ended up isolating
the cause of the problem, but before we reveal it next week (yes, you'll
have to wait), I'd like you to try your hand at finding a solution. If
you can figure out Tad's problems, send your solutions to me at [email protected].
We'll publish some of your answers next week and crown one winner randomly
among you with an MCPmag.com baseball cap and a copy of my book,
Learning Exchange Server 2003.
Here's a hint: There's only one actual "problem"
in this scenario. Everything else turns out to be inherent in the system
design. Have fun!
Contributing Editor Bill Boswell, MCSE, is the principal of Bill Boswell Consulting, Inc. He's the author of Inside Windows Server 2003 and Learning Exchange Server 2003 both from Addison Wesley. Bill is also Redmond magazine's "Windows Insider" columnist and a speaker at MCP Magazine's TechMentor Conferences.