Robocopy issue with CIFS - far too many network packets
I use Robocopy (XP010) to facilitate data migrations from W2k/2003 servers to NetApp appliances. Up til now, I've run the Robocopy command from the source server using the following command:
robocopy <src> <dest> /copyall /r:0 /w:0 /mir /b
where <dest> would be an attached share.I would use this command numerous times from the initial data sync until the final sync (sometimes weeks away). The process worked well, though because it was single threaded and CIFS is notoriously chatty and half-duplex, the process never pushed the network. This also ment that the migration required more time than necessary.I now have a Windows 7 workstation, and I thought I'd try out the multi-threaded capability. Although this ment copying from an attached share to another attached share (and twice the network traffic), the first copy went very, VERY fast! Subsequent invocations of the command took far too long (minutes vs seconds using the XP010 version of Robocopy). After quite a bit of trial and error I realized that both the "/b" and "/mir" options seem to have a significant (negative) impact on the time required as well as extra network traffic when using the Windows 7 (XP027) version of Robocopy but not with the XP010 version (can't speak to other versions).I ran tests using two version of Robocopy and different combinations of options (/b, /e, /mir, etc) between the following source and destination:I) Copy from Windows 2003 SP2 (SMB) to newer NetApp (SMB2 protocol) via Robocopy XP010 (run from my Windows 7 PC)II) Copy from newer NetApp (SMB2 protocol) to Windows 2003 SP2 (SMB) via Robocopy XP010 (run from my Windows 7 PC)III) Copy from Windows 2003 SP2 (SMB) to newer NetApp (SMB2 protocol) via Robocopy XP027 (run from my Windows 7 PC)IV) Copy from newer NetApp (SMB2 protocol) to Windows 2003 SP2 (SMB) via Robocopy XP027 (run from my Windows 7 PC)V) Copy from Windows 2003 SP2 (SMB) to older NetApp (SMB protocol) via Robocopy XP027 (run from my Windows 7 PC)In each instance I sync'd the data, then ran robocopy one more time while capturing the network traffic via WireShark. In total, there were five directories and 20 files - each 1 MB in size. Here are the results:I) Six variations of options using Robocopy XP010 (on my Windows 7 PC) from Windows 2003 server to newer NetApp (SMB2):1. "/e /b" - 232 packets (95 from src, 137 to dest)2. "/e" - 282 packets (95 from src, 187 to dest)3. "/e /z" - 282 packets (95 from src, 187 to dest)4. "/mir /b" - 232 packets (95 from src, 137 to dest)5. "/mir" - 282 packets (95 from src, 187 to dest)6. "/mir /z" - 282 packets (95 from src, 187 to dest)II. Six variations of options using Robocopy XP010 (on my Windows 7 PC) from newer NetApp (SMB2) to Windows 2003:1. "/e /b" - 218 packets (79 from src, 139 to dest)2. "/e" - 248 packets (109 from src, 139 to dest)3. "/e /z" - 248 packets (95 from src, 139 to dest)4. "/mir /b" - 218 packets (95 from src, 139 to dest)5. "/mir" - 248 packets (95 from src, 139 to dest)6. "/mir /z" - 248 packets (95 from src, 139 to dest)III) Six variations of options using Robocopy XP027 (on my Windows 7 PC) from Windows 2003 server to newer NetApp (SMB2):1. "/e /b" - 4,980 packets (271 from src, 4,709 to dest)2. "/e" - 452 packets (207 from src, 245 to dest)3. "/e /z" - 452 packets (207 from src, 245 to dest)4. "/mir /b" - 4,980 packets (271 from src, 4,709 to dest)5. "/mir" - 4,980 packets (271 from src, 4,709 to dest)6. "/mir /z" - 4,980 packets (271 from src, 4,709 to dest)IV. Six variations of options using Robocopy XP027 (on my Windows 7 PC) from newer NetApp (SMB2) to Windows 2003:1. "/e /b" - 3,242 packets (149 from src, 3,093 to dest)2. "/e" - 524 packets (105 from src, 419 to dest)3. "/e /z" - 524 packets (105 from src, 419 to dest)4. "/mir /b" - 3,242 packets (149 from src, 3,093 to dest)5. "/mir" - 3,242 packets (149 from src, 3,093 to dest)6. "/mir /z" - 3,242 packets (149 from src, 3,093 to dest)V) Six variations of options using Robocopy XP027 (on my Windows 7 PC) from Windows 2003 server to newer NetApp (SMB):1. "/e /b" - 6,774 packets (271 from src, 6,503 to dest)2. "/e" - 646 packets (207 from src, 439 to dest)3. "/e /z" - 646 packets (207 from src, 439 to dest)4. "/mir /b" - 6,774 packets (271 from src, 6,503 to dest)5. "/mir" - 6,774 packets (271 from src, 6,503 to dest)6. "/mir /z" - 6,774 packets (271 from src, 6,503 to dest)So, if either the "/b" or "/mir" options are utilized with the XP027 version of Robocopy, the number of packets between my PC and the destination share increased tenfold. More interesting is a look at the packets... I see repeated attempts to read what appears to be random sections of the files on the destination share, though no attempts to read any of the data on the source share (remember, the data has already been copied and is identical in size and dates). Here's the summary information from packet header pertaining to just one file:
Src
Dest
Len
Protocol
Info
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:512 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:512 Off:1048064 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
650
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:1047602 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:1099 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:1040637 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:1039576 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:1042432 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:1047765 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:1047746 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:1041987 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:256 Off:7172 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
394
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:512 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
650
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:512 Off:65024 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
650
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:2 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
140
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:512 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
650
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:4096 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
1514
TCP
[TCP segment of a reassembled PDU]
NetApp
MyPC
1514
TCP
[TCP segment of a reassembled PDU]
MyPC
NetApp
54
TCP
51321 > microsoft-ds [ACK] Seq=285732 Ack=1208513 Win=16425 Len=0
NetApp
MyPC
1314
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:4096 Off:1044480 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
1514
TCP
[TCP segment of a reassembled PDU]
NetApp
MyPC
1514
TCP
[TCP segment of a reassembled PDU]
MyPC
NetApp
54
TCP
51321 > microsoft-ds [ACK] Seq=285849 Ack=1212693 Win=16425 Len=0
NetApp
MyPC
1314
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:4096 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
1514
TCP
[TCP segment of a reassembled PDU]
NetApp
MyPC
1514
TCP
[TCP segment of a reassembled PDU]
MyPC
NetApp
54
TCP
51321 > microsoft-ds [ACK] Seq=285966 Ack=1216873 Win=16425 Len=0
NetApp
MyPC
1314
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:512 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
650
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:64 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
202
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:512 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
650
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:4096 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
1514
TCP
[TCP segment of a reassembled PDU]
NetApp
MyPC
1514
TCP
[TCP segment of a reassembled PDU]
MyPC
NetApp
54
TCP
51321 > microsoft-ds [ACK] Seq=286434 Ack=1222393 Win=16425 Len=0
NetApp
MyPC
1314
SMB2
Read Response
MyPC
NetApp
171
SMB2
Read Request Len:4 Off:0 File: netapp\data$\test\B1\B2\1m-C.txt
NetApp
MyPC
142
SMB2
Read Response
February 18th, 2010 11:49pm
Hi Michael,
That's incredible the amount of detail you've put into this. I would love to know the cause of that extra traffic as well. I notice also you're always running this from your Windows 7 PC (but the data resides on a Windows Server). What if that data were to reside locally on the Windows 7 box and you tested the ROBOCOPY version in the same manner? Interesting problem.
Thanks,Kevin Costain
@calwell on Twitter
Calwell's Blog
Google Profile (Buzz)
Free Windows Admin Tool Kit Click here and download it now
February 19th, 2010 7:55am
Copying from the local Windows 7 PC to a Windows 2003 server yields same results -- too much extraneous CIFS traffic is generated if certain options are specified.I ran even more tests. In each, I used robocopy to copy from the local C: drive to a network share (Windows 2003). The director consisted of twenty 1 MB files contained within five subdirectories. The directories were already sync'd 100% before each test, so the robocopy command should not have needed to do anything other than a quick comparison. What's really interesting is the difference is traffic between "/e /purge" and "/mir" (which should be identical, right?)...robocopy c:\test r:\test /copyall /r:0 /w:0 /e - 497 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s - 233 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /e /purge - 497 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s /purge - 233 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /e /z - 497 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s /z - 233 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /e /purge /z - 497 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s /purge /z - 233 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /e /b - 3757 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s /b - 233 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /e /purge /b - 3757 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s /purge /b - 233 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /e /zb - 3757 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s /zb - 233 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /e /purge /zb - 3757 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s /purge /zb - 233 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /mir - 3757 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /mir /z - 3757 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /mir /b - 3757 packetsrobocopy c:\test r:\test /copyall /r:0 /w:0 /s /mir /zb - 3757 packetsSo, it appears as though "/mir" has problems AND "/e /b" or "/e /zb". Again, far too many packets between the PC and the destination share. - Michael
February 22nd, 2010 8:43pm
Hello,
I run into very similar problems having too many packets of the microsoft-ds type.
Did you find any solution to your problem ?
Thanks,
Simon.
Free Windows Admin Tool Kit Click here and download it now
July 19th, 2010 8:04pm
My solution? Stick with robocopy XP010. Except for keeping current date stamps for directories, it works 100%.
October 28th, 2010 3:18pm
Hi Michael.
I have just come across this post and am interested by what you are reporting. We are in the process of migrating from a Win 2003 fileshare to a CIFS Share on a IBM nSeries (NetApp rebranded). We used Robocopy (xp027) to undertake the migration. We didn't
notice any major problems but were not massively monitoring performance as we copied the data over a week (using the /mir command). However, during the migration testing we noticed real performance issues accesing the new CIFS Shares via a Mac OSX
(Snow Leopard) client. From the nSeries Share a 30MB Illustrator file was taking 5 minutes to open. We have, therefore, started looking at performance and used Wireshark to monitor a Mac copying data to the share. We are still unsure where the issue lies -
with the Mac client using CIFS or the nSeries. We feel it may be the Mac as the nSeries is also running an ESX Environment without issue. However, we are worried that Robocopy may have badly fragmented the data on the volume and may be generating too much
traffic. The majority of the data is Mac so we are also worried Robocopy may not be taking across all the data and resource forks. Have you either a) had any performance issues with your CIFS shares after using Robocopy or b) any idea where we could troubleshoot
further. Apple are currently blaming IBM and IBM are blaming Apple. All good fun.
Jon
Free Windows Admin Tool Kit Click here and download it now
April 5th, 2011 10:32am