Robocopy problem under Windows 8

I use ROBOCOPY to synchronize files between my desktop and laptop machines via a USB flash drive. ROBOCOPY is set to copy only updated files. It worked perfectly under Windows 7, reporting and copying only "newer" files; under Windows 8 it marks and copies the newer files and marks all the others as "modified" and copies them as well, even when the "modified" files haven't been accessed or written to.
Here is the command line I am using:

%SystemRoot%\system32\robocopy C:"\users\Robert Jacobs\Pix" i:\Pix /PURGE /E /XO /FFT /XA:SH /DST /R:5 /W:2 /LOG+:C:"\users\Robert Jacobs\My Documents\work"\leap_down.log /TS /NP

As I said, this worked perfectly under Windows 7; it seems to be broken in windows 8. How can I fix it?

All help will be gratefully appreciated.

Robert

December 14th, 2012 4:30pm

It can be caused by a different file systems. Have a look at this switch:

/FFT :: assume FAT File Times (2-second granularity).

Free Windows Admin Tool Kit Click here and download it now
December 19th, 2012 7:08am

Dear Niki,

I'm already using that switch on both machines -- please look again at the command line I cited in the original post:

%SystemRoot%\system32\robocopy C:"\users\Robert Jacobs\Pix" i:\Pix /PURGE /E /XO /FFT /XA:SH /DST /R:5 /W:2 /LOG+:C:"\users\Robert Jacobs\My Documents\work"\leap_down.log /TS /NP

Both machines and the flash drive are formatted in the NTFS file system.

Thanks so much for the suggestion. I'm coming to the opinion that the problem is a bug in the Windows 8 ROBOCOPY.

Robert

December 19th, 2012 2:16pm

Hi Robert,

Just wanted to mention that I am experiencing the same issue, where my Robocopy scripts which copied files correctly under Vista and Win7 now copies many more files, with this new file class "Modified".  For example:

robocopy "D:\Development\NTIS" "S:\Development\NTIS" /S /XA:RSH /XF *.scn *.tps /XO /NP /LOG+:"UpdatePassport.txt"
now copies files that have not been touched for years.

I cannot find any documentation for this "Modified" class, so I suspect there have been new "features" added to Robocopy in Win8 for which we have not yet received documentation.  Hopefully something will surface soon and we can exclude these files.

Best,
Scott

Free Windows Admin Tool Kit Click here and download it now
January 17th, 2013 7:22pm

Dear Scott,

Thanks for your note. I'm glad to learn that I am not the only sufferer - i.e., it's not my imagination!

I expected better of the TechNet forum -- one would thing there would be someone who knows enough about Win8/Robocopy to provide a solution to this most annoying problem.

All good wishes. If I learn anything I'll be sure to share it with you.

Robert

January 19th, 2013 7:03am

hmmm, I haven't noticed any change so far...but I'm using a simple mirror

Set FileDate=%date:/=%
Robocopy "C:\Data" "F:\DataBackup" /MIR /XA:SH /XD AppData /XJD /R:5 /W:15 /TS /NP /LOG:"F:\BackupLogs\DataBackup %FileDate%.log"

Free Windows Admin Tool Kit Click here and download it now
January 19th, 2013 6:20pm

Dear Mark 522010,

Thanks for your note. I don't think your command line differs from mine in essentials; I've tried it with and without MIR.

I'm starting to wonder whether it has anything to do with the transfer vehicle: the intermediate carrier is an NTFS-formatted USB flash drive. Both the desktop and laptop machines are also NTFS formatted, a RAID array in the case of the former.

Two of the files being synchronized are encrypted and may not be transfer over an FAT32 flash drive. I'll experiment with that and report back.

Robert

Later: FAT32 will not transfer encrypted files, so it's of no help to me. In any event the commands I use worked perfectly under Windows 7.


  • Edited by formerprof Sunday, January 20, 2013 2:03 AM
January 19th, 2013 9:15pm

Could this be caused by the same problem that causes Windows File History to backup unmodified files?

The following thread discusses this topic: Windows 8 File History - Excessive backup - Excessively saving copies of files (photos, videos, documents, etc.)

The fix there was to disable Homegroup functionality.

The question there was:

Why is File History saving so many duplicate copies of my media, when I'm not making any changes to the media or metadata?!?
 Does anybody know? 

and the final solution to date is:

Since I switched HomeGroup off, File History appears to be working sensibly. Only changed files appear to being backed up. My NAS is noticably quieter and my laptop far more responsive (as it isn't constantly backing up!!)

Thanks all - nearly gave up on File History. More than happy to live without HomeGroup as it really isn't needed in Win 8 as Noel helpfully pointed out above.

Could this be it? As far as I know, BOTH the File History and Robocopy use Shadow Copy functionality to copy locked files on-the-fly. So it looks like, the problem lies with how Shadow Copy works when Home Group functionality is turned on.

Gentlemen, could you please verify that and check if switching Home Group off helps here as well o

Free Windows Admin Tool Kit Click here and download it now
January 20th, 2013 10:05am

I just disabled the homegroup on both computers. It will be a day or so before I can tell for sure whether the problem is gone. Thanks so much for the suggestion!

Robert

January 20th, 2013 8:08pm

Robert, could you please notify us of the result no matter positive or negative?
Free Windows Admin Tool Kit Click here and download it now
January 20th, 2013 9:47pm

As I told you, I disabled the homegroup on both machines. Then I worked on the laptop for several hours, viewing a number of files and writing to two or three. On running Robocopy the files to which I had written were marked "newer" and the files which had been viewed but not written were marked "modified." Apparently any access to a file is detected by the new Robocopy which then marks them "modified." There must be some background process which accesses the files; in my case that might be Acronis True Image Home, but I doubt it -- I think it's something inside Windows 8. Not enough time has elapsed yet to see if it is the Homegroup process. I'll report on that in another day or so.

The underlying problem is not the background access to files; it seems to be the Win8 Robocopy's insistence on copying "modified" files which have not in fact been updated.

Please -- if anyone else has any insight into this matter I'd be most grateful for any help.

Robert

 
January 21st, 2013 12:34am

Yesterday, I disabled both Homegroup services, rebooted, and ran my Robocopy script (to flush any current file class attributes).  I worked a bit yesterday and today and ran the Robocopy script again.  There are still hundreds of files in folders that have not been accessed by any software being copied by Robocopy with a class of "Modified".

Glad to get the Homegroup link out of Explorer, but wish that would have been the fix.  It wasn't.

Best,
Scott

Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 3:38pm

Pity...

Could you confirm that affected files (those that appear as modified although have never been) have different formats, or are they mostly media formats like pictures, music, and videos?

If these are mostly media files, do you by chance have Media Streaming enabled?

January 21st, 2013 3:46pm

Actually, these are source code files (primarily text, but some binary).  There are a couple of image files, and they are repeatedly copied, but so are EXE, DLL, and other binaries (such as proprietary database files, etc.).  Also - none of these folders are under "My Documents", so I wouldn't expect most Windows processes to be aware of them.  I do have the indexing service running, and configured this drive to "Allow files on this drive to have contents indexed in addition to file properties".  I am going to disable that and see what happens.

Thanks,
Scott

Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 4:02pm

Dear Scott,

So far my experience with the disabled homegroup is much more promising. After the first run of my script, on which a few "modified" files still turned up, the back and forth between my laptop and my desktop has been clean. I don't discount the possibility that the problem may arise again. Let me suggest that you try (I kind of hope you haven't already done so!) the /FFT switch for Robocopy. Several people have reported that time variations between the machines can produce the "modified" problem.

I should add that my automatic Acronis system backup took place overnight. Every file in the machine must have been touched somehow, but when I ran the script this morning no files were listed as modified.

Dear Exotic Hadron,

All files of whatever type (within the directories which I am synchronizing) have shown up. There's been no distinction between media files, WORD files or database files.

Thanks to you both for helping with this -- all good wishes.

Robert

January 21st, 2013 4:05pm

Hi,

Your problem is most surely related to this one.

http://social.technet.microsoft.com/Forums/en-US/w8itprogeneral/thread/88fc0286-4328-48d9-b945-5c280bce58b6

W8 files are being "touched" on a continuous basis by some  Windows background process.

So both Robocopy and Backup see your files as "modified".

Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 4:11pm

FWIW, I'm running W8 x64 Enterprise, there are no HomeGroups in my domain, and File History is not on.   I'm also experiencing Robert's problem with robocopy when copying Intel drivers from my E: drive to a USB key.
January 21st, 2013 4:21pm

Dear Sebastian,

Thanks for the pointer. There seem to be at least two controllable functions which touch files. One is the "file history" function reachable via the control panel. I have that switched off. The other is file indexing, also seen in the control panel. I seem to have indexing set to "on" for the directories which are included within my script.

At the very least Robocopy should have a switch which lets it ignore modified files which have not in fact been changed. Under windows 7 this problem didn't exist; of course, I don't know whether Robocopy has been changed, or, if it has, why.

I'm going to try to force the indexing to see if "modified" recurs; I haven't seen it since I left the homegroup yesterday.

Robert

later: I forced the reindex, ran the script and had no "Modified" files in the Robocopy log.

  • Edited by formerprof Monday, January 21, 2013 4:47 PM
Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 4:28pm

Hi Sebastian,

the topic you are linking to looks to be exactly the same as described in this topic, which already provides some possible workarounds: Windows 8 File History - Excessive backup - Excessively saving copies of files (photos, videos, documents, etc.).

P.S.: That's just to link similar topics altogether.

January 21st, 2013 4:57pm

By the way, gentlemen, since File History utilizes NTFS USN journal, what if we check it with fsutil to find modifications?

And, by the way, some gentleman here confirms he has been able to workaround the issue with robocopy considering files as modified by installing previous version on rob

Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 4:59pm

This topic reports some workaround.

Robocopy to Network Share, is it just completely broken?

"I found it.  It's the ARCHIVE attribute on the source files.  On the new Robocopy, the presence of the "A" attribute seems to force the file to be copied.  The problem is, Robocopy doesn't then reset the attribute. 

And if you use /M (copy only "A" files, then reset "A"), any files without the "A" attribute set get ignored - even if they should have been copied because they don't exist on the target, etc.  So that doesn't work.

The solution "appears" to be to reset the "A" attribute on the structure you want to copy BEFORE you do the copy.  Then your script "should" work.  (Mine did.)  I used: ATTRIB -A C:\MYDIR\*.* /D /S to clear the attributes.  Then Robocopy started working.

Of course, going forward, modified files will get "A" set, and then Robocopy will start copying those files every time again, so what I did was to add the ATTRIB command AFTER the Robocopy command in my script.  Run Robocopy, then reset "A"s.  That "should" yield the desired result, I hope!"

There's a note however: "Resetting the archive attribute greatly reduces modified file copies, but does not totally eliminate it."

January 21st, 2013 5:13pm

Aha! The archive bit does seem to invoke Robocopy's "modified" switch. I've tried the Robocopy switch COPY:DT (the default is /COPY:DAT) and tried setting and resetting the archive bit on one of the directories in my script. I ran Robocopy both ways and did not get any "modified" files with the /COPY:DT switch set.

The archive bit may not be the only thing which causes files to be marked modified, but it certainly seems to be one of them.

Robert

Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 6:09pm

Well,  both workarounds provided by Exotic Hadron are worth a try.
January 21st, 2013 6:10pm

Drat! I reestablished the homegroup, ran the script with /COPY:DT and was deluged again with modified files. Now I have to try adding the ATTRIB commands found in the link suggested by Exotic Hadron.

Robert

Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 6:42pm

I think that we had better stop using robocopy for now as it's starting to be a problem... When it poses so many possible problems with incorrect comparison/backup, who would know when it quirks the other day?
January 21st, 2013 6:54pm

I'd be happy to abandon robocopy were there something which offers the same facilities: I have to be able to move encrypted files and to exclude certain files, certain subdirectories and, of course, to copy only newer files.

So far, abandoning the homegroup seems to be the most useful.

Robert

Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 7:44pm

Would you please mark it as a temporary answer then for others to be able to clearly see the solution without having to read through all the posts in topic?

January 21st, 2013 7:49pm

Done. I'm a little slow at catching on to how this forum works, but I think I got it right.

Robocopy is the villain -- how does one attract Microsoft's attention so that they will fix the bug, preferably before I get too senile (Wednesday would be excellent) to appreciate it?

Many thanks for your help.

R.

Free Windows Admin Tool Kit Click here and download it now
January 21st, 2013 8:11pm

Same issues here....   I've reverted back to the one from the 2003 resource kit which seems to work ok.

http://www.microsoft.com/en-us/download/details.aspx?id=17657

Arjan

  • Proposed as answer by Exotic Hadron Thursday, February 21, 2013 3:57 PM
February 20th, 2013 6:03pm

My question is how do I find and delete all the copies that were made? My new computer's hard drive 150 GB is FULL and I don't know even how to find the copies to delete them all!!! Yikes!!!
Free Windows Admin Tool Kit Click here and download it now
June 5th, 2013 7:54pm

I assume you are talking about file history taking up your hdd.

Press windows + W to search settings.
Type "File History" and click on it.

Click advanced settings

Click on Clean up versions

OR

c:\windows\system32\FhManagew.exe -cleanup age

where age = days (0=all but current)

Arjan


June 6th, 2013 11:34am

Does anyone know if there is a ROBOCOPY.EXE update in the works to fix this issue?

I, too, started having issues with ROBOCOPY.EXE copying files that were not modified after upgrading to a Windows 8 Professional computer. 

Free Windows Admin Tool Kit Click here and download it now
August 21st, 2013 4:28pm

Any news on this topic?

I just switched from Windows 7 to Windows 10 and ran into this problem as well.
So it still persists. I am doing backups from NTSF to ext4 on a Linux-NAS and having the issue.
Same robocopy script ran fine with Vista and Windows 7.

A lot of files in my backup are reported as "same", but since Windows 10 many are regarded as "changed" which was not the case with robocopy running on Vista or Windows 7.

Just to be clear:
Known and resolved issues already present in Windows 7 (especially for backups to non-NTSF file-systems):

- using the /FFT switch

- using the /DST switch

- i'm using /COPY:DT and /DCOPY:T


Why are many files regarded as "changed" by robocopy?

Any help still appreciated.

September 7th, 2015 5:28am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics