Hi All,

We are in the process of migrating 54 TB (upward of 100 million files) from a group of file servers (mix of 2K8R2 and 2K3R2) to a new storage solution using Robocopy pushing data from the source (destination has no native utilities to pull the data).

I have written some powershell which enumerated each parent directory and kicks off a robocopy process for each sub-directory to attempt to speed things along.

I am wondering if anyone who has done large data migrations like this previously has explorer adding an A/V process exception (We are using SEP) for the robocopy process and whether there are any measurable performance benefits in doing so.

We do need to go through formal approval before making changes, and I know there would be no recommendations in excluding the process, as this would form a security hole, but if we were to place the executable in a set location and only launch that version, then we could apply security through obscurity for the data migration phase.

Thanks in advance.

Hi,

I did not have a change to migrate such number of files. But personally if your antivirus application will check each file written to your system during the migration, it will affect computer performance for sure. 

As it is a Robocopy process, if no one could write to the source during the migration, my opinion is that you can check for virus before it happens and set exception during the process to workaround this. 

Hi Shaon,

There is not a worry about the files being scanned during the process, as we cannot turn off A/V, but any overhead which is seen in the in memory scanning of the process.

The overhead is small with 1 robocopy, but when you are running  600+ simultaneous, then that overhead and management multiplies.

Speaking of which, are you aware of any optimized limits of simultaneous running robocopy processes? I only ask as I seen to be seeing a point where I launch a large number and after a certain number they seem to sit there in a "wait" state waiting for CPU time, and this could be for some time. If this CPU time slicing is preventing other copies from finishing earlier, then, this could produce an overall extension in the time for the large copies to complete.

We are currently running the delta copies to make life a little easier for us come the final migration at some point down the path.

I also think we may plan to take some time for the migration. Simultaneous cannot resolve the issue as it is still depends on system resource.

Robocopy will not copy existing files, so even the migration is stopped in the middle, you can still restart it without overwriting most of these files. 

I have a request in currently with our managed services vendor to make the required changes and will have an idea of the overhead saved once implemented.