I have a new server connected with 8gb Fibre to a Dell TL2000 with an Lto5 drive(IBM LTO-5 Drive). For testing I am pulling directly from an ISCSI device that is using 3 NIC's in MPIO. I can successfully pull 230-250MBps from this Array.

However, The tape is only writing at between 50-60MBps. I can monitor the speed with the Fibre Channel utility. The queue length never exceeds 1.  Also the Nic's never exceed 25% utilization and When I create a large copy form the array I can easily exceed 80% on all of them(when I am going at 250MBps)

CPU does not exceed 5% with 8 cores remaining parked and I still have 8gb of memory free.

So why can DPM not utilize the full bandwidth to feed the LTO5 drive? I need to feed the drive at over 200 so I can get into streaming mode and avoid shoeshinning the tape.

Thanks,

Nick

• Moved by Mike JacquetMicrosoft employee, Moderator Monday, July 23, 2012 11:19 PM (From:Data Protection Manager - General)

Hi,

What kind of data are we talking about - file server with lots of small / medium sizes file, or large SQL / Exchange data bases ?

Is the data compressable or is it mostly files that are already compressed like .vhd's, .jpg., .tiff files ?

Are you doing Disk to Tape (D2T) or Disk to Disk to Tape (D2D2T) backup ?

It is about 500GB of primarily large files 500mb-1gb with some small files added in. They are backup files from 2 software products so they are probably not very compressable.

For this test to remove any other factors I am doing straight Disk2Tape.

Thanks,

Nick

Hi,

OK, for data that is not highly compressable you may not achieve optimal write performance.

LTO Generation 5

With capacity of 3 TB (assuming a 2:1 compression), the latest generation in the LTO family provides data transfer speed of up to 280 MB/s (assuming a 2:1 compression)

About the only way to troubleshoot this would be to try D2D2T so we can see what kind of throughput we have when doing the tape backup from local disk to local tape and eliminate the network. 

That ISCSI san is my Local Storage - it is the same array I use for all D2D2T backups which is why I am doing a D2T test. I can get great performance off it but it is like DPM detects that it is over the network and throttles it to 1gb. It is not really over the network as it is a local drive to the server but running over Iscsi.

Thanks,

nick

Hi,

All D2T backups are performed over the network. 

Ok. I did a D2D2T and I am still getting just the 55-60MBps or about 200GB per hour when it is going to Tape. Is their any logging information related to the tape drivers for why it is throttling the transfer? Or if it is having to start and stop a lot?

Thanks,

Nick

Hi,

We have tools that mimick the way that DPMRA reads from disks and writes to tapes to try to narrow down where the problem is.  

DPMRA reads from a disk shadow copy and we have a tool that mimicks disk reads from a folder and shows disk read speeds in real time.

c:\>DPMFileIoPerf.exe -testIR -sourcedir "V:\Virtual machines\SMBEX01" -usesnap
Creating Snapshot for Volume : V:\
Snapshot Created Successfully : \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\
Starting DPM IR test...
I/O Speed:[67] MB/Sec;
I/O Speed:[71] MB/Sec;
I/O Speed:[69] MB/Sec;
I/O Speed:[67] MB/Sec;
I/O Speed:[67] MB/Sec;
I/O Speed:[66] MB/Sec;
I/O Speed:[66] MB/Sec;
I/O Speed:[66] MB/Sec;
...
..
.
I/O Speed:[63] MB/Sec;
I/O Speed:[63] MB/Sec;
I/O Speed:[63] MB/Sec;
I/O Speed:[63] MB/Sec;
Total Bytes Processed:[15739301352], Average I/O Speed:[66] MB/Sec;
Completed DPM IR test...

DPMRA writes and reads using 64K block size and 192K buffer size and the tape tool does the same to see what the raw write and read speed of the tape drive.

example of output:

C:\>tapeioperf \\.\Tape2147483646 64 192 1000 2

Status: Getting the handle for the tape \\.\Tape2147483646 ...
Success
Status: Setting Tape Information...
Success
Status: Loading the tape...
Success
Writing...[1]

Status: Writing ...
Block Size              : 64 KB
Buffer Size             : 192 KB
Number of Bytes written : 999 MB
Time Taken              : 8221 ms
MegaBytes/Second        : 121.518063

Writing...[2]

Status: Writing ...
Block Size              : 64 KB
Buffer Size             : 192 KB
Number of Bytes written : 999 MB
Time Taken              : 8159 ms
MegaBytes/Second        : 122.441476

Status: Rewinding Tape ...
Success
Reading...[1]

Status: Reading ...
Block Size              : 64 KB
Buffer Size             : 192 KB
Number of Bytes Read    : 999 MB
Time Taken              : 8938 ms
MegaBytes/Second        : 111.769971

Reading...[2]

Status: Reading ...
Block Size              : 64 KB
Buffer Size             : 192 KB
Number of Bytes Read    : 999 MB
Time Taken              : 9032 ms
MegaBytes/Second        : 110.606732

So, obviously, the bottleneck in the above test shows we have slow local disks, so we would never achieve greater throughput than the disk reads.

If you open a support case with us we can run the tools on the DPM server to get disk read and tape write speeds to see what is happening.

Mike,

You've mentioned compressable data and different file sizes before as having an impact on how DPM utilizes space on tapes. Can you elaborate or supply a link providing more info on this subject? I have a ticket in with MS on what I believe to be a compression problem but now am wondering if its possibly the data we are trying to backup is the issue.

Thanks

Other post; http://social.technet.microsoft.com/Forums/en-US/dpmtapebackuprecovery/thread/bc0045d1-ae99-420b-895e-2bf422cb8095

Hi,

Dpm is simply going to send data as read from disk shadow copy  (be that directly from protected server using D2T or  from local Replica) and send that data uncompressed to the tape drive.   If the protection group has the option selected to compress the data, then when DPM Send commands to the tape drive to set the block size (64K), it also sends the command to enable hardware compression.  Below is a snipped of a DPM trace for a PG that had HW compression option checked.

  <MTASetMedia>
    <StorageXml>
      <StorageType xmlns=""Tape</StorageType">http://schemas.microsoft.com/2003/dls/MediaManager/StorageXMLSchema.xsd"">Tape</StorageType>
      <TapeInfo DriveAccessPath=""\\.\Tape2147483643"" PhysicalBlockSize=""65536"" HWCompression=""true"" xmlns=""http://schemas.microsoft.com/2003/dls/MediaManager/StorageXMLSchema.xsd"" />
    </StorageXml>
  </MTASetMedia> 

CMTATapeInfo::InitializeFromDOM => m_ssDrivePath = [\\.\Tape2147483643] m_dwPhysicalBlockSize = [65536] m_bHWCompression = [1]

If the tape drive supports hardware compression, then as data is sent by DPMRA to the tape device, the hardware should compress it.  If the data sent is highly compressable, then you get two benefits.  A) faster write speeds since the data writen to the tape media is actually less than it was sent, so it doesn't need to physically write as much, thus the higher speed.  B) You get more data on the tape and so the bytes written by DPM exceeds the native capacity of the tape media.  Basically all data written that exceeds native capacity before the tape is marked full can help calculate how compressable the data was that was written.   Simply take the total bytes written and divide by native capacity. 

Let me know what you find.

Hi Mike,

thanks for some in deep info about dpm.

can you tell us where to get the tool for measuring i/o we are also having bit poor speeds using 8 lto5 drives with disk2tape backup. Even running only 2 drives to different hosts we only get about 30mb/s. Speed of disk2disk from hyper-v and exchange are ok. storage and network connection is a 1gbit/10gbit mix. But even on 10gbit fileserver (attached 10gbit iscsi storage) to 10gbit dpm server we get poor speeds. So probably we could find the bottleneck.

We also had a proactive premier case with ms regarding tape backup but none of them ever heard about such a tool. Only info was dpm is not that good for tape backup and you can adjust buffer queue size.

best wishes,

stefan

I worked with Non-premier support(normal ticket based on Software Assurance) and they were able to provide me with the tool. I was able to test my disk and my Tape drive. The disk was definitely slower at just 140MBs vs 350MBs for the tape but I was not getting that much speed when I was going to tape - only 90MBs. I still have the case open trying to find out why I don't get 140 but I suspect it is related to the Tape not running at full speed and running it's buffer empty and then stopping but we will see what they say.

Don't know if that info helps but wanted to share the info I had.

Nick

Hi Nick,

Have you ever get solved you slowness issue? I have let's say the same problem.

Regards,

Jozsef