We're running performance tests on two new Standard DS13 (8 Core, 56 GB) VMs (both using the latest/default Windows 2012 R2 image) backed by Premium Storage and have hit a wall with step 1 in testing the local SSD performance.

We understand 25% of the 400GB local SSD for these VMs is made available as temporary storage and the other 75% is used for Premium Storage caching:

http://azure.microsoft.com/blog/2014/12/11/new-premium-storage-backed-virtual-machines/

On the remaining 25%, we expect to see performance along these lines:

http://www.brentozar.com/archive/2014/09/azure-really-60-faster/
http://azure.microsoft.com/blog/2014/10/06/d-series-performance-expectations/

... but Crystal Disk Mark shows it *crawling* along:

  

           Sequential Read :     4.097 MB/s
          Sequential Write :     4.096 MB/s
         Random Read 512KB :     4.112 MB/s
        Random Write 512KB :     4.112 MB/s
    Random Read 4KB (QD=1) :     2.057 MB/s [   502.3 IOPS]
   Random Write 4KB (QD=1) :     2.057 MB/s [   502.2 IOPS]
   Random Read 4KB (QD=32) :     2.048 MB/s [   500.0 IOPS]
  Random Write 4KB (QD=32) :     2.047 MB/s [   499.7 IOPS]

  Test : 50 MB [D: 7.2% (8.1/112.0 GB)] (x5)
  Date : 2015/02/14 15:35:41
    OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

The performance of the OS disk is better but nowhere close to the 150 MB/s you'd expect for a P20 disk (assuming that's what's allocated for the default 127GB OS disk).

Expecting:

http://azure.microsoft.com/en-us/documentation/articles/storage-premium-storage-preview-portal/

Seeing:

  

           Sequential Read :    66.031 MB/s
          Sequential Write :    63.034 MB/s
         Random Read 512KB :    65.861 MB/s
        Random Write 512KB :    63.580 MB/s
    Random Read 4KB (QD=1) :     2.097 MB/s [   511.9 IOPS]
   Random Write 4KB (QD=1) :     2.047 MB/s [   499.7 IOPS]
   Random Read 4KB (QD=32) :     2.086 MB/s [   509.3 IOPS]
  Random Write 4KB (QD=32) :     2.078 MB/s [   507.4 IOPS]

  Test : 50 MB [C: 12.9% (16.4/127.0 GB)] (x5)
  Date : 2015/02/14 15:46:35
    OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

And the performance of the P30 disk (with ReadOnly cache) isn't much better:    

* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   204.567 MB/s
          Sequential Write :    39.677 MB/s
         Random Read 512KB :   204.549 MB/s
        Random Write 512KB :    34.865 MB/s
    Random Read 4KB (QD=1) :    20.951 MB/s [  5114.9 IOPS]
   Random Write 4KB (QD=1) :     1.666 MB/s [   406.7 IOPS]
   Random Read 4KB (QD=32) :    20.893 MB/s [  5100.9 IOPS]
  Random Write 4KB (QD=32) :    20.944 MB/s [  5113.4 IOPS]

  Test : 50 MB [E: 0.0% (0.2/1023.0 GB)] (x5)
  Date : 2015/02/14 15:22:59
    OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

When compared to our current CloudDrive with host caching deployed on D13s (note the performance of 4KB random reads):

           Sequential Read :   136.711 MB/s
          Sequential Write :    10.210 MB/s
         Random Read 512KB :   190.744 MB/s
        Random Write 512KB :     9.063 MB/s
    Random Read 4KB (QD=1) :    10.813 MB/s [  2639.8 IOPS]
   Random Write 4KB (QD=1) :     0.508 MB/s [   107.5 IOPS]
   Random Read 4KB (QD=32) :   106.533 MB/s [ 26009.1 IOPS]
  Random Write 4KB (QD=32) :     9.363 MB/s [  2286.0 IOPS]

  Test : 50 MB [F: 4.1% (24.9/600.0 GB)] (x5)
  Date : 2015/02/14 20:25:01
  OS : Windows Server 2012 Datacenter (Full installation) [6.2 Build 9200] (x64)

And this is what SQLIO reports for the local SSD:

C:\Program Files (x86)\SQLIO>sqlio -dD
sqlio v1.5.SG
1 thread reading for 30 secs from file D:testfile.dat
        using 2KB IOs over 128KB stripes with 64 IOs per run
size of file D:testfile.dat needs to be: 8388608 bytes
current file size:      0 bytes
need to expand by:      8388608 bytes
expanding D:testfile.dat ... done.
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   499.38
MBs/sec:     0.97

And for the P30:

C:\Program Files (x86)\SQLIO>sqlio -dE

sqlio v1.5.SG
1 thread reading for 30 secs from file E:testfile.dat
        using 2KB IOs over 128KB stripes with 64 IOs per run
size of file E:testfile.dat needs to be: 8388608 bytes
current file size:      0 bytes
need to expand by:      8388608 bytes
expanding E:testfile.dat ... done.
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  5103.03
MBs/sec:     9.96

The 5000 IOPS advertised for the P30 is holding up but what about the 200 MB/s throughput per disk?

NOTE: Attempts to create the P30 data disk with ReadWrite cache policy result in:

Update-AzureVm : BadRequest: The disk cache setting ReadWrite is not supported for DataVirtualHardDisk.

Any guidance would be appreciated:

• Why is the local SSD storage throttled at 500 IOPS and 1-4 MB/s throughput?
• How do we achieve 200MB/s on writes as we see with reads on P30s, what's the test to run?
• Can you publish I/O benchmarks that we can run to validate our setup?

• Edited by Nariman H Sunday, February 15, 2015 4:18 PM

Curious as to why that's the guidance as the same scenario repeated with a P30 data disk HostCaching set to None yields poorer performance across the board, at least with this crude benchmark:

           Sequential Read :    66.756 MB/s
          Sequential Write :    40.536 MB/s
         Random Read 512KB :    60.273 MB/s
        Random Write 512KB :    36.146 MB/s
    Random Read 4KB (QD=1) :     1.868 MB/s [   456.1 IOPS]
   Random Write 4KB (QD=1) :     1.679 MB/s [   409.8 IOPS]
   Random Read 4KB (QD=32) :    20.922 MB/s [  5107.8 IOPS]
  Random Write 4KB (QD=32) :    20.911 MB/s [  5105.1 IOPS]

  Test : 50 MB [F: 0.0% (0.2/1023.0 GB)] (x5)
  Date : 2015/02/16 8:19:19
    OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)
  

C:\Program Files (x86)\SQLIO>sqlio -dF
sqlio v1.5.SG
1 thread reading for 30 secs from file F:testfile.dat
        using 2KB IOs over 128KB stripes with 64 IOs per run
size of file F:testfile.dat needs to be: 8388608 bytes
current file size:      0 bytes
need to expand by:      8388608 bytes
expanding F:testfile.dat ... done.
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   532.16
MBs/sec:     1.03

• Edited by Nariman H Monday, February 16, 2015 1:26 PM

Hi Nariman H,

The host drive on DS series currently has a 500-IOPS limit. That is the reason that you see the IOPS ~500 IOPS.  We will be removing the limit in a week or two time frame so you should be able to see higher IOPS number on the host drive.

As for your second question about achieving 200 MB/sec on writes, you can run a test with your favorite load testing tool using higher IO Size such as 256 KB with 5 or 6 worker threads to obtain the highest throughput.  I was able to drive up to 200 MB/sec with the aforementioned configuration.

• Marked as answer by Michael CurdMicrosoft employee, Moderator Friday, March 06, 2015 4:51 PM

Hi Aung,

Can you tell us what the IOPS for the local SSD will be after the 500-IOPS limit is removed?

I am disappointed in the storage performance of the DS-Series compared to the D-Series overall. The DS-Series Premium Storage data disks at ~5,000 IOPS are around 10x faster than Standard disks at 500 IOPS, however they are 5x slower than the D-Series temporary disk (D:) at ~25,000 IOPS. Since the DS-Series does not have that fast local SSD temporary drive, my disk intensive applications are much faster on the D-Series (but only after the cache stored on the temporary SSD disk is warmed up).

What I was hoping for in the DS-Series was, at a minimum, a D-Series with Premium Storage data disks, but that is not what I got.

NOTE: IOPS (read) values obtained by the following on each disk type:

"c:\Program Files (x86)\SQLIO\sqlio.exe" -kR -t8 -f32 -b8 -BN

Thanks,

 - Harlow.

Hi Harlow,<o:p></o:p>

We are planning to update the limit for the local SSD on Azure web site shortly. Once done, I will reply back to you with the URL.  <o:p></o:p>

If they want more IOPS on a single volume, you can group together more Azure data disks (pool) and present a unique volume (space) using Storage Spaces.<o:p></o:p>

With DS 14, you can group up to 32x P30 disks, which will potentially drive up to 50,000 max IOPs per VM.

• Proposed as answer by Aung Oo - Microsoft 39 minutes ago