For the current project I'm working on, I need to optimize the storage performance of the SMB (RDMA) file server. Since I have fast interconnects (40Gbs infiniband), I want to make sure I don't bottleneck on the storage performance. I've been using ATTO Disk Benchmark as a baseline to compare the performance of the different possible configurations, so I can determine their approximate overhead. This is not the ideal testing methodology but I think it gives a pretty good idea of the overhead of different configurations. Once I hone in on a config that is the right balance of speed and redundancy, I'll work on tuning and real world performance simulations. In the second phase of testing which I'll work on tomorrow, I'll test out some of the suggestions that were posted on this thread (thanks to BlackLiner for that link). Keep in mind that this may not be indicative of what you'll see on your hardware, it's very possible I might have problems that enterprise class hardware won't exhibit. 


Test machine specs:
Intel X5 3.2 Ghz, 6 core
24 GB Ram
5 x 120gb OCZ Vertex 3 SSD,  Read: 550MB, Write 500MB
2 x 60gb OCZ Vertex 3 SSD,  Read: 535MB, Write 480MB
4 x 1TB Western Digital WD1003FBYX-01Y7B1 HDD, Read:140MB, Write:140MB 
LSI 9220-8i 8xGBs SAS contoller - 8x PCIE 2.0 = 4GBs max bandwidth 

The mix of SSDs and HDDs was chosen based on the bandwidth limitation of the SAS controller. I wasn't sure how storage spaces would perform with a mix of SSDs and HDDs so I split the test in to two groups, one with just SSDs and another with all the disks. I also ran each test with drives formated in NTFS and ReFS so I could compare the differences in file system performance. 

Speed:
6xSSDs = 3820MBs 
4xHDDs = 560MBs 
Total = 4020MBs
Size:
7SSDs = 783GB 
7 SSDs & 4 HDDs  = 3.49 TB

Layout - simple
Max Read: 2574MBs
Max Write: 2593MBs
Disks - 7SSDs
Provisioning type - thin
Volume format - NTFS 

Layout - 2 way mirror
Max Read: 2776MBs
Max Write: 1344MBs
Disks - 7SSDs
Provisioning type - thin
Volume format - NTFS 

Layout - 3 way mirror
Max Read: 2747MBs
Max Write: 941MBs
Disks - 7SSDs
Provisioning type - thin
Volume format - NTFS 


Layout - simple
Max Read: 2130MBs
Max Write: 2105MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - NTFS 


Layout - Parity
Max Read: 954MBs
Max Write: 151MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - NTFS 


layout - 2 way mirror
Max Read: 810MBs
Max Write: 471MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - NTFS 


Layout - 3 way mirror
Max Read: 874MBs
Max Write: 306MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - NTFS 

Layout - Simple
Max Read: 948MBs
Max Write: 862MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - ReFS

Layout - Parity
Max Read: 908MBs
Max Write: 135MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - ReFS

Layout - 2 way mirror
Max Read: Inconsistant 33-3570MBs
Max Write: 141MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - ReFS


Layout - 3 way mirror
Max Read: 1077MBs
Max Write: 346MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - ReFS

• Edited by Dustyny1 Tuesday, August 28, 2012 10:59 PM

For the current project I'm working on, I need to optimize the storage performance of the SMB (RDMA) file server. Since I have fast interconnects (40Gbs infiniband), I want to make sure I don't bottleneck on the storage performance. I've been using ATTO Disk Benchmark as a baseline to compare the performance of the different possible configurations, so I can determine their approximate overhead. This is not the ideal testing methodology but I think it gives a pretty good idea of the overhead of different configurations. Once I hone in on a config that is the right balance of speed and redundancy, I'll work on tuning and real world performance simulations. In the second phase of testing which I'll work on tomorrow, I'll test out some of the suggestions that were posted on this thread (thanks to BlackLiner for that link). Keep in mind that this may not be indicative of what you'll see on your hardware, it's very possible I might have problems that enterprise class hardware won't exhibit. 


Test machine specs:
Intel X5 3.2 Ghz, 6 core
24 GB Ram
5 x 120gb OCZ Vertex 3 SSD,  Read: 550MB, Write 500MB
2 x 60gb OCZ Vertex 3 SSD,  Read: 535MB, Write 480MB
4 x 1TB Western Digital WD1003FBYX-01Y7B1 HDD, Read:140MB, Write:140MB 
LSI 9220-8i 8xGBs SAS contoller - 8x PCIE 2.0 = 4GBs max bandwidth 

The mix of SSDs and HDDs was chosen based on the bandwidth limitation of the SAS controller. I wasn't sure how storage spaces would perform with a mix of SSDs and HDDs so I split the test in to two groups, one with just SSDs and another with all the disks. I also ran each test with drives formated in NTFS and ReFS so I could compare the differences in file system performance. 

Speed:
6xSSDs = 3820MBs 
4xHDDs = 560MBs 
Total = 4020MBs
Size:
7SSDs = 783GB 
7 SSDs & 4 HDDs  = 3.49 TB

Layout - simple
Max Read: 2574MBs
Max Write: 2593MBs
Disks - 7SSDs
Provisioning type - thin
Volume format - NTFS 

Layout - 2 way mirror
Max Read: 2776MBs
Max Write: 1344MBs
Disks - 7SSDs
Provisioning type - thin
Volume format - NTFS 

Layout - 3 way mirror
Max Read: 2747MBs
Max Write: 941MBs
Disks - 7SSDs
Provisioning type - thin
Volume format - NTFS 


Layout - simple
Max Read: 2130MBs
Max Write: 2105MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - NTFS 


Layout - Parity
Max Read: 954MBs
Max Write: 151MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - NTFS 


layout - 2 way mirror
Max Read: 810MBs
Max Write: 471MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - NTFS 


Layout - 3 way mirror
Max Read: 874MBs
Max Write: 306MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - NTFS 

Layout - Simple
Max Read: 948MBs
Max Write: 862MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - ReFS

Layout - Parity
Max Read: 908MBs
Max Write: 135MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - ReFS

Layout - 2 way mirror
Max Read: Inconsistant 33-3570MBs
Max Write: 141MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - ReFS


Layout - 3 way mirror
Max Read: 1077MBs
Max Write: 346MBs
Disks - 7SSDs & 4HDDs
Provisioning type - thin
Volume format - ReFS

Hi there,

What is most important when creating the storage pool , you can use the disk that are appear on the computer management. You can combine with any type of disk. Storage pool it is just a collection of a physical disk.

Happy to help :-)

SS doesn't seem to be able to split the I/O's according to the hardware, neither can you choose yourself (apart of that Journal thing). I amcurious about your test results with SSD's being dedicated as Journal backing.

Can you provide 4k random r/w numbers? I guess your benchmarks show large block sequential r/w performance.

Howcome you didn't do any tests for the pure SSD array with the ReFS formatting? Truth be told this is probably the most interesting set of stats for myself!

What sort of CPU hit are you seeing during the Parity tests, and is there any CPU difference between NTFS and ReFS reads/writes?

@ChonKit  
Storage pools allow for parity data, disk stripping, virtualized disks & volumes, it is much more then just a collection of disks. It's also pretty apparent in the difference in performance for the various configurations.

@Sean 
2 reasons why I didn't test ReFS for the pure SSD array. 1) I didn't think of testing ReFS until after I destroyed the SSD storage pool 2) I only tested the pure SSD array so that I could compare what kind of impact the HDDs would have on the storage pool. This is really just internal data that I thought would be useful to give others a head start in their own testing. Once I get an overall view of the overhead associated with the different configurations, I'll be able to identify what configuration will give me the best performance that meets the needs of this project. It's a very specific use case so take my results with a grain of salt, different hardware configs might perform very differently. Later, I'll get in to tuning and more exhaustive testing which will include CPU overhead, memory use, que lengths, simulated real world tests, disk failure testing, rebuild times, etc. 

@BlackLiner
I only posted top results because it would have been a lot of work to type up the ATTO results (they really should have an export to CSV option IMO). I wanted to include the screenshots of the ATTO benchmarks but the forum software doesn't let you paste from a word doc.  I'm going to try to do some testing of how dedicated journal disks, if you can suggest some other testing software to show 4k performance I'll be happy to post the results for you. 

Here are the full results of the ATTO tests:

 

Sorry the forum only allows the attachment of 2 images per post.. annoying.. oops just noticed that the screenshots don't have the titles with the config. 

7SSDs-simple-thin-NTFS 

7SSDs-2way-mirror-thin-NTFS

• Edited by Dustyny1 Wednesday, August 29, 2012 1:17 PM

7SSDs-3way-mirror-thin-NTFS

7SSDs-4HDDs-Thin-1TB-1TB-Simple-NTFS

• Edited by Dustyny1 Wednesday, August 29, 2012 1:08 PM

7SSDs-4HDDs-Thin-1TB-1TB-Parity-NTFS

7SSDs-4HDDs-Thin-1TB-1TB-2way-Mirror-NTFS

• Edited by Dustyny1 Wednesday, August 29, 2012 1:09 PM

7SSDs-4HDDs-Thin-1TB-1TB-3way-Mirror-NTFS

7SSDs-4HDDs-Thin-1TB-1TB-Simple-REFS

• Edited by Dustyny1 Wednesday, August 29, 2012 1:09 PM

7SSDs-4HDDs-Thin-1TB-1TB-Parity-REFS

7SSDs-4HDDs-Thin-1TB-1TB-2way-Mirror-REFS

• Edited by Dustyny1 Wednesday, August 29, 2012 1:11 PM

This is a retest of the one above. I wasn't expecting to see such poor results with this config, so I destroyed and recreated the storage pool. Performance did change but didn't eliminate the problem. I'm not sure if this is a software or hardware issue. Later on. 

7SSDs-4HDDs-Thin-1TB-1TB-3way-Mirror-REFS

• Edited by Dustyny1 Wednesday, August 29, 2012 1:11 PM

@Sean 
2 reasons why I didn't test ReFS for the pure SSD array. 1) I didn't think of testing ReFS until after I destroyed the SSD storage pool 2) I only tested the pure SSD array so that I could compare what kind of impact the HDDs would have on the storage pool. This is really just internal data that I thought would be useful to give others a head start in their own testing. Once I get an overall view of the overhead associated with the different configurations, I'll be able to identify what configuration will give me the best performance that meets the needs of this project. It's a very specific use case so take my results with a grain of salt, different hardware configs might perform very differently. Later, I'll get in to tuning and more exhaustive testing which will include CPU overhead, memory use, que lengths, simulated real world tests, disk failure testing, rebuild times, etc.   

Fair enough - I am probably going to look at 2012 sometime this week once I am sure my backups are OK, I have a fairly similar config to yourself, 4x 160GB Intel G2 SSD, 6x Samsung F1 1.5TB, Dual LSI SAS cards on a system with AMD 6 core and 16GB RAM. I am mainly interested in running a strictly SSD volume for VMs on Hyper-V and a parity volume for all my video media files on HD, along with a mirror volume for pictures and work.

I am most keen to see what sort of hit the CPU takes when doing a lot of random read/writes to the SSD pool for the VMs, I don't have hugely lofty expectations but hope it is acceptable.

Thanks for the work done!

As far as i know, the ATTO values represent sequential access. I guess Iometer project has a lot more benchmark possibilities for such kind of SAN.

Throwing SSD's and HDD's into one pool seems to be useless in respect to performance efficiency. On creation of the SS, those SSD's were marked as Data i guess, so ES2012 is just filling it up with data as if it would be "cheap" HDD'space. As access times differ from HDD to SSD, mirroring/parity will become a headache and breaks up performance.

So there might even be an increase of performance if you would use just HDD's in one pool.

From what I can tell my results are off because of my hardware configuration, so at best this is an example of how a improper configuration will affect your performance. 

Based on David Trounce's (very informative) posts, I ordered 2 120GB SSDs to replace the 60GB ones and another 4 1TB HDDs. From what I read, I could better performance by using disks that are all the same size. Hopefully 8SSDs and 8HDDs will perform better. I should get the Hardware in tomorrow (love Newegg!). I'm going to also add in a 2nd SAS controller and test with that first as the SATA ports on the MB are split between controllers. I'll start a new thread to share the results of my new config.  

2. SS/Virtual disk performance guidelines. Based on testing different configurations, I have the following suggestions for parity virtual disks:

(a) Use disks in SS pools in multiples of 8 disks. SS has a maximum of 8 columns for parity virtual disks. But it will use all disks in the pool to create the virtual disk. So if you have 14 disks in the pool, it will use all 14 disks with a rotating parity, but still with 8 columns (1 parity slab per 7 data slabs). Then, and unexpectedly, the write performance of this is a little worse than if you were just to use 8 disks. Also, the efficiency of being able to fully use different sized disks is much higher with multiples of 8 disks in the pool.

I have 32 underlying disks but a maximum of 28 disks available to the OS (due to the 8 array limit for RAIDCore). But my best configuration for performance and efficiency is when using 24 disks in the pool.

• Edited by Dustyny1 Wednesday, August 29, 2012 4:15 PM

Thanks for sharing this, your results are very interesting. It's pretty clear from both your and my testing that writes to parity virtual disks are limited to the speed of a single underlying disk. With an SSD this is ~500MB/sec, with a HD it is ~65MB/sec.

I don't think you'll see any real performance benefit from swapping 60GB SSDs with 120GB SSDs. For me, there was only a slight performance hit from using different sized disks, and only when having >8 disks.

My testing was focused on maximizing parity virtual disk performance with large numbers of disks (>8). In this case, you can increase disk performance by setting up multiple virtual disks and striping them together, thus creating a RAID50. It's a shame the virtual disk doesn't do this automatically.

It's interesting to see how bad the performance of ReFS is too (approximately half that of NTFS). Given this, plus all the other compromises (no deduplication, no DFSR, no changing of volume size, can't be used for CSV - see http://social.technet.microsoft.com/Forums/en-US/winserver8gen/thread/0ff1cfd9-5fa8-40dd-9ac7-edccdce013d6), I can't recommend ReFS until it has matured. But hopefully it should eventually be very good.

• Marked as answer by Dustyny1 Thursday, August 30, 2012 7:36 PM

@BlackLiner What you are saying makes total sense though disappointing. Historically mirroring & parity have been synchronous transfers but with the speed differential between SSDs and HDDs we really need either async transfers or some sort of caching(?) system.

@David Would you be so kind as to give a quick write up on what you did exactly to create these virtual RAID50 set, I'd like to compare results with my HW.

I'm not saying creating a new file system is easy but I really had hoped M.S hit a home run with Storage Spaces and ReFS, they both have so much potential. Compared to the *nix world we are pretty limited file system wise, NTFS is great but having access to different file systems to meet different use cases is really ideal. I know its pie in the sky but I really wish we could install 3rd party file systems without having to use a hack like FUSE.

I totally agree with you about ReFS, it seems to have a very limited use case at this time but I'm not sure what that is. What I'd like to see is some fast iteration of updates instead of this tick-tock cycle of OS releases and service packs. Hopefully they won't make us wait for a SP or R2 to get these features up to speed.

• Edited by Dustyny1 Thursday, August 30, 2012 2:03 PM

I wrote up the steps involved in creating the virtual RAID50 here: http://social.technet.microsoft.com/Forums/en-US/winserver8gen/thread/79ca6d6d-cab7-4ff3-8c17-ec6ce249e641

The relevant section is:

(c) For maximum performance, use Windows to stripe different virtual disks across different pools of 8 disks each. On my hardware, each SS parity virtual disk appears to be limited to 490MB/sec reads (70MB/sec/disk, up to 7 disks with 8 columns) and usually only 55MB/sec writes (regardless of the number of disks). If I use more disks - e.g. 16 disks, this limit is still in place. But you can create two separate pools of 8 disks, create a virtual disk in each pool, and stripe them together in Disk Management. This then doubles the read and write performance to 980MB/sec read and 110MB/sec write.

It is a shame that SS does not parallelize the virtual disk access across multiple 8-column groups that are on different physical disks, and that you need work around this by striping virtual disks together. Effectively you are creating a RAID50 - a Windows RAID0 of SS RAID5 disks. It would be better if SS could natively create and use a RAID50 for performance. There doesn't seem like any advantage not to do this, as with the 8 column limit SS is using 2/16 of the available disk space for parity anyhow.

You may pay a space efficiency penalty if you have unequal sized disks by going the striping route. SS's layout algorithm seems optimized for space efficiency, not performance. Though it would be even more efficient to have dynamic striping / variable column width (like ZFS) on a single virtual disk, to fully be able to use the space at the end of the disks.

.

ReFS is clearly on the right path, but it's not very useful for a production file server yet. For me, key needs it doesn't support are DFSR, deduplication, automatic correction for parity blocks, and volume resizing. I can live without it for CSV for now, as I only use those for app data (SQL, Hyper-V) on a SOFS. SQL has it's own integrity metadata and can self-heal from a mirror using SQL availability groups, and Hyper-V has Hyper-V replica. Both also need the performance of NTFS. See also http://social.technet.microsoft.com/Forums/en-US/winserver8gen/thread/0ff1cfd9-5fa8-40dd-9ac7-edccdce013d6

My only option for file shares right now is NTFS on hardware RAID6 - pretty good, but there's no integrity checking. My experience is that CHKDSK can still take several minutes to fix during a reboot, even if the scanning is now done online with Server 2012, and you still can get silent file corruption.

I have been testing NTFS over a 10TB+ iSCSI block target, which runs using ZFS on Linux in a Hyper-V VM with pass-through disks from the host. It works very well for integrity, except that there is a confirmed bug (in Server 2008 R2 and also 2012 RTM) I have found that causes a random BSOD on the Hyper-V host with large (3TB+) pass-through disks and large I/O from file copies. Microsoft has accepted my business case for a QFE. Details here: http://social.technet.microsoft.com/Forums/en-US/winserver8gen/thread/cb6fbadb-f2a7-4b4e-904a-ccb1c72f413b

• Edited by David Trounce Thursday, August 30, 2012 6:35 PM

I'm starting to think for my purposes I wont be able to use a mix of SSDs and HDDs in storage pools. I knew it was a bit of a overreach but I was hoping Storage Pools could mix the two like ZFS can. 

There should be some kind of "ZFS" performance with CSV caching and dedicated journal backing disks. I don't have the hardware to test, and mine just went crazy

• Edited by BlackLiner Thursday, August 30, 2012 8:00 PM

I figured I'd skip it based on David's feedback but I really should give it a test before moving on. Thanks for the suggestion. 

There should be some kind of "ZFS" performance with CSV caching and dedicated journal backing disks. I don't have the hardware to test, and mine just went crazy

CSV caching does only read caching in RAM and it's not what you want to have. You really need to have read-write caching with write-back policy and use also flash as a level 2 - that's a way to go...

-nismo

• Marked as answer by Dustyny1 Friday, August 31, 2012 8:30 PM

That's the impression that I got from what David was posting, that write performance would take a big hit which would be especially problematic for us. I'm going to give EnhancedIO a test. Any chance you know of any other vendors who have a similar product?  

That's the impression that I got from what David was posting, that write performance would take a big hit which would be especially problematic for us. I'm going to give EnhancedIO a test. Any chance you know of any other vendors who have a similar product?  

Take a look @ FlashSoft (now SanDisk subdivision) and VeloBit. Unfortunately they do read caching only (Google for tricks to enable write caching with FlashSoft but it's out of a production quality yet). Upcoming version of StarWind will add level 2 ditributed write-back cache based on flash (to the one we already have in memory for level 1) but we don't have a public beta yet... There are A LOT of vendors coming with more or less the same solution so stay tuned :)

-nismo

Thanks I've been testing FancyCache it's in beta but I've gotten some really impressive results. 

With your SS testing on SSDs, have you noticed any performance degradation over time? When using conventional RAID5 it's generally recommended to avoid SSDs as the parity calculations will consume a large portion of the sectors even though there's no actual data on the file system, which will require erasing NAND blocks before being able to write again each time.

... You really need to have read-write caching with write-back policy and use also flash as a level 2 - that's a way to go...

-nismo

Afaik, dedicated journal backing disks should exactly improve the write part with the use of flash drives.

... You really need to have read-write caching with write-back policy and use also flash as a level 2 - that's a way to go...

-nismo

Afaik, dedicated journal backing disks should exactly improve the write part with the use of flash drives.

What you say is pretty close to ZIL in ZFS, log put on flash to fix random write issues with "old school" file systems like ZFS and NTFS are. ZIL is not a cache. What we're talking about right now is closer to L2 ARC, second level cache usually kept on flash memory. ZIL and L2 ARC (and DJBD and flash cache, component missing from NTFS and Windows OS out-of-box) are complimentary rather then competitive things. Ideally OS should do both - log-structuring to fight I/O blender and flash caching to improve write performance. @ file system or block level. 

-nismo

this one is interesting to me because it's very similar to my own ATTO result using just a pair of very fast SSDs.

Notice how my peak read performance is just over 4,000 MB/sec, then something happens with the next higher transfer size and the speed plummets.  Similar to what you show here, although your read performance picks back up with the 8192 xfer size.  Note this is NTFS, 4k sector size, 64k allocation unit.

I really do need to make a new thread for this...