Hi guys,

I have a little question about virtualisation and a Hyper-V scenario for a HA Cluster.

In the past I have already set-up a HA Cluster for Hyper-V. This had the following set-up:

• Dell Equallogic (SAN)
• 2 Windows Server 2008 R2 servers which were configured in the same cluster
• The 2 Servers were directly connected through ISCSI with the SAN and the volume was configured as a CSV-volume.
  
• The Hyper-V role was installed on both windows servers
• All of the vhd files were stored directly under C:\ClusterStorage\

At the moment I am studying for my MCSE certification and I just learned about HA in the CBT-nuggets series.
It looks that they implemented another scenario:

• Windows server setup with storage spaces (just to replace the SAN, I guess. Same performance?!)
• 2 Windows Server 2012 servers which were configured in the same cluster
• The 2 Servers were directly connected through ISCSI with the storage-server and the volume was configured as a CSV-volume.
• This CSV-volume is then used in a Scale-Out File Server to make a HA File Server
• The VHD-files are then placed on this SMB 3.0 fileshare

It is a nice scenario, but why go through all the trouble of configuring a SOFS and accessing these VHD-files through SMB?

I know SMB 3.0 has some greatly improved features, but is it still better than directly accessing the CSV which is connected through ISCSI with the SAN/WS2012 ISCI target?

Would this set-up be any different when a SAN or WS 2012 server is used as storage?

In the CBT-nuggets video's, they are talking about HA from end-to-end.. But to be honest, I don't see more HA when using this SOFS. When the storage/SAN is down, the whole set-up won't work?!

Thanks for all the suggestions about this set-up. I just want to encorporate all the new features that WS2012 offers, but at this moment I only see a downside of this feature since it brings another extra system to the set-up which (I guess) will reduce performance?

Thanks again!!

Kind regards,

Sven Celis

Hi Sven,

SoFS and Storage Spaces are new technologies introduced in Windows Server 2012 and further leveraged in Windows Server 2012 R2. I would recommend you to review the following links that provide further information on SoFS and Storage Spaces.

http://social.technet.microsoft.com/wiki/contents/articles/15198.storage-spaces-overview.aspx

http://blogs.technet.com/b/askpfeplat/archive/2012/10/10/windows-server-2012-storage-spaces-is-it-for-you-could-be.aspx

Let me know in case you have any specific questions with these great technologies.

Cheers,

Victor

Hi Victor,

First of all a big thanks for your reply!! Good to mention Windows Server 2012 R2, almost forgot the R2 version was already out. :)

I know the advantages of an SoFS, but to be honest, I am still not convinced:

• When you already have a SAN, would you make the switch to just a windows server (with storage spaces) which is configured as an SoFS?
• When you already have the SAN connected to your physical servers through ISCSI. Would you migrate the current servers to Windows Server 2012, implement a high available SoFS server and then place the vhd files on this SoFS instead of directly to the volume of the SAN?

I am really willing to use storage spaces in combination with SoFS, but if you have already invested in a SAN will it improve the performance? Would it be better to move the storage to an SoFS which will be hosting all of your data (instead of directly on the SAN)? In the CBT-nugget series, they configured it like that.

Thanks in advance!

Kind regards,

Sven

It's difficult question to answer on here...

Just to be clear you don't need Storage Spaces to have SoFS. They are two separate features that do complement one another however...

Scale out file server gives you some options which could be out of reach... For example if you had a FC SAN with 2 controllers and put a 2 node SoFS cluster in front you can present the FC storage as a CSV to the hosts which hosts would then be able to access via SMB, if you've got 16 hosts then you don't need to put in 16 dual port FC cards plus associated FC switches... Additionally you could hook your SoFS hosts to several different types of storage and present them as CSVs. So you could hook your SoFS hosts to a range of SANs FC, iSCSI, FCoE, etc. and present all of this storage to hosts as CSVs. If you've got multiple Hyper-V clusters then you can use a single SoFS cluster for that storage, if you want to move a VM between clusters, under traditional storage rules you'd move all the VHDs for the VM too; using SoFS you don't (provided all hosts have access to the same shares) you just move the config... It makes sense when you start to get a larger infrastructure.

The other to think about is the CSV Cache... Under Windows Server 2012 you can use 20% of RAM as a CSV read cache (that can be a big a read cache). Under 2012 R2 you can use 80%, if you've got 96 GB RAM in the server that's a read cache of 76.8 GB... 

Storage Spaces allows you use commodity hardware (provided it's supported) to create pools of storage and then carve these into spaces for storage. In Windows Server 2012 it's OK, especially if you don't have a SAN. When I first saw this feature my first thought was "poor man's SAN" however R2 makes Storage Spaces a real SAN contender but using commodity hardware. As you're studying for MCSE I'd try and keep the R2 stuff out of your head unless you know you'll be sitting exams with R2 content.

I could go on for hours about this stuff. 2012 is the base of a large framework that's only going to get better...

Wow thanks for your answer. I didn't see it like this before. It makes sense indeed when the environment is getting bigger. This is still unclear to me:

• You mention that when you've got multiple hyper-v clusters you need to move the storage also (under traditional rules). But when all of the pyshical servers are configured in the same cluster, they all have access to the same CSV? So no storage move is required?

CSV cache is something new to me, I will certainly review this feature, thx!! Maybe a stupid question, but where did you learn this stuff? I've read some blogposting etc for the new features of Windows Server but have not seen this before.

This was my thought also :) a poor man's SAN.. But correct me if I'm wrong, but isn't this just a software RAID which will not out-perform a hardware raid? So you wouldn't buy any software/hardware RAID controllers, but just a machine with enough SATA/SAS connectors to just hook up the drives and then use storage spaces in the future?

Thanks for the reply on my thread!!

OK...

If you've got multiple clusters connecting back to a SoFS they don't see a CSV they see a SMB share (or several) so they don't care what's underneath that share storage wise, as long they've got access, i.e. Full Control on the folder and share. For example if Hyper-V Cluster A uses Share 1 for storage and Hyper-V Cluster B uses Share 1 also if you want to move a Virtual Machine Z from Cluster A to B and the files for VM Z are stored on Share 1 what needs to move between the clusters? The vm role in the failover cluster and the memory. You'd be doing a Live Shared Nothing Migration of the entire VM from one cluster to another but the only things that would actually move is the VM role and memory of the VM.

What is a hardware RAID controller? A connection, for example iSCSI, a processor (x86/x64), some RAM, an operating system. What is a server?

For storage spaces to work optimally you're best off using a JBOD, there are some good ones coming out now (make sure to use a certified one in production or you'll have support issues and probably functionality issues too) (http://www.windowsservercatalog.com/results.aspx?&chtext=&cstext=&csttext=&chbtext=&bCatID=1642&cpID=0&avc=79&ava=0&avq=0&OR=1&PGS=25&ready=0)

There are some specific requirements for storage spaces due to shared SAS which is why you want certified hardware.

You can just have a server with a shed load of disks in but that doesn't offer a lot of resiliency/redundancy.

Just a note that Storage Spaces must be able to see the disk, you can't put RAID in front of a disk presented to Storage Spaces.

As to where I learnt this stuff well... Microsoft run a lot of free training events (with MVPs and with their own Technical Evangelists) in the UK, Microsoft Virtual Academy is very good. Additionally I work for a Microsoft Gold partner doing Hyper-V and System Center implementations/VMware to Hyper-V migrations too. I also went to TechEd this year, the content for which is available on Channel 9, but going to TechEd allows you to ask all those really awkward questions of MS Product Managers that are a nightmare to find answers for.

Best advice is to get some kit and play with this stuff - watching a video about installing something is not the same as installing it yourself! 

Hi guys,

I have a little question about virtualisation and a Hyper-V scenario for a HA Cluster.

In the past I have already set-up a HA Cluster for Hyper-V. This had the following set-up:

• Dell Equallogic (SAN)
• 2 Windows Server 2008 R2 servers which were configured in the same cluster
• The 2 Servers were directly connected through ISCSI with the SAN and the volume was configured as a CSV-volume.
  
• The Hyper-V role was installed on both windows servers
• All of the vhd files were stored directly under C:\ClusterStorage\

At the moment I am studying for my MCSE certification and I just learned about HA in the CBT-nuggets series.
It looks that they implemented another scenario:

• Windows server setup with storage spaces (just to replace the SAN, I guess. Same performance?!)
• 2 Windows Server 2012 servers which were configured in the same cluster
• The 2 Servers were directly connected through ISCSI with the storage-server and the volume was configured as a CSV-volume.
• This CSV-volume is then used in a Scale-Out File Server to make a HA File Server
• The VHD-files are then placed on this SMB 3.0 fileshare

It is a nice scenario, but why go through all the trouble of configuring a SOFS and accessing these VHD-files through SMB?

I know SMB 3.0 has some greatly improved features, but is it still better than directly accessing the CSV which is connected through ISCSI with the SAN/WS2012 ISCI target?

Would this set-up be any different when a SAN or WS 2012 server is used as storage?

In the CBT-nuggets video's, they are talking about HA from end-to-end.. But to be honest, I don't see more HA when using this SOFS. When the storage/SAN is down, the whole set-up won't work?!

Thanks for all the suggestions about this set-up. I just want to encorporate all the new features that WS2012 offers, but at this moment I only see a downside of this feature since it brings another extra system to the set-up which (I guess) will reduce performance?

Thanks again!!

Kind regards,

Sven Celis

1) Why you should bother to configure SoFS and go with SMB? That makes sense only if a) you're building a setup from the scratch and you don't have a SAN so can indeed allow Microsoft to eat some of the EMC lunch and b) you have more then 2 Hyper-V servers so implementing switched fabric SAS with a Clustered Storage Spaces would require you to go with a pair of SAS switches and it's both damn expensive (8 Gb FC and 10 GbE are ten times cheaper then 6 Gbps SAS and much more common) and does not really work. For a simple two-node Hyper-V config you don't need SoFS layer absolutely as it just increases complexity (+2 servers, more switches and cabling, + man-hours for config) and implementation costs (+2 servers, switches, NICs, Windows licenses) and slows down everything (obviously feeding SAS directly to hypervisor is faster then to feed the same SAS but using also Ethernet injected in the middle of the I/O pipe).

2) If you want to run a contest of a Microsoft built-in storage tools then the whole thing would go like this:

a) Loser. MS iSCSI target. Extremely dumb implementation (no RAM cache, cannot keep VHDX on a Storage Spaces so no flash cache with R2, cannot scale beyond a single node so multi-node setups are Active/Passive, no built-in clustering featurs so you need to run a standard Windows cluster to make it fault tolerant).

b) Winner. SMB 3.0 share. Solid implementation with some of the drawbacks (SMB is cached on both client and server sides and in a very efficient and aggresive way, can keep files on a Storage Spaces with flash cache, cannot scale beyond a single node so again Active/Passive, no built-in clustering features so again need to be run in a Windows cluster for fault tolerance).

c) Winner but with remarks. Nice implementation being actually wrapped over an SMB share (cached well with RAM and flash, scaled beyond single node with a SMB Multichannel and with R2 should catch up with iSCSI and FC at least on the paper as it should support one-client-to-many-servers load distribution, has built-in features (overwrapped Windows cluster) for fault tolerance). The only issue with SoFS is - it does not support generic workload (so you cannot run a file server) as it can do huge file I/O only. That's why SoFS is not going to replace failover SMB share...

That's for Microsoft. You need to understand however any decent commercial iSCSI target (Windows or FreeBSD/Solaris/Linux based and ZFS powered) would run circles around SoFS config in terms of performance (multiple nodes of course). And any decent 8 Gb FC would kill 10 GbE iSCSI or SMB Direct (RDMA) SMB 3.0 in terms of latency (not even talking about 16 Gb FC). 

3) Why that setup? They wanted to show how SoFS worked so they had built a test lab and demo stand. From the production or performance sides it has zero sense - slow, expensive and having block back end as a single point of failure. So you're absolutely right with your doubts :)

So see again the beginning of my post where SoFS makes sense.

Hope this helped :)

Hi Victor,

First of all a big thanks for your reply!! Good to mention Windows Server 2012 R2, almost forgot the R2 version was already out. :)

I know the advantages of an SoFS, but to be honest, I am still not convinced:

• When you already have a SAN, would you make the switch to just a windows server (with storage spaces) which is configured as an SoFS?
• When you already have the SAN connected to your physical servers through ISCSI. Would you migrate the current servers to Windows Server 2012, implement a high available SoFS server and then place the vhd files on this SoFS instead of directly to the volume of the SAN?

I am really willing to use storage spaces in combination with SoFS, but if you have already invested in a SAN will it improve the performance? Would it be better to move the storage to an SoFS which will be hosting all of your data (instead of directly on the SAN)? In the CBT-nugget series, they configured it like that.

Thanks in advance!

Kind regards,

Sven

1) If you already have a SAN just use it. There's no point in injecting SoFS layer in between the SAN and Hyper-V to do... What? :)

2) Nope, don't do it. Keep your SAN feeding shared storage to your Hyper-V and live happy.

3) SoFS don't host any data (unless they run a Virtual SAN back end like on the link below), in a reference Microsoft design SoFS is just a proxy in front of a physical shared storage back end (SAS, FC or iSCSI). See:

http://www.starwindsoftware.com/sw-configuring-ha-shared-storage-on-scale-out-file-servers

4) You cannot combine Storage Spaces and SAN as SS needs direct access to your SAS spindles. Nothing else is supported. See:

http://blogs.msdn.com/b/clustering/archive/2012/06/02/10314262.aspx

http://blogs.technet.com/b/storageserver/archive/2013/10/19/storage-spaces-jbods-and-failover-clustering-a-recipe-for-cost-effective-highly-available-storage.aspx

Hope this helped :)

It's difficult question to answer on here...

Just to be clear you don't need Storage Spaces to have SoFS. They are two separate features that do complement one another however...

Scale out file server gives you some options which could be out of reach... For example if you had a FC SAN with 2 controllers and put a 2 node SoFS cluster in front you can present the FC storage as a CSV to the hosts which hosts would then be able to access via SMB, if you've got 16 hosts then you don't need to put in 16 dual port FC cards plus associated FC switches... Additionally you could hook your SoFS hosts to several different types of storage and present them as CSVs. So you could hook your SoFS hosts to a range of SANs FC, iSCSI, FCoE, etc. and present all of this storage to hosts as CSVs. If you've got multiple Hyper-V clusters then you can use a single SoFS cluster for that storage, if you want to move a VM between clusters, under traditional storage rules you'd move all the VHDs for the VM too; using SoFS you don't (provided all hosts have access to the same shares) you just move the config... It makes sense when you start to get a larger infrastructure.

The other to think about is the CSV Cache... Under Windows Server 2012 you can use 20% of RAM as a CSV read cache (that can be a big a read cache). Under 2012 R2 you can use 80%, if you've got 96 GB RAM in the server that's a read cache of 76.8 GB... 

Storage Spaces allows you use commodity hardware (provided it's supported) to create pools of storage and then carve these into spaces for storage. In Windows Server 2012 it's OK, especially if you don't have a SAN. When I first saw this feature my first thought was "poor man's SAN" however R2 makes Storage Spaces a real SAN contender but using commodity hardware. As you're studying for MCSE I'd try and keep the R2 stuff out of your head unless you know you'll be sitting exams with R2 content.

I could go on for hours about this stuff. 2012 is the base of a large framework that's only going to get better...

1) Exactly! SoFS acts as a proxy in front of an expensive to expand block storage. Ethernet (even 10 GbE) is cheaper to deploy compared to FC or SAS. iSCSI is another game however...

2) CSV has two major issues: a) it's read-only so does not adsorb writes and that's the most important feature of a file cache and b) it's not distributed so VM moved to another physical host starts always from the "cold" state. So block cache should be moved to some other place (SAN controller, SoFS etc) if possible. 

3) Clustered Storage Spaces is indeed a poor man's SAN. Except the man is not so poor if he buys SAS controllers, spindles, SAS JBODs (and optionally SAS switches). Doing internally mounted SATA (with some flash for cache) + 10 GbE as VMware did with VSAN is a better way to go.

Wow thanks for your answer. I didn't see it like this before. It makes sense indeed when the environment is getting bigger. This is still unclear to me:

• You mention that when you've got multiple hyper-v clusters you need to move the storage also (under traditional rules). But when all of the pyshical servers are configured in the same cluster, they all have access to the same CSV? So no storage move is required?

CSV cache is something new to me, I will certainly review this feature, thx!! Maybe a stupid question, but where did you learn this stuff? I've read some blogposting etc for the new features of Windows Server but have not seen this before.

This was my thought also :) a poor man's SAN.. But correct me if I'm wrong, but isn't this just a software RAID which will not out-perform a hardware raid? So you wouldn't buy any software/hardware RAID controllers, but just a machine with enough SATA/SAS connectors to just hook up the drives and then use storage spaces in the future?

Thanks for the reply on my thread!!

The problem is not amount of SAS ports on the host but amount of ports on a SAS JBOD. With just a pair of them (doubled for A and B) you can connect 4 cables, 2 from each host. Need more hosts? Get a SAS switch (two for redundancy). And that's expensive!  

That's true for all storage networks.

LSI SAS switches are cheap: http://www.amazon.com/SAS6160-16PORT-Switch-Remote-Supply/dp/B0047T7NDC

Check this out to see how it works at scale: http://www.dataonstorage.com/images/PDF/Solutions/NX4K/DataON%20Host%20RAID%20Petabyte%20Storage%20with%20DNS-1660D%204U-60_LSI%206G%20SAS%20Switch%206160_LSI%20MegaRAID%209285-8e.pdf

When you compare cost/GB with Fibre Channel storage it's much cheaper.

In R2 you get the write-back cache (when using tiered storage, i.e. mix of SSD and HDD), which is actually more of a write-through cache. If you enable it - when a write happens it will write the data to the SSD tier and then the inbox optimisation will move data blocks down the HDD tier at the appropriate time.