Hi,

I am looking for some advice on what the best way to configure my disk system is for performance when using Hyper-V. Currently I have two midline 7.2k SATA disks used for the hyper-V OS in RAID 1, and then 14 x 10k 600GB SAS enterprise disks in RAID 10. The RAID 10 is where all my VMs exist on, but not directly on the physical disk, I use a product called StarWind which consumes almost all of the physical disk space, this is a virtual SAN and I use this for my CSV in the cluster, the VMs all exist on this starwind disk, there is a tiny amount of overhead but I dont think thats really where my problem lies. We have 12 Virtual machines, so there is 1 spindle per VM, however its in RAID 10 so does this actually mean I have 7 spindles for 12 VMs?

The entire system almost grinds to a halt whenever I am copying new VMs onto this host. I cant use any other RAID type except for RAID 1 or 10. I know RAID 0 would give me better performance but we are not willing to take the risk. Now, it is my understanding that RAID 1 and 10 are essentially the same with the same performance and capacity, because of this I chose RAID 10 so I can take advantage of the combined capacity, but I am wondering if this is the best configuration.

I can think of a few other variations:

• A single RAID 10 with one large partition but multiple CSVs instead of one large one
• A single RAID 10 with multiple partitions and a CSV on each partition
• Multiple RAID 1, each having a single partition and a single CSV

Reading this site, there are a few pointers I have taken away regarding disk performance, a couple below I have questions about.

http://www.altaro.com/hyper-v/23-best-practices-improve-hyper-v-vm-performance/

Point 6 says Use separate volumes for each VM. Since the administrative tasks of a virtual machine are controlled by its own process (VMWP.exe), keeping several virtual machines on a single volume will cause more Disk I/O operations from each worker process. Hence, it is recommended to use separate volumes for storing virtual machine files (VHD, VHDx, checkpoints, and XML).

Does this mean the arrangement I propose in my second point above multiple partitions and a CSV on each partition? The term Volume seems to be a bit ambiguous depending on where you read

I believe point 14 to be incorrect though. De-fragment Hyper-V Server regularly or before creating a virtual hard disk. It is advisable to defrag Hyper-V Server disks where the VHD and virtual machine configuration files are stored regularly or before creating a large virtual hard disk file.

It is my understanding that defrag only tends to work well if there are many files on the disk, on the physical disk in my case there is one massive starwind file so I dont think defrag on the physical will do anything, defrag in the starwind disk might help but it is mostly large VHDX files will defrag really make much difference?

Additionally, I dont have any generation 2 VMs and theres nothing I can do about this right now, all data disks use SCSI channels on the VMs, all VHDX are fixed size.

Also, a slight side question but kind of related I have both an exchange server and SQL server running as VMs (the log files already exist on separate VHDs, but on the same physical RAID 10 disk), does my physical disks used for log files need to match those used for the DB, or can they be slower lower class disks for example, if I place a 7.2K disk in with a RAID made up of 10K disks, it is my understanding that the entire RAID will lower its performance of the slowest member is this true with log files and databases, if the log files are on slow disks will the database also go slower to match the pace of log files being written?

many thanks

Steve

I recommend using Storage Spaces in the following configuration:

• Use the 14 SAS disks in one pool
• Create 3 spaces, each 1.2TB in size, two-way mirrored, 4 columns, default interleave
• This leaves 2 disks worth of space, equivalent to 2 hot spares

Each read/write operation will write to 8 physical disks, which should be faster than current performance.

Setup one vDisk in each space, one partition, one volume - format NTFS 8KB allocation unit, quick format.

These 3 volumes will appear in Disk Management and can be set as CSV's 

For SQL and other VMs that need more IO, setup VM VHDX disks on different CSVs

hi, thanks for that. I should have mentioned I am using the hyper-v only edition so i dont have the storage spaces feature, however in your example where would my starwind images come into it, i must have this because it performs synchronous synchronisation to the other server - i am assuming "Setup one vDisk in each space, one partition, one volume - format NTFS 8KB allocation unit, quick format." means the vDisk will be my starwind image? going through starwind, and then an additional VHDX vDisk using storage spaces wont add too much overhead right????

also, my current unit allocation size is set to the default, i think this is 4k, should i format both the physical and starwind disk using 8k? what about the VHDX used in the VMs, should these all also be 8k or does this not matter so much?

also, you said each IO will go to 8 physical disks, where do you get the 8 physical disks from? doesn't each storage space use 4 disks? and all spaces will be across 12 disks?

thanks

Steve

• Edited by Steve Mills Sunday, December 07, 2014 1:15 PM

Hi,

[ ... ]

I can think of a few other variations:

• A single RAID 10 with one large partition but multiple CSVs instead of one large one
• A single RAID 10 with multiple partitions and a CSV on each partition
• Multiple RAID 1, each having a single partition and a single CSV

[ ... ]

BOLD

That's the way to go. Make sure you have at least one CSV per physical host. Also from your config (few VMs but very heavy ones like SQL Server and Exchange) you definitely need more IOPS. Think about a) throwing in more spindles b) using flash for either cache or in a combination with Tiering (StarWind will allow you to use SATA here so should not be any cost-related issues).

[ ... ]

Each read/write operation will write to 8 physical disks, which should be faster than current performance.

[ ... ]

BOLD

Unfortunately you need to have very large I/Os for that to happen... Typical VM workload is dominated by small reads and writes (4-8 KB) because of the so-called "I/O Scrabler" effect: you'll virtually never see any particular I/O request being processed by all spindles your RAID group. Run Performance Monitor on host to see what activity you have on your CSV and prove this to be true. See:

Windows Performance Monitor Disk Counters Explained

http://blogs.technet.com/b/askcore/archive/2012/03/16/windows-performance-monitor-disk-counters-explained.aspx

The one you should look @ is called "Avg Disk Bytes / Transfer". See:

"Avg. Disk Bytes/Transfer (Avg. Disk Bytes/Read, Avg. Disk Bytes/Write)  
Displays the average size of the individual disk requests (IO size) in bytes, for the capture interval. Example: If the system had ninety nine IO requests of 8K and one IO request of 2048K, the average will be 28.4K. Calculation = (8k*99) + (1*2048k) / 100 The Avg. Disk Bytes/Read and Avg. Disk Bytes/Write counters break down the results showing the average size for only read requests or only write requests, respectively. "

So aggressive write-back caches and log-structuring help to adsorb writes and for reads you have to have huge read caches (better ones being VM- rather then LUN-aware).

Hope this helped :)

hi, thanks for that. I should have mentioned I am using the hyper-v only edition so i dont have the storage spaces feature, however in your example where would my starwind images come into it, i must have this because it performs synchronous synchronisation to the other server - i am assuming "Setup one vDisk in each space, one partition, one volume - format NTFS 8KB allocation unit, quick format." means the vDisk will be my starwind image? going through starwind, and then an additional VHDX vDisk using storage spaces wont add too much overhead right????

also, my current unit allocation size is set to the default, i think this is 4k, should i format both the physical and starwind disk using 8k? what about the VHDX used in the VMs, should these all also be 8k or does this not matter so much?

also, you said each IO will go to 8 physical disks, where do you get the 8 physical disks from? doesn't each storage space use 4 disks? and all spaces will be across 12 disks?

thanks

Steve

The recommendation I posted does away with Starwind. If you wish to replicate to another server, you can use Hyper-V replication, which replicates the VMs to one or 2 other hosts, and can be setup to maintain several points in time to fail over to. It does require Windows 2012 or 2012 R2 though. 

Otherwise, if you're staying with Starwind, check Anton notes below. He would know more about Starwind..

Thanks,

so I should follow the first point of 1 big RAID 10 and multiple CSV's. next questions:

• how should i group VMs on CSVs
• by average request size?
• 1 CSV per VM?
• a particular amount of IOPS that those VM's produce?
• if my unit allocation size changes to say 8k for the physical disk, do i need to have the starwind image at 8k, AND the VHDX that sits on it? is this what Microsoft refers to as "alignment"?
• would a SATA SSD help keep the performance fast whilst im transferring a really large file to the host? we want to test image based backup restores to this in the future so i really don't want tests affecting production performance.

many thanks

Steve

Steve,

BTW RAID 10 is essentially RAID1+RAID0 so you get mirroring and striping. You reap the RAID 0 benefits when you go to RAID 10. The idea with what these guys are saying is accurate. You want as many spindles as you can get in your configuration.

I would group VM's on your CSV's in a round robin fashion to start. Maybe create two CSV's at 2 TB volumes each and just start saying 1 for you and 1 for you approach to splitting up virtual machines. Make sure your format the volumes as GPT for future growth. Or depending on how many hosts you have you can go off of what VR38DETT has said. Once CSV per physical host and even split up the size of your large RAID 10 amongst them. The thing to take note as well, if you watch the performance once your work load goes live you can live storage migrate your machines between CSV's if needed. 

Alignment is a thing all virtual machines go through. I believe in a Microsoft environment if your using 2008 R2 and later it will handle alignment for you. 2003 does not and Linux/BSD machines need manual alignment as well.

A good free tool for watching performance is http://pal.codeplex.com/ it essentially uses perfrmon to run a bunch of counters and generates a report. The hyper-v one is a good one to look at many variables where your getting lock ups.

I believe the best practice for RAID or the underlying stripe is a 64k size. Then VHDX is going to use a 4K logical sector. The most important thing is if you can multiply your file block size into the raid stripe size. Which both 4k and 8k go into 64k

Hi thanks for the info, my current RAID stripe size is:

Strip Size / Full Stripe Size 256 KiB / 1792 KiB 

is the higher the better or should go down to 64k? my smart array allows me to go from 8k to 1024k, its default was 256k

I'm a bit confused by "Once CSV per physical host and even split up the size of your large RAID 10 amongst them." do you mean keep the large RAID 10 but partition it into smaller chunks, or do you mean breaking the RAID 10 into smaller RAID 10's? I only have 1 server to host the VM's, the other is used for failover (2 node cluster)

many thanks

Steve

Sorry bout that confusing post. You would do One CSV per physical host in a cluster. Since we are talking CSV then we know it will apply to the cluster only. So, take one big RAID 10 and with StarWind create 2 virtual disks to present to the cluster and create two CSV. The CSV's depending on how much space you need could be an even split from the base RAID 10. Lets say your big RAID 10 volume is 4 TB total you would create 2 2TB virtual disks to present to each host in the cluster.

On your Starwind configuration are you using iSCSI? If you are do you have flow control, jumbo frames and MPIO running? I've seen active/active MPIO alone help performance in other storage arrays. That way it allows more threading of TCP sessions.

On the array stripe 256k is fine since 64k will go into 256k. Just for some information the windows storage spaces default is 256k as well. Just for an example on why that will be fine. Typically depending on your workload you could set it large or small. I've seen people recommend for file servers that serve up large files to do the largest stripe size possible. But I've seen people say do 64k for the stripe size for small and medium workloads. I think 256k is a great balance for most workloads.

Hi Steve, how are you? 

Usually when talking about the performance, it would be nice to have some numbers. For example:

• What number of IOPS and MBps your physical storage shows?

• What number of IOPS and MBps StarWind shows? I`m assuming that you are running HA, please let me know if that is not so

• What number of IOPS and MBpsyour network that StarWind uses shows?

Only by knowing those numbers we can now in what direction we should move forward. As usual, I`m gladly offering our assistance in benchmarking - just email us. I can bet you know our email ;) 

• Edited by AnatolySV Monday, December 08, 2014 6:49 PM

Hi, thanks for contributing to my question :)

I dont have those figures to hand at the moment, what caused me to raise the question was that i was looking at the avg. disk sec /read and avg disk sec /write counters, and i noticed that whenever i copied a file onto the starwind disk, the physical disk ms response would raise a bit, and the starwind ms response would raise quite a bit more. part of this may well be attributable to a known bug that you guys are working on, but i really wanted to take the opportunity to ask further and get more info on how to structure my disks in a way which is optimum regardless of any performance there may or may not be with starwind.

i would have to run some further analysis on those figures before i could get a clearer picture i guess, right now im just trying to finish off the virtualisation stuff, and we have just purchased a new cabinet that needs installing, amoungst other things, so i might not get a change to do the proper analysis before christmas.

useful to know what the general opinions are though that might guide me in the right direction.

why is it though that if i have a single RAID 10 array but use multiple CSV's that this could boost performance? surely it is still getting balanced out across all the disks by the controller, so if one CSV is hit with writes it should lower the performance of the others similarly?

Also, yes jumbo frames, flow control, MPIO and iSCSI :)

cheers guys

Steve

The theory is to break up multiple threads that might be smaller IO's to other disks. So, if you had a bunch of 8k writes you might get a bunch of queuing if it had to go to one virtual disk when the underlying disks would be OK. I can see the theory around it but, I believe it would give minimum gain and it can add some headache as you scale up. I Typically do one large CSV in my environment, then only building new CSV's as I need different tiers.

I found this case HP did on the P4500 line of there's. They actually tested multiple CSV's for the different workloads. You actually see it be worse sometimes then having one CSV. When they did 10 CSV's they had minimal gain where IMO not worth the headache for me. Plus I think your small enough where it would be OK for one CSV. StarWind might have some different best practices for there product as well. It's been ages since I played with StarWind. But it was a good product when I did, like 5 years ago.

I think if you could run PAL against your configuration it would yield where your bottleneck is when copying large files.

Hi Steve,

>>but i really wanted to take the opportunity to ask further and get more info on how to structure my disks in a way which is optimum regardless of any performance there may or may not be with starwind.

I prefer to use "A single raid 10" .

As we know 2012 R2 hyper-v provides a I/O limitation feature for VM's disk, if you need to balance/adjust the I/O for VMs , based on this I think you do not need to worry about which wm will exhaust the I/O or split the "single raid 10" to multi disk for CSV .

Best Regards,

Elton Ji

Hi,

I'd like to pick this back up ASAP, we had what I see as a huge problem with StarWind sync yesterday (I've emailed support) where I had installed a HP monitoring tool on one of the servers, rebooted and when it tried to re-sync I saw incredibly long sync times of over 3 days for a 3.5TB HA image as well as bringing the system to an unusable state during that time.

In order to get the sync down to a reasonable length of time (still 6 hours, even though when I had an empty system when testing the starwind sync alone I managed to get it down to 1hr 30min - 1hr 50min) I had to shut down about half of my VM's. I have turned up to the office this morning and the Sync had completed overnight.

I have no idea if this is a starwind issue or my disk config, or a lack of spindles or what at this stage, so what I am asking for in this post is a list of all the counters I need to use to gather all the data i need. I want to run these counters in a data collector set for maybe an hour during normal operational use, and then again when I am performing a sync and hopefully this will get me some good data on what's going on. I am thinking of running the counters against the underlying physical disk and against the starwind HA image (I assume this would be correct?)

what precisely do I need to measure, because there are lots of counters to choose from. Also where do i run these counters, on the host or in the VM guests or both? can you provide a list please.

I also have several csv files containing the output of iometer using all the access specifications when the system had absolutely nothing on it at all if this is useful.

many thanks

Steve

Hey Steve,

For findings like this I like to utilize the PAL diagnostic tool found on codeplex.

Once installed go to your start menu/screen and type "PAL" if UAC is on you will want to run it as administrator.

Once the application is opened go to the "Threshold file" tab. Then select a threshold file title, or storage servers you can choose "Quick System Overview." For the Hyper-v ones you can choose those specific ones. Once you have your threshold file selected click export to Perfmon template file. It will allow you to save an XML file.

Now go into perfmon either on the server or remotely from your perfmon on your machine connecting to the servers.

Go to Data Collector Sets > User Defined and right click it. Select New Data collector set. Give it a name you can identify and select the radio button create from a template. On the next screen click on Browse and select your exported xml. You can next through if you'd like or you can click on finish. On the created data collector you can create a variable in the properties to stop it after it has ran for an hour. Or you can manually do it. Once your ready to collect data press play or right click start on your collector set.

After it has ran navigate to the collectors save location that was setup during the creation of it. Or you can look at the properties and see where this is. Grab the cab file or the .BLG file that is left and put it on your machine where PAL is installed. 

Open PAL again and go to "Counter Log" and select the BLG file that was created by the collector. Press next and now you should be on "Threshold File" and select the template you used for the perfmon collector. Press Next and answer the questions its asking about the collector. Press Next and go through all other tabs selecting optional options youd like. Once on the execute tab press finish.

It will run a bit possibly as it chews through the log. The output will be a neat html report with best practices and interesting info that you can match up with other collectors in an easy to read format.

Hi! 

Just want to give you the quick heads up: right now we think that the issue is related to the SW cache, and we are prepearing the correct instructions for you to switch it off without downtime. Hopefuly that will help. 

Thanks for that, I have just captured a few minutes to see what kind of data comes back, it's extremely comprehensive. I assume that if i want to run other analysis like SQL and Exchange, i would actually run these data collector sets from within the virtual machines even though they are on the physical host - the counters would be correct right? or is gathering counters inside a VM a pointless exercise?

I will run this for an hour when i get back from the christmas break. 

thanks

Steve

Correct, you will want to run it in the VM where the workload runs. Essentially its just loading the performance monitor counters for the instances that are installed. If it is not installed and you loaded it in, then it would return counter not found. For SQL and Exchange and even backups with commvault it has helped me greatly in troubleshooting issues. 

So, your generic quick system overview and system overview and a few others work on pretty much any thing that is 2008 and later. If you go into edit instead of export you can see what perfmon counters its using, also add your own and build it into the report if you wanted. They tried there best to align it with technet articles for thresholds as well. Sometimes not perfect but its just great to visualize everything.

Since you have starwind on a physical server you could run it there I picked the quick system overview. Then on your hyper-v hosts you could use the Hyper-V counter.

Cheers and Merry Christmas!

1. +1 to last Jugganutz1871 post

2. Update for everyone: right now we removed caching from SW HA by modifying its header files a little bit. As the result Fast Synchronization is the only one that actually kicks in, so it takes seconds, minutes maximum. That should do the job. 

Hi,

Just to update this, I have ran the PAL for an hour and got quite a few warnings and errors. I got several relating to RAM and even CPU which is surprising since it has 2 x 12 cores Intel Xeon! I dont really know what can be safely ignored and what needs action on it, I guess it justifies its own thread.

Anyway, some of the alerts relating to disk are as follows - what do I need to do to correct these?

The above are common throughout each alert time range (there are 30 time ranges, each being 2 minutes), I've numbered each alert for ease of reference in your response.

D drive is the physical disk drive, its low on space because the starwind image takes up the majority should this be a problem? I wouldnt have thought so, since the starwind image is capped at its size and thats all that ever goes on the disk. I still have 335GB free

Physical disk 2 is the starwind image disk

HarddiskVolume1 looks like its Windows system partition

My C drive is a pair of 500GB 7200 RPM SATA midline disks in RAID 1, I thought this would be suitable since the host OS is only hyper-v, nothing else.

From what I can tell, the disk I/Os Queue length and the greater than 25ms response are the most important ones here relating to starwind disk.

Of course, I have all the nice graphs with the raw data as well if someone needs to take a look.

I didnt know what to post, so I only posted the above but I also have the data under the following areas:

Hyper-V Virtual Storage Device

• Hyper-V Virtual Storage Device Read Bytes/second (Stats only)
• Hyper-V Virtual Storage Device Write Bytes/Second (Stats only)
• Hyper-V Virtual Storage Device Error Count (Alerts: 0|0)
• Hyper-V Virtual Storage Device Error Count (Stats only)
• Hyper-V Virtual Storage Device Read Bytes/sec (Stats only)
• Hyper-V Virtual Storage Device Write Bytes/sec (Stats only)

LogicalDisk

• LogicalDisk Disk Transfers/sec (Stats only)
• LogicalDisk % Free Space (Alerts: 0|30)
• LogicalDisk Avg. Disk sec/Read (Alerts: 0|2)
• LogicalDisk Avg. Disk sec/Write (Alerts: 0|0)
• LogicalDisk % Idle Time (Alerts: 0|0)
• LogicalDisk Read/Write Ratio (Stats only)
• LogicalDisk Bytes/Read (Alerts: 0|30)
• LogicalDisk Bytes/Write (Alerts: 0|0)
• LogicalDisk Avg. Disk Queue Length (Alerts: 0|0)
• LogicalDisk Current Disk Queue Length (Alerts: 0|0)
• LogicalDisk Disk Bytes/sec (Stats only)
• LogicalDisk Free Megabytes (Alerts: 30| 0)
• LogicalDisk Disk Overwhelmed (Alerts: 0|0)

PhysicalDisk

• PhysicalDisk Disk Transfers/sec (Stats only)
• PhysicalDisk Read Latency Analysis (Alerts: 0|2)
• PhysicalDisk Write Latency Analysis (Alerts: 8|5)
• PhysicalDisk Current Disk Queue Length (Alerts: 0|1)
• PhysicalDisk Avg. Disk Queue Length (Alerts: 0|17)
• PhysicalDisk Disk Bytes/sec (Stats only)

Once again, thanks for any help.

Steve

If you can zip up the html report that was generated and put it in a onedrive, google drive, dropbox or something like that to share via http. Also this was of the StarWind server and not the hyper-v hosts where your workload will be running correct?

Some of those alerts are not concerning and the pair of 7200k SATA disks for the hyper-v host should be just fine. Some of the values look crazy by themselves but when matched with other workloads they really are not. I'll eyeball it for you and tell you if I see anything.

oh thanks that's great. I will try to zip it and share it by the end of the week. the starwind and hyper-v server are both installed on the same bare metal. 

thanks

Steve

Hi,

You should be able to get the html report from here PAL Report

thanks

Steve

Hey Steve,

Thanks supplying the report. Can you copy the accompanying PAL folder with the same name as the html file? This will make it so we have the graphs for your PAL report. Also, was this report ran when you did a test copy?

Next thing I am seeing is it looks like your Virtual Machines are getting some pressure and starving for memory. This is most likely making them page to disk, bumping up writes and causing unneeded IO. The Read disk counters look great. But the queue length on writes and the write performance looks like you have issues for just about the whole hour. This could be related to the paging of memory as well. I wonder, on the CS-EXCH01, APP2-SERV Contoso-SERVER or WatchgaurdDimension, do any of those appear to grind to a halt as well? I'm assuming they use static memory values and that is why we don't see any dynamic memory pressure counters.

The last thing is, what model raid card is in the server?

• Edited by Jugganutz187 Monday, January 12, 2015 3:55 PM

Hi,

during the analysis I did not perform a test file copy, i wanted to see how the counters looked like without adding any extra strain.

Exch01, watchguard, and Contoso-SERV is static RAM, but app2-serv is a terminal server with dynamic RAM

on all VM's which use dynamic RAM i have made sure that the memory demand shown in hyper-v management console is less than the maximum RAM by a few GB. for example, right now (and this is usual figures) the demand on app-serv2 is 8.2GB but the maximum is 12GB, assigned is 9.6GB. Why would this cause paging to disk, i thought that would only occur if the demand is higher than the maximum you allow? it probably does need tweaking a little bit but to understand this first will help.

the RAID card is Smart Array P420i.

can you tell from the figures if my unit allocation size needs to be adjusted on either the physical or starwind disk? does it give any indication as to whether multiple CSV's on the same underlying RAID 10 would help?

thanks

Steve

Thanks Steve, that raid card is mixed on queue depth when I'm looking for info on it. looks like it should be 1024, which should be more than enough for what your doing.

To the paging, it could be a false positive. You might have to look into your machines to verify. Which is sounds like you have. On the PAL report i was looking at the VM Average Memory pressure. Some machines hit went over 100% which is going into the negatives for memory. The balancer on the host also went up to 89% but overall is over 80% always. That should be the system wide value I believe. I say it's a false positive possibly because you can view anything over 80% possibly as having an issue is how technet talks about it. If you feel confident it's not then things should be good and you can ignore. Also, systems page sometimes for non memory issues. But, you are correct in believing in 100% memory usage means you will start paging.

I think one CSV should be sufficient for the configuration. I don't think it would net you much at all hoestly to break it up. After all you are utilizing the same spindles and same hardware no matter what. As for allocation is concerned i think its fine where you have it. It is one static file that i'm sure was created sequentially and doesn't require defragmenting. It should perform OK, especially since I'm sure your CSV volume is no where near full.

I think you have a good hour long baseline. Now you need to test what happens when a copy takes place to see your results.

Ok, I'm going to test it when doing a file transfer, hopefully later today but it might only be for 15 minutes because users will start complaining.

as for the queue depth, i can modify my controller settings (currently automatic) but it only goes up to 64. what does this queue depth actually mean, what is the affect of a larger or smaller amount?

the controller also has 512MB of cache, but there is currently 304MB available.

Also the default for this controller is to disable physical drive write cache

thanks

Steve

Hi

I have performed another performance recording, you can find the report here the file is called PAL2.zip

for this, i recorded for about 4-5 minutes before copying a 50GB file to the physical D drive, the collector ran for 15 minutes. I haven't looked at these results myself yet but I am not really sure what exactly I'd be looking for. I haven't made any adjustments to dynamic memory or anything else since my last capture, this is purely just adding a file transfer to the load. my client machine was sending at roughly 60MBps, its an SSD which is capable of around 300MBps and I'm sending through a 1Gbps connection, i should have been getting around 100-120MBps sending out, so i can only assume the receiving disk system wasn't able to handle the receive at any more than approx 60MBps (it also has a 1Gbps connection).

Hope that helps, thanks for your continued attention on this

Steve

Hey Steve,

I went through the report, the only thing really stand out that I can see is warnings for virtual hard disk errors. I've never really seen that happen. I did some research and there really isn't any clear cut why? It does seem that people get it and it doesn't present any issues or slowness like you see. I also see your dynamic memory is now in check and nothing is over 100%. I see your network adapters doing about 50% like you were saying. I don't see anything to crazy from this report. There is a possibility you could be hitting a max IO threshold with the workload you are running on this system as well. I see this in the report. Granted it might not be 100% disk IO. But depending on the workloads going on a disk rated for 200 IOPS at 4k in testing might be 100% of read or write, not a breakdown of 60%-70% read to write ratio. Also the latency usually goes up. To battle this adding more spindles or flash caching could help.

This leads me into the next question. When you spun up this environment did you do any local disk benchmarks? To test in theory what you should be getting outside of the network to rule it out, but also to help with capacity planning from a Disk IO/Latency perspective? Diskspd is good for this, check this blog for a good way to use it.

For your raid card and queue depth, leave it to Auto. From what I can tell you shouldn't be hitting that barrier. Essentially its how many outstanding IO's it will handle to be processed. Some software defined storage implementations choke with cards that have small queue depths. Example look at VSAN and h310 dell perc cards. It's just because the amount of operations it has to do to keep all the bits sync's and distributed for example. One downside to queue depth can be an increase in latency. Though very minimal in the grand-scheme of things.  In the report if you look at queue length that number the operating system is doing can translate into queue depth being the bottle neck. Your queue length looks OK enough to where it wouldn't cause a pause.By chance do you have a battery for the raid card? If not you should get one and enable write caching. That could help boost performance.

Have you connected with StarWind support to make sure your StarWind is meeting there best practices as well? Maybe ask if they have any requirements around raid cards as well.

Hi,

So from what you're saying then in your first paragraph, is it doesn't appear to need adjusting on the disks. maybe the slowness just happens when there is a little extra work going on and i happened to be copying large VHDs to the server (this is a new system i was moving/converting machines onto it whilst running anything already on there). Like you said... maybe I am just on the cusp of hitting the threshold with these disks, when starwind sort out their cache problems i certainly was intending to buy an SSD for L2 cache, and possibly move our biggest line of business VM onto 15k spindles. One of my biggest reasons for bringing this thread to light was that I experienced some slow performance without having any host level backup solution, i was extremely concerned that performing backups every hour would end up grinding the system to a halt, we do run backups inside the VMs but only about 4 or 5 of them.

in terms of benchmarking i did run a comprehensive set of tests using IOMeter before any workloads were placed on it, I did this against the physical disks and the starwind disks, and then from inside one of the VM's after i moved it on as a test VM. I dont claim to understand all the results but I'm happy to share the full spreadsheet if it would help.

 I have 1 battery for the RAID card. the default settings for this are cache ratio 25% Read/75% write

enable write cache when battery not present - Disabled, so I assume therefore write cache is already enabled when the battery is present.

I have been in regular contact with Starwind, they don't appear to be saying that anything is misconfigured and i've had plenty of remote sessions to see how it's all configured. they have been very supportive, but if Starwind are still looking at this thread maybe they could take a look at the further input on this thread and the resources on my onedrive account and just double check things?

let me know if you want the IOMeter spreadsheet, ill dig it out and upload.

many thanks

Steve

Awesome! Please do upload the IO meter spread sheets you have. This will possibly help give us some numbers.

The one thing I forgot to mention, how is read performance? Meaning the large file you copy that makes it come to a crawl on writes, does it do the same on reads? Can you possibly PAL a 15 minute report for reads as well and see if things come to a crawl? I wonder if it just happens on writes only and not reads.

Agree on the raid card, it sounds like read caching is enabled. That is good :-)

Starwind support is great! I wouldn't disagree at all with you on that. When the L2 cache gets sorted it will be a big boost for sure.

As for the 15k drive idea, I might just hold off on that idea or deploy SSD. 15k disks give you some IOPS for sure but they do not really help so much in the latency space. The latency is slightly better than 10k so your gain might not be anything massive. You will still need many spindles possibly for the larger systems.

This graph is from 2012, I know drives are slgihtly better now when using SAS based ones. Mileage may vary of course.

It's great that you're on the same page as me and confirming what I have already read from other sources that 15k wont give me anything massive. I was going to buy these disks in 15k originally but at the time they weren't available so I settled with 10k thinking these would be fine. I am holding off as long as I can to buy anything more just in case there is a config error. I am of the firm belief that you can't just throw SSD at everything and assume all is well, this wouldn't fix (or make my understanding any clearer) an underlying config mal-practice.

if it genuinely needs SSD either for the workload or the L2 cache then fine, i'll get them, but I want to make sure everything else has been exhausted first. on that note, I shall upload the IOPS file I got from IOMeter, I performed all of the included tests so there is quite a lot of data, but I think I only ran each one for about 2 minutes to see what performance I got (it was an empty system at the time, so I didn't think longer tests would prove anything), I will try my best to do this Monday.

thanks once again for helping, it's great to increase my knowledge in an area that is slightly lacking, it's thanks to people like you and starwinds that I get smarter with this :)

have a nice weekend.

Steve

I agree with you. SSD shouldn't be the answer to everything. It is good your doing deep investigations into the why and if everything is performing the best it can side of things.

You are very welcome on the help! We have all been in those scenarios where we need it. Without others helping the industry couldn't move forward.

I look forward to seeing your results and things when you get them. Have a great weekend!

Hi,

Sorry this has taken so long to respond, we had a bit of a disaster which we are still recovering from. I found the spreadsheet with all the data in it, each sheet is a separate test. we tested with 1 test VM running at the time as well, we did both starwind (SW) disks and physical disk (PD), we also did tests with the VM running directly on physical disk to compare against when they ran on top of the starwind image.

the spreadsheet can be found Here

thanks

Steve

Stay tuned while I review these results.

Hi,

thanks for looking at this for me. I really appreciate it. hope the data makes sense and gives some indication of what I might need to improve things... if anything :)

thanks

Steve

Sorry for the delay Steve,

From what I can tell the response times are terrible for the EFCAS/1 VM. With response times that high your going to get a non-response because those times are out of hand. I'm assuming when you call out SW disk and PD disk your doing those tests directly on the host? Then on the EFCAS VM when you say test physical disk E that means your actually running the vhdx on the physical disk outside of the SW layer? The strange thing is while the response times are high the IOPS are just the same as the other scores with tolerable response times.

Am I thinking off base with those results?

Your other numbers look on par for your disk type and raid type IMO.

Hi

these two profiles

EFACS-TEST E drive on PD
EFACS-TEST E drive on SW

EFACS-SERVER E drive (this one is where the VHDX is mounted directly on the host, the VHDX file existed on the physical disks because I don't think I could mount it when it existed on the SW image, due to the SW image being a CSV and part of the cluster disks)

means the EFACS virtual machine was running and I tested the IO from inside the VM (the VM has multiple VHDs attached, this was the E drive which is used for SQL DB files). On PD is the results from the physical disk and SW is from when that E drive was situated on the SW image.

I should have labelled the tests more clearly but didn't think at the time. By the way when I use the QOS 50 IOP.... I knew this would be extremely long response times, this is using QOS on the VHDX file on the VM settings page, I just wanted to test the impact of using a low IO number using QOS, QOS isn't currently implemented for our real workloads

you say about the response times being high and strange that the IO are the same as other scores..... what are you thinking? is that when used on the SW layer or the PD?

thanks

• Edited by Steve Mills Saturday, February 07, 2015 1:39 PM

That could be why the latency is high if the vhdx is mounted directly on the host while being on a csv. interesting... I would say anytime you see a response time of 6ms your going to really feel that impact... Hmmm, can you go down the latest version of PAL and install it? This time use the "quick system overview" thresholds. Re-do a copy and post the results. Also, maybe run it in your other guest machines to see how they react to the copy.

When you put in the onedrive can you also zip up the images of the graphs?

I will try and get round to this sometime this week, I am pretty sure I know where the issue is though now, I have recently bought a couple of SSD's and the performance when using the Starwind CSV really is very poor (under 40MBps) yet when I do any normal file copy the SSD copes with over 400MBps.

I don't know how accurate task manager is for the ms response time, each second gives me a different figure, one refresh it could be showing like 1900ms the very next refresh could be 200ms, the next back up the next could be 45ms....probably not an accurate way to gauge the response times but I just though it interesting that they are so high sometimes, I am not convinced that a user base of around 50-75 is putting such a massive strain on a SAS 14 disk RAID 10 at 10k RPM. I could try moving all the VM's onto the physical disk and seeing how that reacts but then I would lose my CSV and synchronisation for a while. As it stands I have given the latest log files to Starwind support and i'm hoping their going to come back with saying it's a bug in their software because otherwise i'm really lost with this.

thanks

Steve