I am converting a small office from a physical server infrastructure to a virtual infrastructure using Hyper-V and iSCSI storage. I am using Hyper-V Server 2012 R2 (no GUI) on a Dell PE R430 with a 5-bay Synology. Using 2 x 1Gbps NICs for iSCSI with MPIO. I appear to have good iSCSI sessions established, a VM that is running on the LUN that can transfer files quickly from the host OS drives to the VMDK on the LUN, BUT my issues is the disk latency on the VM is horrendous. Within the VM the disk time is always at 100% and the response time is on average 600ms to 3000ms. I feel lost without my GUI right now, but any help would be greatly appreciated.

 

Host: Dell PowerEdge R430 (1 x Xeon 6-core CPU, 32GB RAM, 6 x 1Gbps NICS, and RAID1 with 2 x 120GB Intel SSDs)

Storage: Synology DS1515+ with SHR (5 x 2TB WD Red) and 2 NICs for LAN and 2 NICs for iSCSI

 

To get them started I setup a new Dell PowerEdge R430 and installed Microsoft Hyper-V Server 2012 R2. Typically I have installed Microsoft Windows Server 2012 R2 with Hyper-V installed so I can have the GUI as needed. However, this time I thought I would go with the free core version to save on software costs and I didn't have any trouble configuring this in labs from time to time. I have a Synology DS1515+ with a 5-drive SHR and a 7TB LUN for VM files and a 1GB LUN for a Witness disk as I plan to make this into a Failover Cluster with a  CSV.

 

On the host I used the  New-NetLbfoTeam cmdlet to setup a 2 NIC Team for Mgmt LAN, a 2 NIC Team for Hyper-V LAN vSwitch, a 2 separate NICs for iSCSI traffic. I use netsh commands and Get-NetAdapter cmdlets to set the IP addresses and made sure there were not any MAC address conflicts.

 

I used Keith Mayer's guide (http://blogs.technet.com/b/keithmayer/archive/2013/03/12/speaking-iscsi-with-windows-server-2012-and-hyper-v.aspx) to setup the iSCSI connection and sessions to my iSCSI Target on my Synology. I also setup MPIO. I created a new VM running Windows 8.1 for management.

 

I noticed that the VM wasn't performing well. I found the disk time was nearly always at 100% and the Disk response times were extremely bad ranging from 60ms to 50000ms. I would say the consistent good was around 600ms and the spikes were on average in the 3000ms range. Absolutely awful. I have more background in VMware and so I don't know really where to begin. It can transfer large files from the host local disk to the VM disk around 80 MB/s, but the experience shows the latency.

 

Two more examples. If I start the VM and open Task Manager and go to the Performance tab and select Disk it will show the response time (Also look in Resource Manager) and it will look okay <20ms, but open Chrome or Resource Monitor and spikes to >3000ms. Also, the System and Explorer processes show an average of 300-500ms response time at essentially idle. I think you get the idea.

 

I have installed the latest DSM on the Synology and I have installed missing updates to the OS on the host. I tried removing the iSCSI target connections and adding them back on the host, but that didn't help either. Restarted the host, storage, and VM multiple times in the process of troubleshooting. About to phone Microsoft.

 

Including some random outputs from PowerShell on the iSCSI and network stuff.

PS Info:
PS C:\Users\ben> get-iscsiconnection


ConnectionIdentifier : ffffe00167f5a020-0
InitiatorAddress     : 0.0.0.0
InitiatorPortNumber  : 1984
TargetAddress        : 10.1.1.51
TargetPortNumber     : 3260
PSComputerName       :



PS C:\Users\ben> Get-IscsiSession


AuthenticationType      : NONE
InitiatorInstanceName   : ROOT\ISCSIPRT\0000_0
InitiatorNodeAddress    : iqn.1991-05.com.microsoft:client-hv01
InitiatorPortalAddress  : 0.0.0.0
InitiatorSideIdentifier : 400001370000
IsConnected             : True
IsDataDigest            : False
IsDiscovered            : False
IsHeaderDigest          : False
IsPersistent            : True
NumberOfConnections     : 1
SessionIdentifier       : ffffe00167f5a020-4000013700000001
TargetNodeAddress       : iqn.2000-01.com.synology:client-ds01.target-1.644bfbc96d
TargetSideIdentifier    : 0200
PSComputerName          :



PS C:\Users\ben> Get-IscsiTargetPortal


InitiatorInstanceName  : ROOT\ISCSIPRT\0000_0
InitiatorPortalAddress : 10.1.1.11
IsDataDigest           : False
IsHeaderDigest         : False
TargetPortalAddress    : 10.1.1.51
TargetPortalPortNumber : 3260
PSComputerName         :

InitiatorInstanceName  : ROOT\ISCSIPRT\0000_0
InitiatorPortalAddress : 10.1.1.12
IsDataDigest           : False
IsHeaderDigest         : False
TargetPortalAddress    : 10.1.1.51
TargetPortalPortNumber : 3260
PSComputerName         :



PS C:\Users\ben> Get-Disk | Where-Object BusType -eq "iSCSI"

Number Friendly Name                            OperationalStatus                                                 Total Size Partition Style
------ -------------                            -----------------                                                 ---------- ---------------
1      SYNOLOGY iSCSI Storage                   Online                                                                  4 TB GPT
2      SYNOLOGY iSCSI Storage                   Online                                                                  1 GB GPT


PS C:\Users\ben> get-partition


   Disk Number: 0

PartitionNumber  DriveLetter Offset                                                              Size Type
---------------  ----------- ------                                                              ---- ----
1                            1048576                                                           350 MB IFS
2                C           368050176                                                      110.91 GB IFS


   Disk Number: 1

PartitionNumber  DriveLetter Offset                                                              Size Type
---------------  ----------- ------                                                              ---- ----
1                            17408                                                             128 MB Reserved
2                E           135266304                                                           4 TB Basic


   Disk Number: 2

PartitionNumber  DriveLetter Offset                                                              Size Type
---------------  ----------- ------                                                              ---- ----
1                            17408                                                              32 MB Reserved
2                F           33619968                                                       991.88 MB Basic


PS C:\Users\ben> ping 10.1.1.51

Pinging 10.1.1.51 with 32 bytes of data:
Reply from 10.1.1.51: bytes=32 time=1ms TTL=64
Reply from 10.1.1.51: bytes=32 time<1ms TTL=64
Reply from 10.1.1.51: bytes=32 time<1ms TTL=64
Reply from 10.1.1.51: bytes=32 time<1ms TTL=64

Ping statistics for 10.1.1.51:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms
PS C:\Users\ben>

PS C:\Users\ben> get-netadapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
StorageNIC4               Broadcom NetXtreme Gigabit Ethernet #4       17 Up           44-A8-42-39-3C-B1         1 Gbps
StorageNIC3               Broadcom NetXtreme Gigabit Ethernet #3       16 Up           44-A8-42-39-3C-B0         1 Gbps
Ethernet 3                Realtek PCIe GBE Family Controller #2        18 Up           00-13-3B-10-49-87         1 Gbps
NIC2                      Broadcom NetXtreme Gigabit Ethernet #2       15 Up           44-A8-42-39-3C-AF         1 Gbps
NIC1                      Broadcom NetXtreme Gigabit Ethernet          13 Up           44-A8-42-39-3C-AE         1 Gbps
vEthernet (Mgmt LAN)      Hyper-V Virtual Ethernet Adapter #4          31 Up           44-A8-42-39-3C-04        10 Gbps
vEthernet (Hyper-V LAN)   Hyper-V Virtual Ethernet Adapter #2          29 Up           00-13-3B-10-49-02        10 Gbps
LAN                       Microsoft Network Adapter Multiple...#2      25 Up           44-A8-42-39-3C-02         2 Gbps
Hyper-V                   Microsoft Network Adapter Multiple...#3      27 Up           00-13-3B-10-49-03         2 Gbps
Ethernet 2                Realtek PCIe GBE Family Controller           14 Up           00-13-3B-10-49-88         1 Gbps


PS C:\Users\ben> netsh interface show interface

Admin State    State          Type             Interface Name
-------------------------------------------------------------------------
Enabled        Connected      Dedicated        Ethernet 2
Enabled        Connected      Dedicated        Hyper-V
Enabled        Connected      Dedicated        LAN
Enabled        Connected      Dedicated        vEthernet (Hyper-V LAN)
Enabled        Connected      Dedicated        vEthernet (Mgmt LAN)
Enabled        Connected      Dedicated        NIC1
Enabled        Connected      Dedicated        NIC2
Enabled        Connected      Dedicated        Ethernet 3
Enabled        Connected      Dedicated        StorageNIC3
Enabled        Connected      Dedicated        StorageNIC4

PS C:\Users\ben>

From the way I read your post, it sounds like you have both iSCSI IP addresses on the same subnet - 10.1.1.x.  That is not the recommended way to configure iSCSI.  The only time I have seen where that is acceptable is with a Dell storage platform that actually requires it.  Otherwise, Windows tends to not like that sort of configuration.

RE: "I thought I would go with the free core version to save on software costs " - that is only true if you are not running Windows Server VMs.  Operating systems running as VMs under Hyper-V Server have to be properly licensed.  So if you are running Windows Server VMs, the cost will be exactly the same.

Hey Ben,

Can you run diskspd and post the results?

https://gallery.technet.microsoft.com/DiskSpd-a-robust-storage-6cd2f223

Hello Tim, thanks for the response. I changed StorageNIC4 to 10.2.1.11 on the host and I changed NIC4 on the storage to 10.2.1.51. I updated the host iSCIS configuration to reflect this. I am restarting now and will try the DiskSpd suggestion below as well, and then I will update the post to see if this helps.

My understanding of Microsoft licensing for Windows Server Standard is that it licenses a physical host OR 2 virtual instances. So, you pay to license your host if you want to run full Windows Server with Hyper-V. Alternatively, I am looking to see if I can reliably deploy Hyper-V Server which is free for the host, and then only pay for Windows Server for the VMs. I know with Windows Server Enterprise you can now buy that it licenses you host and all the Windows Server VMs it will run, but that isn't in the budget for the size clients I am trying this solution for. Am I misunderstanding the licensing?

Thanks again for your feedback!

Hello Jeff, thanks for the feedback. I am going to run DiskSpd on the VM and I will post the results.

I made the iSCSI IP separation recommended:

PS C:\Users\ben> Get-IscsiTargetPortal

 

 

InitiatorInstanceName  : ROOT\ISCSIPRT\0000_0

InitiatorPortalAddress : 10.1.1.11

IsDataDigest           : False

IsHeaderDigest         : False

TargetPortalAddress    : 10.1.1.51

TargetPortalPortNumber : 3260

PSComputerName         :

 

InitiatorInstanceName  : ROOT\ISCSIPRT\0000_0

InitiatorPortalAddress : 10.2.1.11

IsDataDigest           : False

IsHeaderDigest         : False

TargetPortalAddress    : 12.2.1.51

TargetPortalPortNumber : 3260

PSComputerName         :

The VM seems much more responsive now. I ran some DiskSpd tests and  from what I can tell it looks acceptable. 

diskspd.exe -c1G -d10 -r -w0 -t8 -o8 -b8K -h -L IO4.dat

 

Input parameters:

 

        timespan:   1

        -------------

        duration: 10s

        warm up time: 5s

        cool down time: 0s

        measuring latency

        random seed: 0

        path: 'IO4.dat'

                think time: 0ms

                burst size: 0

                software and hardware write cache disabled

                performing read test

                block size: 8192

                using random I/O (alignment: 8192)

                number of outstanding I/O operations: 8

                thread stride size: 0

                threads per file: 8

                using I/O Completion Ports

                IO priority: normal

 

 

 

Results for timespan 1:

*******************************************************************************

 

actual test time:       10.00s

thread count:           8

proc count:             2

 

CPU |  Usage |  User  |  Kernel |  Idle

-------------------------------------------

   0|  94.69%|  23.75%|   70.94%|   5.31%

   1|  85.47%|  27.19%|   58.28%|  14.53%

-------------------------------------------

avg.|  90.08%|  25.47%|   64.61%|   9.92%

 

Total IO

thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file

-----------------------------------------------------------------------------------------------------

     0 |      1321885696 |       161363 |     126.07 |   16136.35 |    0.494 |     0.609 | IO4.dat (1024MB)

     1 |      1480220672 |       180691 |     141.17 |   18069.15 |    0.442 |     0.625 | IO4.dat (1024MB)

     2 |      1316806656 |       160743 |     125.58 |   16074.35 |    0.496 |     0.605 | IO4.dat (1024MB)

     3 |      1486848000 |       181500 |     141.80 |   18150.05 |    0.440 |     0.600 | IO4.dat (1024MB)

     4 |      1325785088 |       161839 |     126.44 |   16183.95 |    0.493 |     0.599 | IO4.dat (1024MB)

     5 |      1477197824 |       180322 |     140.88 |   18032.25 |    0.443 |     0.640 | IO4.dat (1024MB)

     6 |      1326432256 |       161918 |     126.50 |   16191.85 |    0.493 |     0.594 | IO4.dat (1024MB)

     7 |      1483644928 |       181109 |     141.49 |   18110.95 |    0.441 |     0.608 | IO4.dat (1024MB)

-----------------------------------------------------------------------------------------------------

total:       11218821120 |      1369485 |    1069.91 |  136948.89 |    0.466 |     0.611

 

Read IO

thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file

-----------------------------------------------------------------------------------------------------

     0 |      1321885696 |       161363 |     126.07 |   16136.35 |    0.494 |     0.609 | IO4.dat (1024MB)

     1 |      1480220672 |       180691 |     141.17 |   18069.15 |    0.442 |     0.625 | IO4.dat (1024MB)

     2 |      1316806656 |       160743 |     125.58 |   16074.35 |    0.496 |     0.605 | IO4.dat (1024MB)

     3 |      1486848000 |       181500 |     141.80 |   18150.05 |    0.440 |     0.600 | IO4.dat (1024MB)

     4 |      1325785088 |       161839 |     126.44 |   16183.95 |    0.493 |     0.599 | IO4.dat (1024MB)

     5 |      1477197824 |       180322 |     140.88 |   18032.25 |    0.443 |     0.640 | IO4.dat (1024MB)

     6 |      1326432256 |       161918 |     126.50 |   16191.85 |    0.493 |     0.594 | IO4.dat (1024MB)

     7 |      1483644928 |       181109 |     141.49 |   18110.95 |    0.441 |     0.608 | IO4.dat (1024MB)

-----------------------------------------------------------------------------------------------------

total:       11218821120 |      1369485 |    1069.91 |  136948.89 |    0.466 |     0.611

 

Write IO

thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file

-----------------------------------------------------------------------------------------------------

     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | IO4.dat (1024MB)

     1 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | IO4.dat (1024MB)

     2 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | IO4.dat (1024MB)

     3 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | IO4.dat (1024MB)

     4 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | IO4.dat (1024MB)

     5 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | IO4.dat (1024MB)

     6 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | IO4.dat (1024MB)

     7 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | IO4.dat (1024MB)

-----------------------------------------------------------------------------------------------------

total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A

 

 

  %-ile |  Read (ms) | Write (ms) | Total (ms)

----------------------------------------------

    min |      0.014 |        N/A |      0.014

   25th |      0.293 |        N/A |      0.293

   50th |      0.380 |        N/A |      0.380

   75th |      0.483 |        N/A |      0.483

   90th |      0.651 |        N/A |      0.651

   95th |      0.946 |        N/A |      0.946

   99th |      2.228 |        N/A |      2.228

3-nines |      9.429 |        N/A |      9.429

4-nines |     17.590 |        N/A |     17.590

5-nines |     27.031 |        N/A |     27.031

6-nines |     31.255 |        N/A |     31.255

7-nines |     33.677 |        N/A |     33.677

8-nines |     33.677 |        N/A |     33.677

    max |     33.677 |        N/A |     33.677

• Edited by benofthecloud 11 hours 26 minutes ago

Microsoft licenses the physical host, not the VMs.  So whether you are running Windows Server on the physical host or Windows Server in a VM, the physical host has to have a license assigned.  After all, Microsoft has to accommodate running a hypervisor other than Hyper-V, too, and have the same licensing cost no matter what hypervisor is running.

The only difference between Windows Server 2012 R2 Standard Edition and Enterprise Edition is the licensing rights that are conferred with the license.  With a Windows Server Standard Edition assigned to the host, you can install Windows Server Standard w/ GUI, install the Hyper-V role, and simultaneously run two virtual instances of Windows Server.  The caveat is that you cannot run any other roles in the parent partition.  For example, you could not install file services in the parent partition to serve up shares for the virtual machines running on the system.  With Datacenter Edition, you have the right to run an unlimited number of instances of Windows Server on that physical machine.  And, if desired, though it is not recommended, you could even install other roles in the parent partition when the parent partition is running Datacenter.

If you are concerned about licensing - and you should be because it is a legal agreement between you and Microsoft - you should consult with a trained Microsoft licensing specialist.