Event ID 106 when a Hyper-V virtual switch is bound to an LBFO team

I have followed the instructions in KB2974384 But I'm still getting the Error 106. The results of Get-NetAdapterRSS is as follows:

PS C:\Windows\system32> Get-NetAdapterRss "Virtual - Caribe*"


Name                                            : Virtual - Caribe2
InterfaceDescription                            : Intel(R) Gigabit 4P I350-t rNDC #2
Enabled                                         : True
NumberOfReceiveQueues                           : 2
Profile                                         : Closest
BaseProcessor: [Group:Number]                   : :10
MaxProcessor: [Group:Number]                    : :63
MaxProcessors                                   : 8
RssProcessorArray: [Group:Number/NUMA Distance] :
IndirectionTable: [Group:Number]                :

Name                                            : Virtual - Caribe1
InterfaceDescription                            : Intel(R) Gigabit 4P I350-t Adapter #3
Enabled                                         : True
NumberOfReceiveQueues                           : 2
Profile                                         : Closest
BaseProcessor: [Group:Number]                   : :0
MaxProcessor: [Group:Number]                    : :63
MaxProcessors                                   : 8
RssProcessorArray: [Group:Number/NUMA Distance] :
IndirectionTable: [Group:Number]                :

Name                                            : Virtual - Caribe
InterfaceDescription                            : Microsoft Network Adapter Multiplexor Driver
Enabled                                         : True
NumberOfReceiveQueues                           : 0
Profile                                         : NUMAStatic
BaseProcessor: [Group:Number]                   : 0:0
MaxProcessor: [Group:Number]                    : 0:38
MaxProcessors                                   : 4
RssProcessorArray: [Group:Number/NUMA Distance] : 0:0/0  0:2/0  0:4/0  0:6/0  0:8/0  0:10/0  0:12/0  0:14/0
                                                  0:16/0  0:18/0  0:20/0  0:22/0  0:24/0  0:26/0  0:28/0  0:30/0
                                                  0:32/0  0:34/0  0:36/0  0:38/0
IndirectionTable: [Group:Number]                :

Any help would be appreciated

August 7th, 2015 5:39pm

Can you tell us which LBFO teaming mode you are running for the virtual switch?
Can you tell us which physical adapters are in the team? Then run a Get-NetAdaptervmq

RSS wont be used on an LBFO team with a virtual switch bound to it.

Free Windows Admin Tool Kit Click here and download it now
August 11th, 2015 10:14am

LBFO teaming mode is "Switch Independent"

Virtual - Caribe1 and Virtual - Caribe2 are the physical adapters in the team

Name                           InterfaceDescription              Enabled BaseVmqProcessor MaxProcessors NumberOfReceive
                                                                                                        Queues
----                           --------------------              ------- ---------------- ------------- ---------------
Virtual - Caribe               Microsoft Network Adapter Mult... True    0:0                            14
Virtual - Caribe2              Intel(R) Gigabit 4P I350-t r...#2 True    0:10             8             7
Virtual - Caribe1              Intel(R) Gigabit 4P I350-t A...#3 True    0:0              8             7

Thanks

Keith

August 11th, 2015 12:31pm

Could be close to an answer but which load distribution mode are you using? ie:
Switch Independent - Dynamic
Switch Independent - Hyper-V Port
Switch Independent - Address Hash

The load distribution affects how we should assign VMQ Procs

Also are you running Intel or AMD Processors, do you have Hyper Threading enabled? How many real, physical cores per socket? How many sockets? Do you know the max number of VMQ's and RSS Queues supported by the NICs (the driver will normally tell you in device manager)

  • Edited by Microsoft Jim Tuesday, August 11, 2015 1:16 PM forgot some additional info
Free Windows Admin Tool Kit Click here and download it now
August 11th, 2015 1:12pm

I just spotted one more thing... How come you have VMQ enabled for 1Gb NICs? did you enable this yourself? There isn't much point in using VMQ until you are using 10GB adapters as 2 1Gb NICs are very unlikely to max out a single core. We get to about 4Gbits/sec before we peg a CPU core at 100%.
August 11th, 2015 1:29pm

Switch Independent - Dynamic. These are for use by my Hyper-V VMs

Intel Xeon processors with Hyper Threading enabled

Logical to Physical Processor Map:
**--------------------------------------  Physical Processor 0 (Hyperthreaded)
--**------------------------------------  Physical Processor 1 (Hyperthreaded)
----**----------------------------------  Physical Processor 2 (Hyperthreaded)
------**--------------------------------  Physical Processor 3 (Hyperthreaded)
--------**------------------------------  Physical Processor 4 (Hyperthreaded)
----------**----------------------------  Physical Processor 5 (Hyperthreaded)
------------**--------------------------  Physical Processor 6 (Hyperthreaded)
--------------**------------------------  Physical Processor 7 (Hyperthreaded)
----------------**----------------------  Physical Processor 8 (Hyperthreaded)
------------------**--------------------  Physical Processor 9 (Hyperthreaded)
--------------------**------------------  Physical Processor 10 (Hyperthreaded)
----------------------**----------------  Physical Processor 11 (Hyperthreaded)
------------------------**--------------  Physical Processor 12 (Hyperthreaded)
--------------------------**------------  Physical Processor 13 (Hyperthreaded)
----------------------------**----------  Physical Processor 14 (Hyperthreaded)
------------------------------**--------  Physical Processor 15 (Hyperthreaded)
--------------------------------**------  Physical Processor 16 (Hyperthreaded)
----------------------------------**----  Physical Processor 17 (Hyperthreaded)
------------------------------------**--  Physical Processor 18 (Hyperthreaded)
--------------------------------------**  Physical Processor 19 (Hyperthreaded)

Logical Processor to Socket Map:
********************--------------------  Socket 0
--------------------********************  Socket 1

As for having VMQ enabled, this was enabled by default

Max RSS Queues - 8

Max VMQ Queues - I do not see a section for VMQ Queues, but there is a setting for VMQ Vports - 7

Thanks

Keith

Free Windows Admin Tool Kit Click here and download it now
August 11th, 2015 3:12pm

Ok so "Switch Independant - Dynamic" run ins "Sum of Queues" mode. Meaning VMQ processors should not be overlapping.
So your config so far looks pretty good. I'd tweak your config slightly though to avoid LP0

Looking at your CPU allocation you would be crossing NUMA nodes which is BAD for VMQ. Rmember VMQ will only use non-hyperthreaded REAL cores. So if you allocate 8 procs you would be using:
LP0 LP2 LP4 LP6 LP8 LP10 LP12 LP14 (With LP10 and onwards being on a different socket \ NUMA node) Do you have the model number of the CPU(s) so I can research?

EventID 106 can also occure when the queues cross NUMA Node, I know there are known issues around crossing NUMA when running MIN mode, not sure for SUM mode though. we run MIN.

Try this first as you are only running 1Gbbit NICs the odds of a single 1Gbs stream maxing a core are slim to none, unless you have very low clockspeed cores on the CPUs. I very much doubt you would need more than 2 cores with your current config. The VMQs themselves can share cores and will only move over to an additional core when the first core hits >90% CPU utilization.

Set-NetAdapterVMQ Name "Virtual - Caribe1" BaseProcessorNumber 2 MaxProcessors 2

Set-NetAdapterVMQ Name "Virtual - Caribe2" BaseProcessorNumber 6 MaxProcessors 2

If this clears your error (could we get a post of the exact EventID that you are seeing?) then you are good to go.

Could you also run a Get-NetAdapterVMQQueue so we can see what allocations (if any) are currently taking place?

August 12th, 2015 8:29am

Ok, I ran the commands you suggested, will let you know if I get the error again. The Error I'm getting is:

Log Name:      System
Source:        Microsoft-Windows-Hyper-V-VmSwitch
Date:          8/12/2015 3:07:53 AM
Event ID:      106
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      CRHYPERV01.cariberoyale.com
Description:
Available processor sets of the underlying physical NICs belonging to the LBFO team NIC /DEVICE/{90CAE69A-E89C-482C-9655-D1C7ACE374C9} (Friendly Name: Microsoft Network Adapter Multiplexor Driver) on switch 8250DBC5-43BB-40E8-BD72-20E310665333 (Friendly Name: Virtual - Caribe) are not configured correctly. Reason: The processor sets overlap when LBFO is configured with sum-queue mode.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Hyper-V-VmSwitch" Guid="{67DC0D66-3695-47C0-9642-33F76F7BD7AD}" />
    <EventID>106</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2015-08-12T07:07:53.520706600Z" />
    <EventRecordID>32690</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="9700" />
    <Channel>System</Channel>
    <Computer>CRHYPERV01.cariberoyale.com</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="NicNameLen">46</Data>
    <Data Name="NicName">/DEVICE/{90CAE69A-E89C-482C-9655-D1C7ACE374C9}</Data>
    <Data Name="NicFNameLen">44</Data>
    <Data Name="NicFName">Microsoft Network Adapter Multiplexor Driver</Data>
    <Data Name="SwitchNameLen">36</Data>
    <Data Name="SwitchName">8250DBC5-43BB-40E8-BD72-20E310665333</Data>
    <Data Name="SwitchFNameLen">16</Data>
    <Data Name="SwitchFName">Virtual - Caribe</Data>
    <Data Name="QueueMode">2</Data>
  </EventData>
</Event>

I get this error twice

PS C:\Windows\system32> Get-NetAdapterVmqQueue "Virtual - Caribe*"

Name                           QueueID MacAddress        VlanID Processor VmFriendlyName
----                           ------- ----------        ------ --------- --------------
Virtual - Caribe               0                                0:0       CRHYPERV01
Virtual - Caribe               1       00-15-5D-01-CA-0E        0:6       CRDELINT
Virtual - Caribe               2       00-15-5D-01-CA-10        0:2       CRGMS
Virtual - Caribe               3       00-15-5D-01-CA-0F        0:6       CRGIFTCERT
Virtual - Caribe               4       00-15-5D-01-D1-14        0:6       BVSADP
Virtual - Caribe               5       00-15-5D-01-CA-16        0:2       CRALT
Virtual - Caribe               6       00-15-5D-01-D1-16        0:6       CRANALYZER
Virtual - Caribe               7       00-15-5D-01-D1-1A        0:2       CRXIRRUS
Virtual - Caribe               8       00-15-5D-01-D1-1F        0:6       HRADP
Virtual - Caribe               9       00-15-5D-01-D1-20        0:2       MONITOR
Virtual - Caribe2              0                                0:0       CRHYPERV01
Virtual - Caribe2              1       00-15-5D-01-CA-0E        0:6       CRDELINT
Virtual - Caribe2              2       00-15-5D-01-CA-0F        0:6       CRGIFTCERT
Virtual - Caribe2              3       00-15-5D-01-D1-14        0:6       BVSADP
Virtual - Caribe2              4       00-15-5D-01-D1-16        0:6       CRANALYZER
Virtual - Caribe2              5       00-15-5D-01-D1-1F        0:6       HRADP
Virtual - Caribe1              0                                0:0       CRHYPERV01
Virtual - Caribe1              1       00-15-5D-01-CA-10        0:2       CRGMS
Virtual - Caribe1              2       00-15-5D-01-CA-16        0:2       CRALT
Virtual - Caribe1              3       00-15-5D-01-D1-1A        0:2       CRXIRRUS
Virtual - Caribe1              4       00-15-5D-01-D1-20        0:2       MONITOR

The Model Number of the processors is Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz (10 cores)

Thanks

Free Windows Admin Tool Kit Click here and download it now
August 12th, 2015 11:27am

Looks like MVs are being assigned VMQ correctly. Please monitor and let us know the outcome. How often were you seeing the error?
  • Marked as answer by KWF1976 10 hours 46 minutes ago
August 12th, 2015 3:54pm

I was seeing them every couple of hours, but since I made the last change you suggested, nothing

Thanks Keith

Free Windows Admin Tool Kit Click here and download it now
August 14th, 2015 4:22pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics