Can anyone verify whether they have successfully bare metal provisioned an IBM x3650 M4 (7915) host using VMM 2012??

I have managed to deliver the M2 and M3 models successfully, but have gotten hold of our first M4 which is giving us grief.

I think I may have some faulty hardware as the process is falling over at the uEFi PXE boot screen. The host to be provisioned does not connect to the VMM PXE server and just times out. I have tried replicating the BIOS settings the same as are in the M2 and M3, but really don't think I should be having the grief I am.

Anyone who can verify one way or the other would be appreciated for me to feed back to IBM.

Cheers,

adrian

Hi Adrian,

We are in the same boat as you with SCCM PXE. Actually the issue isn't even PXE. We've done some extensive tests with IBM and Microsoft. It seems the NIC is unable to even get an DHCP address, let alone contact the PXE server. We've checked the communication with WireShark and basically what happens is the NIC requests a DHCP address, gets response from DHCP server and never sends ACK.

Looks like a known issue with the firmware. IBM supposedly has released a firmware update, but this doesn't seem to resolve the issue.

The problem stems from IBM's desire to seepd up UEFI boot process by not initializing all devices during boot (so called optimized boot). Unfortunately this means that the NICs don't get properly initialized until it's too late. The firmware update was supposed to have forced initialization of devices that are listed in boot options, but for some reasons this still doesn't work.

George G.

Hey George,

Thanks for your reply, and I can certainly concur with everything you have written. I performed all of that too, and really didn't get much in the way of help from IBM. They were not able to "replicate" the issue.

However, we had still purchased 24 of these M4's so I needed to come up with something....

After much trial and tribulation I was able to determine that the 1.40 uEFI firmware (here is the link to v1.41)http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5092269&brandind=5000008) did in fact return some different results that I could work with....

Without this update, the Wireshark traces returned exactly what you found. If you were to examine the "VM Manager" event log on the VMM server, you should find the following entry around the time you are expecting the PXE boot to actually commence.

Virtual Machine Manager will not deploy an OS to computer with SMBIOS GUID: 00000000-0000-0000-0000-000000000000 MAC: F0:92:1C:5E:0B:EC because the GUID does not match any that have pending OS deployments in VMM. The PXE request is being passed on to the next WDS PXE provider.

Odd behaviour, and I have no idea why a string of zero's is returned! When the 1.40 update was applied, I found the following event logged.

Virtual Machine Manager will not deploy an OS to computer with SMBIOS GUID: 6328989E-5008-11E4-A17A-8BC5B90CD0CA MAC: F0:92:1C:5E:0B:EC because the GUID does not match any that have pending OS deployments in VMM. The PXE request is being passed on to the next WDS PXE provider.

The PXE boot process was still not working, but I could now see that my desired NIC was actually participating in a conversation with the VMM server. It still didn't like the GUID that was being advertised by the new server NIC, but this was a step closer!

The GUID being advertised by the NIC, and the one that is returned by the IMM query during the provisioning process are actually very similar. The first three segments returned by the NIC are just out of order (compared to the IMM queried value).

To make this really simple to understand, lets say the IMM query during the wizard component of the provisioning process returns the following GUID: 12345678-1234-1234-1234-1234567890AB

In the "VM Manager" event log when the machine tries to PXE boot, the following GUID value is advertised by the NIC of the server being provisioned: 78563412-3412-3412-1234-1234567890AB

If you compare these two strings, the last two segments are identical, and the first three are just out of order. So how does this help you might ask?? Well....here's where the beauty of powershell comes in! I am using Orchestrator to tie lots of different things together, and this Hyper-V host provisioning process is just one step in our server deployment process overall.

So what I did was customize the provisioning script that VMM uses, and if it detects an M4 server is being provisioned, then I simply rearrange the IMM queried GUID into what the NIC will actually broadcast, and use that value when I provision the host. Below is the code that I use for this:

$ComputerName="SERVER1
$IMMAddress="192.168.60.101"
$MACAddress="001122334455"
$IPAddress="192.168.60.111"
$ServerModel="M4"

$HostGroup = Get-SCVMHostGroup -Name "CORP"
$RunAsAccount = Get-SCRunAsAccount -Name "BMC - IBM IMM RAA"
$HostProfile = Get-SCVMHostProfile -Name "IBM x3650 $ServerModel - Windows Server 2008 R2 SP1"
$LogicalNetwork = Get-SCLogicalNetwork -Name "NETWORK1"

If ($ServerModel -eq "M4") {
  $VMH = Find-SCComputer -BMCAddress $IMMAddress -BMCRunAsAccount $RunAsAccount -BMCProtocol "IPMI"
  $GUID = $VMH.SMBiosGUID
  $1 = $GUID.Substring(6,2)
  $2 = $GUID.Substring(4,2)
  $3 = $GUID.Substring(2,2)
  $4 = $GUID.Substring(0,2)
  $5 = $GUID.Substring(11,2)
  $6 = $GUID.Substring(9,2)
  $7 = $GUID.Substring(16,2)
  $8 = $GUID.Substring(14,2)
  $9 = $GUID.Substring(18,18)
  $SMBiosGUID = $1 + $2 + $3 + $4 + "-" + $5 + $6 + "-" + $7 + $8 + $9

  }

Else {
  $VMHost = Find-SCComputer -BMCAddress $IMMAddress -BMCRunAsAccount $RunAsAccount -BMCProtocol "IPMI"
  $SMBiosGUID = $VMHost.SMBiosGUID
  }

New-SCVMHost -ComputerName $ComputerName -VMHostProfile $HostProfile -VMHostGroup $HostGroup -BMCAddress $IMMAddress -SMBiosGuid $SMBiosGUID -BMCRunAsAccount $RunAsAccount -RunAsynchronously -BMCProtocol "IPMI" -BMCPort "623" -BypassADMachineAccountCheck -LogicalNetwork $LogicalNetwork -Subnet "192.168.60.0/24" -ManagementAdapterMACAddress $MACAddress -IPAddress $IPAddress

The variables at the top of the script are just example values, as they are actually parsed in via other means.

You wrote in your post that you have encountered this with SCCM. This should be even easier to remedy than VMM if you are importing your new servers into ConfigMgr before building them (prestaging computers as opposed to Unknown computer builds).

If you are importing them one-at-a-time then you will be able to just rearrange the GUID value of the computer object as you create it. Alternatively, if you are importing them from a file, then simply rearrange the GUIDs in the file before importing it.

I can confirm that this problem caused a great amount of angst and frustration and much head-banging-against walls. The end result for me was very satisfying in that I had a repeatable process which works flawlessly!

Hope this very long explanation makes some sense to you and hopefully gets you started!

Cheers,

adrian

• Edited by Adrian Clenshaw Tuesday, September 17, 2013 9:01 AM
• Marked as answer by Adrian Clenshaw Monday, September 23, 2013 9:10 PM

Hey George,

Thanks for your reply, and I can certainly concur with everything you have written. I performed all of that too, and really didn't get much in the way of help from IBM. They were not able to "replicate" the issue.

However, we had still purchased 24 of these M4's so I needed to come up with something....

After much trial and tribulation I was able to determine that the 1.40 uEFI firmware (here is the link to v1.41)http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5092269&brandind=5000008) did in fact return some different results that I could work with....

Without this update, the Wireshark traces returned exactly what you found. If you were to examine the "VM Manager" event log on the VMM server, you should find the following entry around the time you are expecting the PXE boot to actually commence.

Virtual Machine Manager will not deploy an OS to computer with SMBIOS GUID: 00000000-0000-0000-0000-000000000000 MAC: F0:92:1C:5E:0B:EC because the GUID does not match any that have pending OS deployments in VMM. The PXE request is being passed on to the next WDS PXE provider.

Odd behaviour, and I have no idea why a string of zero's is returned! When the 1.40 update was applied, I found the following event logged.

Virtual Machine Manager will not deploy an OS to computer with SMBIOS GUID: 6328989E-5008-11E4-A17A-8BC5B90CD0CA MAC: F0:92:1C:5E:0B:EC because the GUID does not match any that have pending OS deployments in VMM. The PXE request is being passed on to the next WDS PXE provider.

The PXE boot process was still not working, but I could now see that my desired NIC was actually participating in a conversation with the VMM server. It still didn't like the GUID that was being advertised by the new server NIC, but this was a step closer!

The GUID being advertised by the NIC, and the one that is returned by the IMM query during the provisioning process are actually very similar. The first three segments returned by the NIC are just out of order (compared to the IMM queried value).

To make this really simple to understand, lets say the IMM query during the wizard component of the provisioning process returns the following GUID: 12345678-1234-1234-1234-1234567890AB

In the "VM Manager" event log when the machine tries to PXE boot, the following GUID value is advertised by the NIC of the server being provisioned: 78563412-3412-3412-1234-1234567890AB

If you compare these two strings, the last two segments are identical, and the first three are just out of order. So how does this help you might ask?? Well....here's where the beauty of powershell comes in! I am using Orchestrator to tie lots of different things together, and this Hyper-V host provisioning process is just one step in our server deployment process overall.

So what I did was customize the provisioning script that VMM uses, and if it detects an M4 server is being provisioned, then I simply rearrange the IMM queried GUID into what the NIC will actually broadcast, and use that value when I provision the host. Below is the code that I use for this:

$ComputerName="SERVER1
$IMMAddress="192.168.60.101"
$MACAddress="001122334455"
$IPAddress="192.168.60.111"
$ServerModel="M4"

$HostGroup = Get-SCVMHostGroup -Name "CORP"
$RunAsAccount = Get-SCRunAsAccount -Name "BMC - IBM IMM RAA"
$HostProfile = Get-SCVMHostProfile -Name "IBM x3650 $ServerModel - Windows Server 2008 R2 SP1"
$LogicalNetwork = Get-SCLogicalNetwork -Name "NETWORK1"

If ($ServerModel -eq "M4") {
  $VMH = Find-SCComputer -BMCAddress $IMMAddress -BMCRunAsAccount $RunAsAccount -BMCProtocol "IPMI"
  $GUID = $VMH.SMBiosGUID
  $1 = $GUID.Substring(6,2)
  $2 = $GUID.Substring(4,2)
  $3 = $GUID.Substring(2,2)
  $4 = $GUID.Substring(0,2)
  $5 = $GUID.Substring(11,2)
  $6 = $GUID.Substring(9,2)
  $7 = $GUID.Substring(16,2)
  $8 = $GUID.Substring(14,2)
  $9 = $GUID.Substring(18,18)
  $SMBiosGUID = $1 + $2 + $3 + $4 + "-" + $5 + $6 + "-" + $7 + $8 + $9

  }

Else {
  $VMHost = Find-SCComputer -BMCAddress $IMMAddress -BMCRunAsAccount $RunAsAccount -BMCProtocol "IPMI"
  $SMBiosGUID = $VMHost.SMBiosGUID
  }

New-SCVMHost -ComputerName $ComputerName -VMHostProfile $HostProfile -VMHostGroup $HostGroup -BMCAddress $IMMAddress -SMBiosGuid $SMBiosGUID -BMCRunAsAccount $RunAsAccount -RunAsynchronously -BMCProtocol "IPMI" -BMCPort "623" -BypassADMachineAccountCheck -LogicalNetwork $LogicalNetwork -Subnet "192.168.60.0/24" -ManagementAdapterMACAddress $MACAddress -IPAddress $IPAddress

The variables at the top of the script are just example values, as they are actually parsed in via other means.

You wrote in your post that you have encountered this with SCCM. This should be even easier to remedy than VMM if you are importing your new servers into ConfigMgr before building them (prestaging computers as opposed to Unknown computer builds).

If you are importing them one-at-a-time then you will be able to just rearrange the GUID value of the computer object as you create it. Alternatively, if you are importing them from a file, then simply rearrange the GUIDs in the file before importing it.

I can confirm that this problem caused a great amount of angst and frustration and much head-banging-against walls. The end result for me was very satisfying in that I had a repeatable process which works flawlessly!

Hope this very long explanation makes some sense to you and hopefully gets you started!

Cheers,

adrian

• Edited by Adrian Clenshaw Tuesday, September 17, 2013 9:01 AM
• Marked as answer by Adrian Clenshaw Monday, September 23, 2013 9:10 PM

Hey George,

Thanks for your reply, and I can certainly concur with everything you have written. I performed all of that too, and really didn't get much in the way of help from IBM. They were not able to "replicate" the issue.

However, we had still purchased 24 of these M4's so I needed to come up with something....

After much trial and tribulation I was able to determine that the 1.40 uEFI firmware (here is the link to v1.41)http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5092269&brandind=5000008) did in fact return some different results that I could work with....

Without this update, the Wireshark traces returned exactly what you found. If you were to examine the "VM Manager" event log on the VMM server, you should find the following entry around the time you are expecting the PXE boot to actually commence.

Virtual Machine Manager will not deploy an OS to computer with SMBIOS GUID: 00000000-0000-0000-0000-000000000000 MAC: F0:92:1C:5E:0B:EC because the GUID does not match any that have pending OS deployments in VMM. The PXE request is being passed on to the next WDS PXE provider.

Odd behaviour, and I have no idea why a string of zero's is returned! When the 1.40 update was applied, I found the following event logged.

Virtual Machine Manager will not deploy an OS to computer with SMBIOS GUID: 6328989E-5008-11E4-A17A-8BC5B90CD0CA MAC: F0:92:1C:5E:0B:EC because the GUID does not match any that have pending OS deployments in VMM. The PXE request is being passed on to the next WDS PXE provider.

The PXE boot process was still not working, but I could now see that my desired NIC was actually participating in a conversation with the VMM server. It still didn't like the GUID that was being advertised by the new server NIC, but this was a step closer!

The GUID being advertised by the NIC, and the one that is returned by the IMM query during the provisioning process are actually very similar. The first three segments returned by the NIC are just out of order (compared to the IMM queried value).

To make this really simple to understand, lets say the IMM query during the wizard component of the provisioning process returns the following GUID: 12345678-1234-1234-1234-1234567890AB

In the "VM Manager" event log when the machine tries to PXE boot, the following GUID value is advertised by the NIC of the server being provisioned: 78563412-3412-3412-1234-1234567890AB

If you compare these two strings, the last two segments are identical, and the first three are just out of order. So how does this help you might ask?? Well....here's where the beauty of powershell comes in! I am using Orchestrator to tie lots of different things together, and this Hyper-V host provisioning process is just one step in our server deployment process overall.

So what I did was customize the provisioning script that VMM uses, and if it detects an M4 server is being provisioned, then I simply rearrange the IMM queried GUID into what the NIC will actually broadcast, and use that value when I provision the host. Below is the code that I use for this:

$ComputerName="SERVER1
$IMMAddress="192.168.60.101"
$MACAddress="001122334455"
$IPAddress="192.168.60.111"
$ServerModel="M4"

$HostGroup = Get-SCVMHostGroup -Name "CORP"
$RunAsAccount = Get-SCRunAsAccount -Name "BMC - IBM IMM RAA"
$HostProfile = Get-SCVMHostProfile -Name "IBM x3650 $ServerModel - Windows Server 2008 R2 SP1"
$LogicalNetwork = Get-SCLogicalNetwork -Name "NETWORK1"

If ($ServerModel -eq "M4") {
  $VMH = Find-SCComputer -BMCAddress $IMMAddress -BMCRunAsAccount $RunAsAccount -BMCProtocol "IPMI"
  $GUID = $VMH.SMBiosGUID
  $1 = $GUID.Substring(6,2)
  $2 = $GUID.Substring(4,2)
  $3 = $GUID.Substring(2,2)
  $4 = $GUID.Substring(0,2)
  $5 = $GUID.Substring(11,2)
  $6 = $GUID.Substring(9,2)
  $7 = $GUID.Substring(16,2)
  $8 = $GUID.Substring(14,2)
  $9 = $GUID.Substring(18,18)
  $SMBiosGUID = $1 + $2 + $3 + $4 + "-" + $5 + $6 + "-" + $7 + $8 + $9

  }

Else {
  $VMHost = Find-SCComputer -BMCAddress $IMMAddress -BMCRunAsAccount $RunAsAccount -BMCProtocol "IPMI"
  $SMBiosGUID = $VMHost.SMBiosGUID
  }

New-SCVMHost -ComputerName $ComputerName -VMHostProfile $HostProfile -VMHostGroup $HostGroup -BMCAddress $IMMAddress -SMBiosGuid $SMBiosGUID -BMCRunAsAccount $RunAsAccount -RunAsynchronously -BMCProtocol "IPMI" -BMCPort "623" -BypassADMachineAccountCheck -LogicalNetwork $LogicalNetwork -Subnet "192.168.60.0/24" -ManagementAdapterMACAddress $MACAddress -IPAddress $IPAddress

The variables at the top of the script are just example values, as they are actually parsed in via other means.

You wrote in your post that you have encountered this with SCCM. This should be even easier to remedy than VMM if you are importing your new servers into ConfigMgr before building them (prestaging computers as opposed to Unknown computer builds).

If you are importing them one-at-a-time then you will be able to just rearrange the GUID value of the computer object as you create it. Alternatively, if you are importing them from a file, then simply rearrange the GUIDs in the file before importing it.

I can confirm that this problem caused a great amount of angst and frustration and much head-banging-against walls. The end result for me was very satisfying in that I had a repeatable process which works flawlessly!

Hope this very long explanation makes some sense to you and hopefully gets you started!

Cheers,

adrian

• Edited by Adrian Clenshaw Tuesday, September 17, 2013 9:01 AM
• Marked as answer by Adrian Clenshaw Monday, September 23, 2013 9:10 PM

FYI This is still a problem. I've raised a case with IBM and we are experiencing this with the x3850 and x3950. For some reason IBM can't get a handle on this situation and resolve it.