Hi, I am having a reboot issue with a number of my servers. They get stuck at "shutdown" for 5 minutes - each time I reboot them. I want to debug winlogon at shutdown, but when I try to do this I am blue screening my test server each time and I am not sure why? Heres how I have set it up: On target machine: Installed Debugger SDK bcdedit /dbgsettings as default on w2k8R2 Null modem cable connected to serial - com1 Boot into debugger mode Attach kernel debug to winlogon.exe via the registry [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\winlogon.exe] "Debugger"="c:\program files......\ntsd.exe -d" (full path not listed) On the other side I have: An XP laptop with installed debugger SDK connected to the other end of the null modem cable - com 1 - (115200) running windbg with the correct settings com1 - 115200 baud rate etc. When I reboot my target server (for the degugger registry settings to take effect) the server blue screens with 00000000F4 and I am not able to get the server to boot again? I have also tried attaching the debugger to winlogon via NTSD at the command line on the target, but the server just hangs (not sure if this is the debugger working or not?) and my windbg session on the other said says not connected. I cant do anything to get it connected - F5/Go/Break etc. Am I doing anything wrong? Thanks

According to the documentation with WinDbg (Use debugger command .hh to open or use the help menu, under Debugging Techniques\Advanced Debugging Techniques\Debugging Winlogon), it appears that you may want the registry value to read ntsd.exe -d -x -g and you also need to set the GlobalFlag value under the winlogon.exe key to 0x000400F0 (FLG_ENABLE_KDEBUG_SYMBOL_LOAD) -- Mike Burr

Hi Mike, Thanks for your reply. How do I set the GlobalFlag value under the winlogon.exe key to 0x000400F0 (FLG_ENABLE_KDEBUG_SYMBOL_LOAD) ?

It's weird, but the documentation says that it is a string value under that same WinLogon.exe key (I ended up having to create this one manually) HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\WinLogon.EXE From there, it says that the Debugger and the GlobalFlag values are supposed to be Strings (REG_SZ) Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\WinLogon.exe] "Debugger"="ntsd -d -x -g" "GlobalFlag"="0x000400F0" You may also have to set the DebuggerPath value described here . -- Mike Burr

Hi, I was using ntsd -d -x -g before, I have the full path to ntsd.exe listed and I have tried inserting the additional GlobalFlag as discussed - but the machine still blue screens? Any other ideas? Thanks,

At this point, I'm not sure, I will have to work with it in a VM at home... -- Mike Burr

A quick update, I'm sort of in the same spot, so I am still looking at what mix of registry entries allows us to debug winlogon. kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* WINLOGON_FATAL_ERROR (c000021a) The Winlogon process terminated unexpectedly. Arguments: Arg1: fffff8a005edad70, String that identifies the problem. Arg2: 0000000000000001, Error Code. Arg3: ffffffffc0000034 Arg4: 00000000001004d0 Debugging Details: ------------------ BUGCHECK_STR: 0xc000021a_1 ERROR_CODE: (NTSTATUS) 0xc000021a - {Fatal System Error} The %hs system process terminated unexpectedly with a status of 0x%08x (0x%08x 0x%08x). The system has been shut down. EXCEPTION_CODE: (NTSTATUS) 0xc000021a - {Fatal System Error} The %hs system process terminated unexpectedly with a status of 0x%08x (0x%08x 0x%08x). The system has been shut down. EXCEPTION_PARAMETER1: fffff8a005edad70 EXCEPTION_PARAMETER2: 0000000000000001 EXCEPTION_PARAMETER3: ffffffffc0000034 EXCEPTION_PARAMETER4: 1004d0 ADDITIONAL_DEBUG_TEXT: initial session process or DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT PROCESS_NAME: System CURRENT_IRQL: 2 LAST_CONTROL_TRANSFER: from fffff800027c0732 to fffff800026c17a0 STACK_TEXT: fffff880`02f890b8 fffff800`027c0732 : 00000000`c0000201 fffffa80`0910f680 00000000`00000065 fffff800`02707c24 : nt!RtlpBreakWithStatusInstruction fffff880`02f890c0 fffff800`027c151e : fffffa80`00000003 00000000`00000000 fffffa80`0a938c00 00000000`00000000 : nt!KiBugCheckDebugBreak+0x12 fffff880`02f89120 fffff800`026c9844 : 00000000`00000001 fffff800`02915926 fffffa80`00000040 fffffa80`0a7ea548 : nt!KeBugCheck2+0x71e fffff880`02f897f0 fffff800`029190a7 : 00000000`0000004c 00000000`c000021a fffff880`02e29608 fffffa80`0a8bba30 : nt!KeBugCheckEx+0x104 fffff880`02f89830 fffff800`02919954 : 00000000`00000001 ffffffff`800002f8 ffffffff`800002f8 00000000`00000000 : nt!PopGracefulShutdown+0x257 fffff880`02f89870 fffff800`026c8993 : fffffa80`0910f680 00000000`00401800 00000000`00000420 00000000`00000000 : nt!NtSetSystemPowerState+0x864 fffff880`02f899b0 fffff800`026c4f30 : fffff800`02b37eb9 00000000`00000004 00000000`00000001 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 fffff880`02f89b48 fffff800`02b37eb9 : 00000000`00000004 00000000`00000001 00000000`00000000 fffff800`026d0992 : nt!KiServiceLinkage fffff880`02f89b50 fffff800`02b3813c : 00000000`00000004 fffff880`010f9e00 00000000`00000004 fffff800`026d42a1 : nt!PopIssueActionRequest+0x1d9 fffff880`02f89be0 fffff800`026885fd : 00000000`00000001 fffffa80`0910f600 ffffffff`00000000 00000002`00000005 : nt!PopPolicyWorkerAction+0x4c fffff880`02f89c40 fffff800`026d6961 : fffff880`00000002 fffff880`00000004 fffff800`02688500 00000000`00000000 : nt!PopPolicyWorkerThread+0xfd fffff880`02f89cb0 fffff800`0296dc06 : 00000000`00000000 fffffa80`0910f680 00000000`00000080 fffffa80`090f5040 : nt!ExpWorkerThread+0x111 fffff880`02f89d40 fffff800`026a7c26 : fffff800`02843e80 fffffa80`0910f680 fffffa80`0910fb60 00000000`00000000 : nt!PspSystemThreadStartup+0x5a fffff880`02f89d80 00000000`00000000 : fffff880`02f8a000 fffff880`02f84000 fffff880`02f89400 00000000`00000000 : nt!KxStartSystemThread+0x16 STACK_COMMAND: kb FOLLOWUP_IP: nt!NtSetSystemPowerState+864 fffff800`02919954 cc int 3 SYMBOL_STACK_INDEX: 5 SYMBOL_NAME: nt!NtSetSystemPowerState+864 FOLLOWUP_NAME: MachineOwner MODULE_NAME: nt IMAGE_NAME: ntkrnlmp.exe DEBUG_FLR_IMAGE_TIMESTAMP: 4c1c44a9 FAILURE_BUCKET_ID: X64_0xc000021a_1_nt!NtSetSystemPowerState+864 BUCKET_ID: X64_0xc000021a_1_nt!NtSetSystemPowerState+864 Followup: MachineOwner --------- kd> g Shutdown occurred at (Sat Oct 23 11:10:05.342 2010 (UTC - 6:00))...unloading all symbol tables. Waiting to reconnect... Connected to Windows 7 7600 x64 target at (Sat Oct 23 11:11:29.333 2010 (UTC - 6:00)), ptr64 TRUE Kernel Debugger connection established. Symbol search path is: srv*c:\Symbols*http://msdl.microsoft.com/download/symbols Executable search path is: Windows 7 Kernel Version 7600 MP (1 procs) Free x64 Built by: 7600.16617.amd64fre.win7_gdr.100618-1621 Machine Name: Kernel base = 0xfffff800`01a1c000 PsLoadedModuleList = 0xfffff800`01c59e50 System Uptime: not available -- Mike Burr

I got it! I realized a couple of things, we were potentially missing a system-wide global flag: http://msdn.microsoft.com/en-us/library/ff541722%28v=VS.85%29.aspx and I also realized that I needed to add the Debugging Tools for Windows executables path to the system-wide path environment variable as well as create a system-wide _NT_SYMBOL_PATH variable. My system variables: _NT_SYMBOL_PATH = srv*c:\Symbols*http://msdl.microsoft.com/download/symbols PATH = %SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files\Debugging Tools for Windows (x64) Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64 Copyright (c) Microsoft Corporation. All rights reserved. Opened \\.\pipe\7guest Waiting to reconnect... Connected to Windows 7 7600 x64 target at (Sat Oct 23 12:28:34.805 2010 (UTC - 6:00)), ptr64 TRUE Kernel Debugger connection established. Symbol search path is: srv*c:\Symbols*http://msdl.microsoft.com/download/symbols Executable search path is: Windows 7 Kernel Version 7600 MP (1 procs) Free x64 Built by: 7600.16385.amd64fre.win7_rtm.090713-1255 Machine Name: Kernel base = 0xfffff800`02663000 PsLoadedModuleList = 0xfffff800`028a0e50 System Uptime: not available Microsoft (R) Windows Debugger Version 6.1.7600.16385 AMD64 Copyright (c) Microsoft Corporation. All rights reserved. CommandLine: winlogon.exe The call to LoadLibrary(ext) failed, Win32 error 0n2 "The system cannot find the file specified." Please check your debugger configuration and/or network access. The call to LoadLibrary(exts) failed, Win32 error 0n2 "The system cannot find the file specified." Please check your debugger configuration and/or network access. The call to LoadLibrary(uext) failed, Win32 error 0n2 "The system cannot find the file specified." Please check your debugger configuration and/or network access. The call to LoadLibrary(ntsdexts) failed, Win32 error 0n2 "The system cannot find the file specified." Please check your debugger configuration and/or network access. Symbol search path is: srv*c:\Symbols*http://msdl.microsoft.com/download/symbols Executable search path is: ModLoad: 00000000`ff5e0000 00000000`ff641000 winlogon.exe ModLoad: 00000000`76e70000 00000000`7701b000 ntdll.dll ModLoad: 00000000`76c50000 00000000`76d6f000 C:\Windows\system32\kernel32.dll ModLoad: 000007fe`fcf20000 000007fe`fcf8b000 C:\Windows\system32\KERNELBASE.dll ModLoad: 00000000`76d70000 00000000`76e6a000 C:\Windows\system32\USER32.dll ModLoad: 000007fe`fd750000 000007fe`fd7b7000 C:\Windows\system32\GDI32.dll ModLoad: 000007fe`fdd30000 000007fe`fdd3e000 C:\Windows\system32\LPK.dll ModLoad: 000007fe`fdc60000 000007fe`fdd2a000 C:\Windows\system32\USP10.dll ModLoad: 000007fe`fe050000 000007fe`fe0ef000 C:\Windows\system32\msvcrt.dll ModLoad: 000007fe`fbd30000 000007fe`fbd6d000 C:\Windows\system32\WINSTA.dll ModLoad: 000007fe`fef40000 000007fe`ff06e000 C:\Windows\system32\RPCRT4.dll ModLoad: 000007fe`fd1a0000 000007fe`fd1ce000 C:\Windows\system32\IMM32.DLL ModLoad: 000007fe`ff070000 000007fe`ff179000 C:\Windows\system32\MSCTF.dll ModLoad: 000007fe`fd840000 000007fe`fd91b000 C:\Windows\system32\ADVAPI32.dll ModLoad: 000007fe`fd6b0000 000007fe`fd6cf000 C:\Windows\SYSTEM32\sechost.dll ModLoad: 000007fe`fcdd0000 000007fe`fcddf000 C:\Windows\system32\profapi.dll ModLoad: 000007fe`fcdb0000 000007fe`fcdc4000 C:\Windows\system32\RpcRtRemote.dll ModLoad: 000007fe`fbd20000 000007fe`fbd24000 C:\Windows\system32\KBDUS.DLL ModLoad: 000007fe`fbd10000 000007fe`fbd14000 C:\Windows\system32\KBDUS.DLL ModLoad: 000007fe`fcca0000 000007fe`fccf7000 C:\Windows\system32\apphelp.dll (270.280): Unknown exception - code 000006ba (first chance) -- Mike Burr

Hi Mike, Thanks for the update. I have tried this in my VM but I cant get it to work. Are you still using the global flag under winlogon.exe key? Where are you setting the system wide global flag? Where are you creating those system wide variables on the debugger or the machine being debugged? Thanks

Yeah, I got it to work both with and without the Image Execution Options\WinLogon.exe GlobalFlag value. Here are my current registry entries for global flags. Here is the one for the image file execution options: Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\WinLogon.exe] "Debugger"="ntsd -d -x -g" "GlobalFlag"="0x000400F0" And here is the system-wide one. Note that this one exported more than actually changed, the only thing that is important here is the GlobalFlag value Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager] ... "GlobalFlag"=dword:04000400 ... One thing that is odd is that the documentation calls for a string, where the gflags.exe utility sets the system wide one as a DWORD value. As for the system-wide environment variable, are you able to get into the system at all? I created it by going to Start -> right click computer, select Properties -> Advanced System Settings -> Environment variables and I modified/added to the ones labeled "System variables." If you only have access to the registry, then you might be able to try the _NT_SYMBOL_PATH and PATH values from this registry key that I found. Note that the one that you likely need to edit (based on this older article ) is the one in HKLM\System\CurrentControlSet (though you will also see ones in ControlSet001 and ControlSet002). Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment] "_NT_SYMBOL_PATH"="srv*c:\\Symbols*http://msdl.microsoft.com/download/symbols" "ComSpec"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\ 74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,63,\ 00,6d,00,64,00,2e,00,65,00,78,00,65,00,00,00 "FP_NO_HOST_CHECK"="NO" "NUMBER_OF_PROCESSORS"="1" "OS"="Windows_NT" "Path"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,\ 00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,3b,00,25,00,\ 53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,3b,00,25,\ 00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,\ 53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,57,00,62,00,65,00,6d,\ 00,3b,00,25,00,53,00,59,00,53,00,54,00,45,00,4d,00,52,00,4f,00,4f,00,54,00,\ 25,00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,57,00,69,\ 00,6e,00,64,00,6f,00,77,00,73,00,50,00,6f,00,77,00,65,00,72,00,53,00,68,00,\ 65,00,6c,00,6c,00,5c,00,76,00,31,00,2e,00,30,00,5c,00,3b,00,43,00,3a,00,5c,\ 00,50,00,72,00,6f,00,67,00,72,00,61,00,6d,00,20,00,46,00,69,00,6c,00,65,00,\ 73,00,5c,00,44,00,65,00,62,00,75,00,67,00,67,00,69,00,6e,00,67,00,20,00,54,\ 00,6f,00,6f,00,6c,00,73,00,20,00,66,00,6f,00,72,00,20,00,57,00,69,00,6e,00,\ 64,00,6f,00,77,00,73,00,20,00,28,00,78,00,36,00,34,00,29,00,00,00 "PATHEXT"=".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC" "PROCESSOR_ARCHITECTURE"="AMD64" "PROCESSOR_IDENTIFIER"="Intel64 Family 6 Model 15 Stepping 11, GenuineIntel" "PROCESSOR_LEVEL"="6" "PROCESSOR_REVISION"="0f0b" "PSModulePath"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,\ 00,74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,\ 57,00,69,00,6e,00,64,00,6f,00,77,00,73,00,50,00,6f,00,77,00,65,00,72,00,53,\ 00,68,00,65,00,6c,00,6c,00,5c,00,76,00,31,00,2e,00,30,00,5c,00,4d,00,6f,00,\ 64,00,75,00,6c,00,65,00,73,00,5c,00,00,00 "TEMP"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,\ 00,25,00,5c,00,54,00,45,00,4d,00,50,00,00,00 "TMP"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,\ 25,00,5c,00,54,00,45,00,4d,00,50,00,00,00 "USERNAME"="SYSTEM" "windir"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,\ 00,25,00,00,00 -- Mike Burr

Hi Mike, I was wondering how you got your machines to debug using vmware - I am using vmware ESXi and following this guide http://blog.electric-cloud.com/2009/08/05/how-to-set-up-kernel-debugging-for-windows-in-vmware-esx/ but my target machine just hangs before boot - and my debugger machine just says waiting to reconnect - debuggee not connected?

This I do not know, my background is in Hyper-V. The way it works in Hyper-V is that you create a COM port that maps to a named pipe in the host system, but in this case it doesn't map across the network. Based on the blog post, it looks similar in ESXi, but it seems to connect to the system that the infrastructure client is running. Are you using the latest version of the infrastructure client as well as the ESXi OS?-- Mike Burr