Hi all, I have been struggling with a BSOD for the past 5 weeks and have scoured the web trying in vain to find someone else with the same issue. Environment: 8 x 2008 R2 SP1 Windows Servers (8Gb RAM, 25Gb HDD) with Remote Desktop Services Roles installed, running as part of an RDS Farm. All Servers are VM Guests (hardware version 7) running on VMware vSphere v4.1.0-260247 Hosts (Dell PowerEdge R710 - 128Gb RAM). Our vSphere 'farm' has 5 Hosts that connect to our EMC SAN via iSCSI with multipath routes. Each RDS Server is load balanced via a Connection Broker, and each server has the same set of software / vm hardware installed. In a nutshell, each has Symantec Endpoint Protection v11.0.5002.333, Symantec Altiris v7.0, Microsoft Office 2007 as well as other various software essential to these servers. Symptoms: Randomly throughout the day, one (or more) of the RDS Servers will crash with a BSOD more often than not with "caused by driver ntoskrnl.exe" sometimes with "cng.sys" and once with "ksecpkg.sys". So far in the 5 weeks I have had 90 crashes. Yesterday all 8 of the RDS Servers crashed at some point throughout the day. On a typical BSOD, it says: ----------------------------- The problem seems to be caused by the following file: ntoskrnl.exe PAGE_FAULT_IN_NONPAGED_AREA Technical Information: *** STOP: 0x00000050 (0xfffffa800c153284, 0x0000000000000001, 0xfffff880053dc0c9, 0x0000000000000000) *** ntoskrnl.exe - Address 0xfffff8000169ac40 base at 0xfffff8000161e000 DateStamp 0x4e02aaa3 ------------------------------ Using BlueScreenView it says "caused by address: ntoskrnl.exe+7cc40" nearly every time. I have analysed as best I could using Microsoft WinDbg, and this is the output of a typical mini-dump file: ------------------------------ Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [\\hqrds01\c$\Windows\Minidump\030112-19359-01.dmp] Mini Kernel Dump File: Only registers and stack trace are available Symbol search path is: SRV*C:\Program Files\Debugging Tools for Windows (x64)\Symbols*http://msdl.microsoft.com/download/symbols Executable search path is: Windows 7 Kernel Version 7601 (Service Pack 1) MP (2 procs) Free x64 Product: Server, suite: TerminalServer Built by: 7601.17640.amd64fre.win7sp1_gdr.110622-1506 Machine Name: Kernel base = 0xfffff800`01609000 PsLoadedModuleList = 0xfffff800`0184e670 Debug session time: Thu Mar 1 09:14:00.921 2012 (UTC + 0:00) System Uptime: 0 days 21:31:41.950 Loading Kernel Symbols ............................................................... ................................................................ ............. Loading User Symbols Loading unloaded module list .............. ******************************************************************************* * Bugcheck Analysis * ******************************************************************************* Use !analyze -v to get detailed debugging information. BugCheck 50, {fffffa800be83284, 1, fffff8800576f0c9, 0} Could not read faulting driver name Probably caused by : RDPWD.SYS ( RDPWD!memcpy+1d9 ) Followup: MachineOwner --------- 1: kd> !analyze -v ******************************************************************************* * Bugcheck Analysis * ******************************************************************************* PAGE_FAULT_IN_NONPAGED_AREA (50) Invalid system memory was referenced. This cannot be protected by try-except, it must be protected by a Probe. Typically the address is just plain bad or it is pointing at freed memory. Arguments: Arg1: fffffa800be83284, memory referenced. Arg2: 0000000000000001, value 0 = read operation, 1 = write operation. Arg3: fffff8800576f0c9, If non-zero, the instruction address which referenced the bad memory address. Arg4: 0000000000000000, (reserved) Debugging Details: ------------------ Could not read faulting driver name WRITE_ADDRESS: GetPointerFromAddress: unable to read from fffff800018b8100 fffffa800be83284 FAULTING_IP: RDPWD!memcpy+1d9 fffff880`0576f0c9 668901 mov word ptr [rcx],ax MM_INTERNAL_CODE: 0 CUSTOMER_CRASH_COUNT: 1 DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP BUGCHECK_STR: 0x50 PROCESS_NAME: svchost.exe CURRENT_IRQL: 0 TRAP_FRAME: fffff8800bf70a80 -- (.trap 0xfffff8800bf70a80) NOTE: The trap frame does not contain all registers. Some register values may be zeroed or incorrect. rax=000000000000023d rbx=0000000000000000 rcx=fffffa800be83284 rdx=ffffffffffe7e63b rsi=0000000000000000 rdi=0000000000000000 rip=fffff8800576f0c9 rsp=fffff8800bf70c18 rbp=0000000000000001 r8=000000000000001c r9=fffff8a0033401e8 r10=fffff8a0033401e8 r11=fffffa800be83268 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl nz na pe nc RDPWD!memcpy+0x1d9: fffff880`0576f0c9 668901 mov word ptr [rcx],ax ds:0c40:fffffa80`0be83284=???? Resetting default scope LAST_CONTROL_TRANSFER: from fffff800016319fc to fffff80001685c40 STACK_TEXT: fffff880`0bf70918 fffff800`016319fc : 00000000`00000050 fffffa80`0be83284 00000000`00000001 fffff880`0bf70a80 : nt!KeBugCheckEx fffff880`0bf70920 fffff800`01683d6e : 00000000`00000001 fffffa80`0be83284 00000000`00000000 fffff8a0`0be85820 : nt! ?? ::FNODOBFM::`string'+0x4611f fffff880`0bf70a80 fffff880`0576f0c9 : fffff880`057547cf 00000000`00000000 00000000`00000022 00000000`00000002 : nt!KiPageFault+0x16e fffff880`0bf70c18 fffff880`057547cf : 00000000`00000000 00000000`00000022 00000000`00000002 fffff880`0576c99d : RDPWD!memcpy+0x1d9 fffff880`0bf70c20 fffff880`0576c9fc : fffff8a0`0f938010 00000000`00000022 00000000`00000019 00000000`00000002 : RDPWD!SM_MCSSendDataCallback+0x303 fffff880`0bf70c60 fffff880`0576b354 : fffff880`0bf70da0 fffff8a0`033401e8 00000000`00000000 fffff880`0576abfd : RDPWD!HandleAllSendDataPDUs+0x188 fffff880`0bf70d10 fffff880`0576af64 : 00000000`00000031 fffffa80`0bd01895 00000006`0000001f fffff880`05739079 : RDPWD!RecognizeMCSFrame+0x28 fffff880`0bf70d50 fffff880`029ba1f8 : fffff8a0`03345000 fffffa80`0bae6e80 fffffa80`0a5c0e60 fffff880`05737e00 : RDPWD!MCSIcaRawInputWorker+0x3d4 fffff880`0bf70df0 fffff880`057378d0 : 00000000`00000000 fffff880`0bf70f10 fffff880`0bf70f08 00000000`00000000 : termdd!IcaRawInput+0x50 fffff880`0bf70e20 fffff880`05736d85 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tssecsrv!CRawInputDM::PassDataToServer+0x2c fffff880`0bf70e50 fffff880`057367c2 : fffffa80`088e8a28 fffffa80`00000000 00000000`00000031 fffff800`00000000 : tssecsrv!CFilter::FilterIncomingData+0xc9 fffff880`0bf70ef0 fffff880`029ba1f8 : fffff880`009b8180 00000000`00000001 00000000`00000000 00000000`00000000 : tssecsrv!ScrRawInput+0x82 fffff880`0bf70f60 fffff880`0572c4c5 : fffffa80`088e8a10 fffffa80`0bd01658 00000000`00000000 fffffa80`088e8a10 : termdd!IcaRawInput+0x50 fffff880`0bf70f90 fffff880`029baf3e : fffffa80`0bd01620 fffffa80`0c100420 fffffa80`0bd4b450 fffffa80`0973b9b0 : tdtcp!TdInputThread+0x465 fffff880`0bf71810 fffff880`029b9ae3 : fffffa80`09d902b0 fffffa80`0973b9b0 fffffa80`093d8520 fffffa80`0bd4b450 : termdd!IcaDriverThread+0x5a fffff880`0bf71840 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : termdd!IcaDeviceControlStack+0x827 STACK_COMMAND: kb FOLLOWUP_IP: RDPWD!memcpy+1d9 fffff880`0576f0c9 668901 mov word ptr [rcx],ax SYMBOL_STACK_INDEX: 3 SYMBOL_NAME: RDPWD!memcpy+1d9 FOLLOWUP_NAME: MachineOwner MODULE_NAME: RDPWD IMAGE_NAME: RDPWD.SYS DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7ab45 FAILURE_BUCKET_ID: X64_0x50_VRF_RDPWD!memcpy+1d9 BUCKET_ID: X64_0x50_VRF_RDPWD!memcpy+1d9 Followup: MachineOwner ------------------------------ The RDS servers are set to reboot automatically, and after a period of 5 minutes or so, the users can reconnect and log back in. On a typical day each server will have around 10 people RDP'd in to them. The Users connecting to the RDS Servers included XP laptops/desktops and IGEL UD-120-LX Thin Terminals. The XPs have SP3 installed and are fully patched via Symantec Altiris. Things I have tried: - Analyse the dump-files (as per above). - I have tracked each user logging on to the RDS Farm (via batch scripts) and tried to determine if this is caused by the same individual(s) but it appears random. - Check to see if the crashing Virtual Machine is running on a specific host, but it has happened on all Hosts. - Check to see if there was anything specific that happened on the day that the crashes started. There were about 5 new poeple introduced to the RDS Farm at that time, but there were using (a) client machines that had been used previously elsewhere with no issues, (b) software that had been used previously, (c) in a remote location that had previous users using RDS, (d) have not been logged on to a RDS Server when it has crashed. - Updated Windows Server 2008 R2 SP1 to the latest patches (as of Feb 2012). - Turned on Verifier (using recommended settings), and then analysed dump-files with the same reference to rdpwd.sys. - Fixed the Memory Resource Reservation in vSphere to the full 8Gb for all these RDS Servers (so that the memory is not shared at all). - Ran MEMTEST on a VM Guest with the full 8Gb RAM, on a couple of the ESX Hosts. - Changed the VMTools Video Driver to the SVGA II driver from the Standard VGA Driver. - Ran a full AV Scan (using SEP). - Isolated the Printer Drivers using the Printer Management MMC. - Ran sfc /scannow of all RDS Servers and rebooted. The mini-dump file mentioned above is here:https://skydrive.live.com/redir.aspx?cid=48f471f287af2349&resid=48F471F287AF2349!105&parid=48F471F287AF2349!103 I hope someone can help, as what hair I have left (from pulling it out) is turning grey! Andy

8 VM's on 1 server blue screening alot sounds to me like a hardware issue. I'm not sure how much you can play around with that server as maybe it needs to be up all the time, but I would consider taking out alot of memory and just running a couple of VMs. If they blue screen, then replace the remaining memory with memory you pulled out and see what happens, and see if you can find some faulty memory stick(s). I know you have run a memory test, but that doesnt mean 100% that its ok. You could try running some of the Dell hardware checks too as they test CPU, chipset, harddisks etc.

Hi Rylandd, Thanks for replying. I apologise, I wasn't quite clear in my post. I have 8 virtual RDS Servers running on 5 different ESX Hosts. eg. RDS1, RDS3 & RDS5 running on ESX Host1 - RDS2, RDS4 running on ESX Host2 etc. It doesn't matter which ESX Host a VM is on, it will still crash - which to my mind, the likelihood of having 5 individual physical ESX Hosts all with RAM issues, highly unlikely. These ESX Host Servers have been in and running for between 3 years and 6 months. Andy

******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* PAGE_FAULT_IN_NONPAGED_AREA (50) Invalid system memory was referenced. This cannot be protected by try-except, it must be protected by a Probe. Typically the address is just plain bad or it is pointing at freed memory. Arguments: Arg1: fffffa800c153284, memory referenced. Arg2: 0000000000000001, value 0 = read operation, 1 = write operation. Arg3: fffff880053dc0c9, If non-zero, the instruction address which referenced the bad memory address. Arg4: 0000000000000000, (reserved) Debugging Details: ------------------ Could not read faulting driver name WRITE_ADDRESS: GetPointerFromAddress: unable to read from fffff800018cd100 fffffa800c153284 FAULTING_IP: RDPWD!memcpy+1d9 fffff880`053dc0c9 668901 mov word ptr [rcx],ax MM_INTERNAL_CODE: 0 CUSTOMER_CRASH_COUNT: 1 DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP BUGCHECK_STR: 0x50 PROCESS_NAME: svchost.exe CURRENT_IRQL: 0 TRAP_FRAME: fffff8800aa48a80 -- (.trap 0xfffff8800aa48a80) NOTE: The trap frame does not contain all registers. Some register values may be zeroed or incorrect. rax=00000000000001ff rbx=0000000000000000 rcx=fffffa800c153284 rdx=ffffffffffee6b8b rsi=0000000000000000 rdi=0000000000000000 rip=fffff880053dc0c9 rsp=fffff8800aa48c18 rbp=0000000000000001 r8=000000000000001c r9=fffff8a0123923a8 r10=fffff8a0123923a8 r11=fffffa800c153268 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl nz na pe nc RDPWD!memcpy+0x1d9: fffff880`053dc0c9 668901 mov word ptr [rcx],ax ds:8c40:fffffa80`0c153284=???? Resetting default scope LAST_CONTROL_TRANSFER: from fffff800016469fc to fffff8000169ac40 STACK_TEXT: fffff880`0aa48918 fffff800`016469fc : 00000000`00000050 fffffa80`0c153284 00000000`00000001 fffff880`0aa48a80 : nt!KeBugCheckEx fffff880`0aa48920 fffff800`01698d6e : 00000000`00000001 fffffa80`0c153284 00000000`00000000 fffff8a0`10919830 : nt! ?? ::FNODOBFM::`string'+0x4611f fffff880`0aa48a80 fffff880`053dc0c9 : fffff880`053c17cf 00000000`00000000 00000000`00000022 00000000`00000002 : nt!KiPageFault+0x16e fffff880`0aa48c18 fffff880`053c17cf : 00000000`00000000 00000000`00000022 00000000`00000002 fffff880`053d999d : RDPWD!memcpy+0x1d9 fffff880`0aa48c20 fffff880`053d99fc : fffff8a0`10cf30d0 00000000`00000022 00000000`00000019 00000000`00000002 : RDPWD!SM_MCSSendDataCallback+0x303 fffff880`0aa48c60 fffff880`053d8354 : fffff880`0aa48da0 fffff8a0`123923a8 00000000`00000000 fffff880`053d7bfd : RDPWD!HandleAllSendDataPDUs+0x188 fffff880`0aa48d10 fffff880`053d7f64 : 00000000`00000031 fffffa80`0c039de5 00000006`0000001f fffff880`053a6079 : RDPWD!RecognizeMCSFrame+0x28 fffff880`0aa48d50 fffff880`012c01f8 : fffff8a0`12393000 fffffa80`0bb7aa60 fffffa80`0b81e9c0 fffff880`053a4e00 : RDPWD!MCSIcaRawInputWorker+0x3d4 fffff880`0aa48df0 fffff880`053a48d0 : 00000000`00000000 fffff880`0aa48f10 fffff880`0aa48f08 fffffa80`0c039ba8 : termdd!IcaRawInput+0x50 fffff880`0aa48e20 fffff880`053a3d85 : fffff880`01716890 fffffa80`0c0327e8 00000000`00000000 00000000`00000000 : tssecsrv!CRawInputDM::PassDataToServer+0x2c fffff880`0aa48e50 fffff880`053a37c2 : fffffa80`0c16e598 fffffa80`00000000 00000000`00000031 fffff800`00000000 : tssecsrv!CFilter::FilterIncomingData+0xc9 fffff880`0aa48ef0 fffff880`012c01f8 : fffff880`009b8180 00000000`00000001 00000000`00000000 00000000`00000000 : tssecsrv!ScrRawInput+0x82 fffff880`0aa48f60 fffff880`052994c5 : fffffa80`0c16e580 fffffa80`0c039ba8 00000000`00000000 fffffa80`0c16e580 : termdd!IcaRawInput+0x50 fffff880`0aa48f90 fffff880`012c0f3e : fffffa80`0c039b70 fffffa80`0acccf20 fffffa80`0a95c450 fffffa80`0abf9620 : tdtcp!TdInputThread+0x465 fffff880`0aa49810 fffff880`012bfae3 : fffffa80`0c0a6560 fffffa80`0abf9620 fffffa80`087eee80 fffffa80`0a95c450 : termdd!IcaDriverThread+0x5a fffff880`0aa49840 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : termdd!IcaDeviceControlStack+0x827 STACK_COMMAND: kb FOLLOWUP_IP: RDPWD!memcpy+1d9 fffff880`053dc0c9 668901 mov word ptr [rcx],ax SYMBOL_STACK_INDEX: 3 SYMBOL_NAME: RDPWD!memcpy+1d9 FOLLOWUP_NAME: MachineOwner MODULE_NAME: RDPWD IMAGE_NAME: RDPWD.SYS DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7ab45 FAILURE_BUCKET_ID: X64_0x50_VRF_RDPWD!memcpy+1d9 BUCKET_ID: X64_0x50_VRF_RDPWD!memcpy+1d9 Followup: MachineOwner --------------------------------------------------------------------------------------------------------------- Bug Check Code 0x50:http://msdn.microsoft.com/en-us/library/windows/hardware/ff559023%28v=vs.85%29.aspx Please start by that: Update all possible driversUninstall all unused programsDisable all security softwares you haveRun chkdsk /r /f and sfc /scannowRun memtest86+ to check if all is okay with your RAM. If an error was detected then replace the faulty RAM or contact your manufacturer Technical Support If this does not help then upload MEMORY.DMP file (You can zip it and divide it using 7-ZIP) using Microsoft Skydrive and post a link here. You can also contact Microsoft CSS for assistance. This posting is provided "AS IS" with no warranties or guarantees , and confers no rights. Microsoft Student Partner 2010 / 2011 Microsoft Certified Professional Microsoft Certified Systems Administrator: Security Microsoft Certified Systems Engineer: Security Microsoft Certified Technology Specialist: Windows Server 2008 Active Directory, Configuration Microsoft Certified Technology Specialist: Windows Server 2008 Network Infrastructure, Configuration Microsoft Certified Technology Specialist: Windows Server 2008 Applications Infrastructure, Configuration Microsoft Certified Technology Specialist: Windows 7, Configuring Microsoft Certified Technology Specialist: Designing and Providing Volume Licensing Solutions to Large Organizations Microsoft Certified IT Professional: Enterprise Administrator Microsoft Certified IT Professional: Server Administrator Microsoft Certified Trainer

Hi Mr X, Many thanks for your reply. I have uploaded 7zip'd memory.dmp files to this location: https://skydrive.live.com/redir.aspx?cid=48f471f287af2349&resid=48F471F287AF2349!103&parid=48F471F287AF2349!101&authkey=!AFEzH_00Z3uMbaU We have updated as many hardware drivers as possible, and uninstalled any programs that are not used. We have turned off Symantec Endpoint for a while (but still had a crash) and have also previously ran a chkdsk and sfc as well. As these RDS servers are virtual we cannot run a physical memtest on them, and cannot turn off an ESX Host to run the test, as each server has 128Gb of ram and hosts about 10 virutal servers - we unfortunately do not have enough spare capacity to allow a Host to be offline. However I am running memtest in a test vm with 8Gb ram similar to the RDS servers, but as of yet, it hasn't encountered a problem. Please let me know if you find anything in the memory.dmp file, as I'm running out of options :-( I am starting to build a fresh RDS Server, but this is really a last resort. Thanks for any help. Andy

Hi AndyPM, I'm having the same problem since middle February. If you managed to find a solution can you please post it? Thanks you!

Hi AndyPM, Thank you for your answer. I will try that as well.

Well, it's been 3 day now an no crash. Looks like that has resolved my issue. I had to subscribe TechNet and use one of the Professional Services tickets to get the dump analysed, but it was worth it to stop the crashes. Not sure how I cloe this or allocate points as Microsoft fixed it ultimately!

We have this same issue but we're a 100% Hyper V shop. Six node cluster, 70 VMs, total of 9 RDS servers, all of which have blue screened randomly citing the same PAGE FAULT IN NON PAGED AREA error as well as a Multi-User Win32 Driver error. I get that by running RDS virtually we're multiplexing CPU calls on top of a system that's multiplexing CPU calls (virtualization within virtualization) but come on...our Citrix hosts never had this problem.

Hi, I wonder if you guys found any new info about this problem. I know that changing GPO settings to above solves crashing, but we have RDS farm in our company and RDP Security Layer means dual login for us :(. Hope MS released some new KB and fix. Regards Meehow

Hi, I wonder if you guys found any new info about this problem. I know that changing GPO settings to above solves crashing, but we have RDS farm in our company and RDP Security Layer means dual login for us :(. Hope MS released some new KB and fix. Regards Meehow I'm wondering the same thing. RDP Security may be a quick and dirty work around but it's by no means a solution for most companies. Does anybody have any idea what Windows Update might be causing this issue? I'm assuming it's a Windows Update because most posts relating to this issue have been posted in the last two months. We have almost the same environment as the OP. Three ESXi 4.1 Servers running Server 2008 R2 RDSH Farm consisting of four servers. The crash happens on every server on a semi-regular basis. All the dump files point to rdpwd.sys as the faulting service.

We had (or have?) the same issue with two Terminal Servers 2008 R2 (all windows updates installed), changed the settings AndyPM provided, since then its stable for 3 days. I think it should take another 4 weeks to be sure things are stable now. By the way our two terminal servers are virtual and running on ESX. Maybe a live move to another ESX server caused this, are you guys also running virtual? Edit: It's been 9 days stable now. Hey Pkluver, any chance you're still running stable after over a month??

Tried to test settings mentioned before, but after week of quiet one of RDS SH crashed .... Still 0x00000050 PAGE_FAULT_IN_NONPAGED_AREA Crying for help ;(

HI AndyPM Just wondering if you had a repeat of this problem on your RDS servers? I've been having the same dramas on and off. Thanks Muppet

Hi MuppetMutant76, Everything has been fine since the recommended change. Still getting double logins to the Farm though, but that is a different story! AndyPM

Hi MuppetMutant76, Everything has been fine since the recommended change. Still getting double logins to the Farm though, but that is a different story! AndyPM I was having the exact same issue and so far your solution has worked for me too. We haven't had a crash in the three weeks following the changes.

Hi Andy I just have it crash on me. Similar enviornment with 6 terminal servers in a farm. I've done the change originally only halfway by changing the security layer to RDP security layer as suggested. I did not change the GPO as it reads, it will just use the encryption level set in rdp session host whether it's disabled or not configured... Unless I'm wrong?

Hi Andy I just have it crash on me. Similar enviornment with 6 terminal servers in a farm. I've done the change originally only halfway by changing the security layer to RDP security layer as suggested. I did not change the GPO as it reads, it will just use the encryption level set in rdp session host whether it's disabled or not configured... Unless I'm wrong? You are correct, if you set it in RDSH properties you shouldn't need to set it in the GPO, though you might want to run RSOP against it just to be sure all the correct settings are being applied.

Hi Andy This is Mutantmuppet, but I can't remember my signin :) Anyway, turns out the fix you suggested didn't work for me. The crashes were still happening and I had no choice but to call Microsoft but fortunately I had one incident ticket with SA. Submitted my memory.dmp file and got the following reply the next day: "The debug team have reviewed the dump file. As per the debug team it appears that the driver named vsepflt.sys is causing the random reboots. This driver vsepflt.sys belongs to VMware. Please contact VMware to update this driver to a newer version or have it disabled to fix this random reboot issue." I found the following forum article for Citrix, though I'm sure it's the same deal: http://forums.citrix.com/thread.jspa?threadID=307666 Just updated my TS servers yesterday and now hoping for the best! Will update again next week. Chris

Hi Andy This is Mutantmuppet, but I can't remember my signin :) Anyway, turns out the fix you suggested didn't work for me. The crashes were still happening and I had no choice but to call Microsoft but fortunately I had one incident ticket with SA. Submitted my memory.dmp file and got the following reply the next day: "The debug team have reviewed the dump file. As per the debug team it appears that the driver named vsepflt.sys is causing the random reboots. This driver vsepflt.sys belongs to VMware. Please contact VMware to update this driver to a newer version or have it disabled to fix this random reboot issue." I found the following forum article for Citrix, though I'm sure it's the same deal: http://forums.citrix.com/thread.jspa?threadID=307666 Just updated my TS servers yesterday and now hoping for the best! Will update again next week. Chris