Zombie processes stuck in AFD on Server 2008 R2
Hello everyone, I'm experiencing a really awkward issue. On my personal dedicated server, some processes - only very network intensive ones - cannot terminate properly. They are somehow stuck in afd!AfdCleanupCore, keeping them resident forever (or at least a month). The event log is clean, however I attached using LiveKD and it always results in this: 0: kd> !process C38 Searching for Process with Cid == c38 Cid handle table at fffff8a001fd4000 with 3233 entries in use PROCESS fffffa8019bc02d0 SessionId: 3 Cid: 0c38 Peb: 7efdf000 ParentCid: 1d08 DirBase: 49d12d000 ObjectTable: fffff8a013302460 HandleCount: 129. Image: uTorrent.exe VadRoot fffffa8019b11720 Vads 191 Clone 0 Private 11503. Modified 257527. Locked 0. DeviceMap fffff8a0084c6700 Token fffff8a0142d4060 ElapsedTime 15:19:20.079 UserTime 00:05:47.468 KernelTime 00:05:25.015 QuotaPoolUsage[PagedPool] 0 QuotaPoolUsage[NonPagedPool] 0 Working Set Sizes (now,min,max) (13570, 50, 345) (54280KB, 200KB, 1380KB) PeakWorkingSetSize 16906 VirtualSize 180 Mb PeakVirtualSize 237 Mb PageFaultCount 6186116 MemoryPriority BACKGROUND BasePriority 8 CommitCharge 12351 THREAD fffffa8019c1db10 Cid 0c38.0c5c Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable fffff88009564dd0 NotificationEvent IRP List: fffffa801aaf9830: (0006,0358) Flags: 00000404 Mdl: 00000000 Not impersonating DeviceMap fffff8a0084c6700 Owning Process fffffa8019bc02d0 Image: uTorrent.exe Attached Process N/A Image: N/A Wait Start TickCount 3531960 Ticks: 24316 (0:00:06:19.937) Context Switch Count 1542205 LargeStack UserTime 00:00:17.640 KernelTime 00:00:10.781 Win32 Start Address 0x0000000000906a90 Stack Init fffff880095653f0 Current fffff88009564a00 Base fffff88009566000 Limit fffff8800955b000 Call fffff88009565670 Priority 10 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`09564a40 fffff800`018856c2 nt!KiSwapContext+0x7a fffff880`09564b80 fffff800`01896baf nt!KiCommitThreadWait+0x1d2 fffff880`09564c10 fffff880`013b1b14 nt!KeWaitForSingleObject+0x19f fffff880`09564cb0 fffff880`013b1e02 afd!AfdCleanupCore+0x444 fffff880`09564e30 fffff800`01b9677f afd!AfdDispatch+0x42 fffff880`09564e80 fffff800`01b85eb4 nt!IopCloseFile+0x11f fffff880`09564f10 fffff800`01b85c71 nt!ObpDecrementHandleCount+0xb4 fffff880`09564f90 fffff800`01b45f90 nt!ObpCloseHandleTableEntry+0xb1 fffff880`09565020 fffff800`01b45e90 nt!ObpCloseHandleProcedure+0x30 fffff880`09565060 fffff800`01b4651a nt!ExSweepHandleTable+0x74 fffff880`095650a0 fffff800`01b647d2 nt!ObKillProcess+0x62 fffff880`095650e0 fffff800`01b460d8 nt!PspExitThread+0x522 fffff880`095651e0 fffff800`0188efd3 nt!NtTerminateProcess+0x138 fffff880`09565260 00000000`770715da nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`09565260) 00000000`0008cb18 00000000`7474601a ntdll!NtTerminateProcess+0xa 00000000`0008cb20 00000000`00000000 0x7474601a I don't really know what to do from there on, the IRP list isn't helpful either 0: kd> !irp fffffa801aaf9830 Irp is active with 4 stacks 4 is current (= 0xfffffa801aaf99d8) No Mdl: No System Buffer: Thread fffffa8019c1db10: Irp stack trace. cmd flg cl Device File Completion-Context [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 >[ 12, 0] 0 0 fffffa80133018e0 fffffa8019c7bbb0 00000000-00000000 \Driver\AFD Args: 00000000 00000000 00000000 00000000 I'm not very experienced with kernel debugging, tried to make any sense of the nt!KeWaitForSingleObject call, however from what I found as parameters for it, this didn't make sense to me either 0: kd> kb *** Stack trace for last set context - .thread/.cxr resets it RetAddr : Args to Child : Call Site fffff800`018856c2 : fffffa80`19c1db10 fffffa80`19c1db10 fffffa80`00000000 00000000`00000000 : nt!KiSwapContext+0x7a fffff800`01896baf : 00000000`00000002 fffff880`09564d08 fffff880`00000000 fffff880`017181bb : nt!KiCommitThreadWait+0x1d2 fffff880`013b1b14 : fffffa80`13308800 00000000`00000000 01cd14d5`3f586100 fffffa80`18442900 : nt!KeWaitForSingleObject+0x19f fffff880`013b1e02 : 00000000`00000000 fffffa80`19c7bbb0 fffffa80`184428d0 fffff800`018a3e96 : afd!AfdCleanupCore+0x444 fffff800`01b9677f : fffffa80`1aaf9830 fffffa80`19bc02d0 00000000`00000000 fffffa80`19c7bbb0 : afd!AfdDispatch+0x42 fffff800`01b85eb4 : 00000000`00000000 fffff8a0`13302460 00000000`00000000 fffff800`0189900c : nt!IopCloseFile+0x11f fffff800`01b85c71 : fffff8a0`13302460 fffff8a0`00000001 fffff8a0`13302460 00000000`00000002 : nt!ObpDecrementHandleCount+0xb4 fffff800`01b45f90 : 00000000`00000560 fffff8a0`13302460 fffff8a0`14392580 00000000`00000560 : nt!ObpCloseHandleTableEntry+0xb1 fffff800`01b45e90 : 00000000`00000404 00000000`00000000 fffffa80`19bc02d0 fffff800`01b33411 : nt!ObpCloseHandleProcedure+0x30 fffff800`01b4651a : fffff8a0`142d4001 00000000`00000001 fffffa80`19bc02d0 00000000`00000001 : nt!ExSweepHandleTable+0x74 fffff800`01b647d2 : fffff8a0`142d4060 00000000`00000000 00000000`00000000 00000000`00000000 : nt!ObKillProcess+0x62 fffff800`01b460d8 : 00000000`c000004b 00000000`00000001 00000000`7efdb000 00000000`00000000 : nt!PspExitThread+0x522 fffff800`0188efd3 : fffffa80`19bc02d0 fffff880`c000004b fffffa80`19c1db10 fffffa80`18010c80 : nt!NtTerminateProcess+0x138 00000000`770715da : 00000000`7474601a 00000000`00000000 00000000`74786138 00000000`0008cb00 : nt!KiSystemServiceCopyEnd+0x13 00000000`7474601a : 00000000`00000000 00000000`74786138 00000000`0008cb00 00000000`747401cc : ntdll!NtTerminateProcess+0xa 00000000`00000000 : 00000000`74786138 00000000`0008cb00 00000000`747401cc 00000000`7efdd000 : 0x7474601a Considering this is related to networking, the server has a Intel 82574L Ethernet card running the latest Intel PROSet 17.0 drivers, however I also experienced it with the previously installed 16.5 drivers. I tried installing the latest hotfix affecting AFD and in the hope it would fix anything, however this was not the case. Is there anything I can do from here on? On a side note, you've probably noticed that this exemplary dump is of uTorrent. It's being used for legal seeding purposes and the issue is not uTorrent specific either. Other very I/O intense processes are also affected.
April 8th, 2012 12:51pm

Hi, Please uninstall uTorrent.exe to check the result. If it does not work, you may contact Microsoft Customer Service and Support (CSS) via telephone so that a dedicated Support Professional can assist with your request. To troubleshoot this kind of kernel crash issue, we need to debug the crashed system dump. Unfortunately, debugging is beyond what we can do in the forum. Please be advised that contacting phone support will be a charged call. To obtain the phone numbers for specific technology request please take a look at the web site listed below: http://support.microsoft.com/default.aspx?scid=fh;EN-US;OfferProPhone#faq607 Regards, Arthur Li TechNet Subscriber Support If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.Arthur Li TechNet Community Support
Free Windows Admin Tool Kit Click here and download it now
April 9th, 2012 5:11am

Hi, thanks for you answer. I am aware this requires kernel debugging; I was rather asking for tips in what direction I could search to find the culprit. Uninstalling uTorrent will not resolve the issue, in the past we also experienced this with other applications. Plus, it's stuck in kernel mode and AFD, that's an issue uTorrent might trigger, but not something it could cause itself - after all, it has to be some kernel mode component locking it up. About phone support, as I let shine through, 'we' is a group of a few individuals who rent a dedicated server at a hosting provider. I'm aware of the phone support options, but you'll understand that $259 for a call which might resolve an issue is a lot for us - certainly for an issue that is non critical. After all, the server does not crash, it just has occasionally ghost processes, which is a nuisance, but it's not something I'd pay $259 for. Rather, I came here in the hope somebody could give me some guidance for what I could do myself to fix it. (yes, I'm a TN Subscriber, but the subscription is unrelated to my spare time activities - which this rental is. Redeeming the free calls of it on this issue is not really an option) Regards, ZetaZynK
April 9th, 2012 5:56am

Hi, Please uninstall uTorrent.exe to check the result. If it does not work, you may contact Microsoft Customer Service and Support (CSS) via telephone so that a dedicated Support Professional can assist with your request. To troubleshoot this kind of kernel crash issue, we need to debug the crashed system dump. Unfortunately, debugging is beyond what we can do in the forum. Please be advised that contacting phone support will be a charged call. To obtain the phone numbers for specific technology request please take a look at the web site listed below: http://support.microsoft.com/default.aspx?scid=fh;EN-US;OfferProPhone#faq607 Regards, Arthur Li TechNet Subscriber Support If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.Arthur Li TechNet Community Support
Free Windows Admin Tool Kit Click here and download it now
April 9th, 2012 12:10pm

Hi, As I mentioned that debugging is beyond what we can do in the forum. Phone support is the best option for you. Thanks for your understanding. Based on the current situation, I would like to confirm what is the current situation? If the issue persists, I will also try to involve a high level support for you to troubleshoot this issue together with you. Regards, Arthur Li TechNet Subscriber Support If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.Arthur Li TechNet Community Support
April 21st, 2012 2:11am

Hi, I would like to confirm what is the current situation? If there is anything that I can do for you, please do not hesitate to let me know, and I will be happy to help. Regards, Arthur Li TechNet Subscriber Support If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.Arthur Li TechNet Community Support
Free Windows Admin Tool Kit Click here and download it now
April 23rd, 2012 1:25am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics