Application pool WSS 3.0 unresponsive from time to time - ONETUTIL problem?
Hi,
One of our application stops responding sometimes. After recycle of app pool site us back in track. There are no waring/errors information in eventviewer, app pool consuming around 150MB of memory and there are a lot of free resources on server. During
last "hang on" I ran Debug Diagnostic tool on PID responsible for this app pool here is what I got:
Detected possible blocking or leaked critical section at
ONETUTIL!Vifstream::Cin+84 owned by thread
10 in w3wp.exe__APP_NEW__PID__1104__Date__10_23_2010__Time_03_42_14PM__522__Manual Dump.dmp
Impact of this lock
5.88% of threads blocked
(Threads
30
31
32)
The following functions are trying to enter this critical section
ONETUTIL!VprocessGlobals::acquireFPGlobalCriticalSection+b
The following module(s) are involved with this critical section
C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN\ONETUTIL.DLL from
Microsoft Corporation
The following vendors were identified for follow up based on root cause analysis
Microsoft Corporation
Please follow up with the vendors identified above Consider the following approach to determine root cause for this critical section problem:
Enable 'lock checks' in Application Verifier
Download Application Verifier from the following URL:
Microsoft Application Verifier
Enable 'lock checks' for this process by running the following command:
Appverif.exe -enable locks -for w3wp.exe See the following document for more information on Application Verifier:
Testing Applications with AppVerifier
Use a DebugDiag crash rule to monitor the application for exceptions
Warning
9 client connection(s) in w3wp.exe__APP_NEW__PID__1104__Date__10_23_2010__Time_03_42_14PM__522__Manual Dump.dmp have been executing a request for more than
90 seconds.
And Thread 10 points on:
Note - Times include both user mode and kernel mode for each thread
Thread ID:
10
Total CPU Time: 00:00:07.796
Entry Point for Thread: mscorwks!ThreadpoolMgr::intermediateThreadProc
Thread ID:
31
Total CPU Time: 00:00:00.749
Entry Point for Thread: mscorwks!Thread::intermediateThreadProc
Thread ID:
3
Total CPU Time: 00:00:00.500
Entry Point for Thread: w3tp!THREAD_MANAGER::ThreadManagerThread
Thread ID:
8
Total CPU Time: 00:00:00.484
Entry Point for Thread: mscorwks!ThreadpoolMgr::intermediateThreadProc
Thread ID:
30
Total CPU Time: 00:00:00.421
Entry Point for Thread: mscorwks!Thread::intermediateThreadProc
Locked critical section report
Critical Section
ONETUTIL!Vifstream::Cin+84
Lock State
Locked
Lock Count
3
Recursion Count
6
Entry Count
0
Contention Count
5
Spin Count
0
Owner Thread
10
Owner Thread System ID
7228
Thread 10 - System ID 7228
Entry point
mscorwks!ThreadpoolMgr::intermediateThreadProc
Create time
10/23/2010 1:15:52 AM
Time spent in user mode
0 Days 00:00:05.375
Time spent in kernel mode
0 Days 00:00:02.421
This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required.
Function
Source
ntdll!KiFastSystemCallRet
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+13c
mscorwks!Thread::DoAppropriateWait+40
mscorwks!CLREvent::WaitEx+f7
mscorwks!CLREvent::Wait+17
mscorwks!AwareLock::EnterEpilog+8c
mscorwks!AwareLock::Enter+61
mscorwks!JIT_MonEnterWorker_Portable+b3
0x0881a286
mscorwks!SimpleRWLock::LeaveRead+d
mscorwks!Holder<SimpleRWLock *,&SimpleRWLock::AcquireReadLock,&SimpleRWLock::ReleaseReadLock,0,&CompareDefault<SimpleRWLock *>,2>::~Holder<SimpleRWLock *,&SimpleRWLock::AcquireReadLock,&SimpleRWLock::ReleaseReadLock,0,&CompareDefault<SimpleRWLock
*>,2>+2b
0x088167d4
0x0733cf6c
0x064e6fbe
0x064e5b2f
mscorwks!COMToCLRWorkerBody+1de
0x03c97639
mscorwks!COMToCLRWorkerBody+1de
0x03c9556a
mscorwks!COMToCLRWorkerBody+1de
mscorwks!COMToCLRWorkerDebuggerWrapper+37
mscorwks!COMToCLRWorker+52f
0x020aa295
webengine!HttpCompletion::ProcessRequestInManagedCode+1cb
webengine!HttpCompletion::ProcessCompletion+48
webengine!CorThreadPoolWorkitemCallback+1a
mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+9a
mscorwks!ThreadpoolMgr::ExecuteWorkRequest+af
mscorwks!ThreadpoolMgr::WorkerThreadStart+20b
mscorwks!ThreadpoolMgr::intermediateThreadProc+49
kernel32!BaseThreadStart+34
Maybe somebody have any ideas where a problem could exist or what further diagnostic steps should I do ?
Thank You in advance
regards
October 24th, 2010 11:50am
I think you may have resource clashes. We had the same issue and configured our app pools to recycle under certain conditions.
Start here:
http://www.cjvandyk.com/blog/Lists/Posts/Post.aspx?ID=250
I trust that answers your question...
Thanks
C
http://www.cjvandyk.com/blog
Free Windows Admin Tool Kit Click here and download it now
October 25th, 2010 11:30pm
Thanks a lot Cornelius! I will follow up your suggestion and will update with result this thread soon,
Thanks again!
October 26th, 2010 10:43am
Today "Runtime.Error" has been return on browser while connecting to this site portal, in event viewer I found:
Event Type: Error
Event Source: W3CTRS
Event Category: None
Event ID: 2001
Date: 10/27/2010
Time: 9:41:09 AM
User: N/A
Computer: sharepoint-server
Description:
It has taken too long to refresh the W3SVC counters, the stale counters are being used instead.
-------
And before it such warining:
Event Type: Warning
Event Source: ASP.NET 2.0.50727.0
Event Category: Web Event
Event ID: 1309
Date: 10/27/2010
Time: 9:38:57 AM
User: N/A
Computer: sharepoint-server
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 10/27/2010 9:38:57 AM
Event time (UTC): 10/27/2010 1:38:57 PM
Event ID: 7e4066c03e6442b5a3256cef64fa8203
Event sequence: 411
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/1970869955/Root-1-129326307522120095
Trust level: WSS_Minimal
Application Virtual Path: /
Application Path: C:\Inetpub\wwwroot\wss\VirtualDirectories\80\
Machine name: sharepoint-server
Process information:
Process ID: 8860
Process name: w3wp.exe
Account name: domain\apppooluser
Exception information:
Exception type: TypeInitializationException
Exception message: The type initializer for 'Microsoft.SharePoint.CoreResource' threw an exception.
Request information:
Request URL: http://site/pmo/default.aspx
Request path: /pmo/default.aspx
User host address: 172.31.10.10
User: domain\user
Is authenticated: True
Authentication Type: NTLM
Thread account name: domain\apppooluser
Thread information:
Thread ID: 8
Thread account name: domain\apppooluser
Is impersonating: False
Stack trace: at Microsoft.SharePoint.CoreResource.GetString(ResourceGroup rg, String name, Object[] values)
at Microsoft.SharePoint.SPResource.GetString(String name, Object[] values)
at Microsoft.SharePoint.Utilities.SPUtility.GetErrorMessageFromException(Exception e, Boolean renderForEdit, String genericErrorText)
at Microsoft.SharePoint.Utilities.SPUtility.GetErrorMessageFromException(Exception e, Boolean renderForEdit)
at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.ErrorHandler(HttpApplication app)
at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.ErrorAppHandler(Object oSender, EventArgs ea)
at System.Web.HttpApplication.RaiseOnError()
-----------------------------
After recycle app pool site back on track , any ideas ?
Free Windows Admin Tool Kit Click here and download it now
October 27th, 2010 6:06pm
You say your "application" stops responding can you give us more details about this? Is it a site? Some custom application? Does it have dependencies outside SharePoint, etc? Or do you mean the sites in a specific web application are innaccessible?
What is the seen behavior by an end user?
How often is this issue occurring?
Happening for only one web application?
Happening for all sites in the web application?
Custom code or webparts in the afflicted web application/site?
Have you reviewed verbose and informational ULS logging for the timeframes this is occuring?Fred Ellis - MSFT - "Please remember to click Mark as Answer on the post that helps you, and to click Unmark as Answer if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread."
November 2nd, 2010 5:35pm
- Its a WSS 3.0 Sharepoint Portal which consist plenty of sub-sites. When problem occurs users are not able to open main portal website:
http://portal/default.aspx
- End users got TIME OUT information when trying to hit this portal. Recycle App pool fixing issue in a moment
- "Happening for all sites in the web application?" Good question! I didnt never check if access to http://portal/subsite/default.aspx when main portal site doesnt respond.
- "Custom code or webparts in the afflicted web application/site?" hard to find which exactly part is executed while problem occurs. I just got information from users that Sharepoint site is down again.
- "Have you reviewed verbose and informational ULS logging for the timeframes this is occuring?" Could You please point wht ULS logging is?
Thank You Fred for your help !
Free Windows Admin Tool Kit Click here and download it now
November 5th, 2010 4:47pm
The ULS logs can be stored where you configure them to be saved. You can check this path in Central Administration -> Operations -> Diagnostic Logging
Determining what on a page/site is causing problems will take some digging you could open a support case with SharePoint to try to help get to the bottom of this.
If you don't find the answer you need in the forums, and you'd like additional help, please feel free to visit the link below and see if any paid support option available may suit your needs:
http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophone
In the meantime feel free to continue seeking advice here. Other members of the community may have encountered the issue you're seeing, and have a solution to offer.Fred Ellis - MSFT - "Please remember to click Mark as Answer on the post that helps you, and to click Unmark as Answer if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread."
November 10th, 2010 10:06pm
I enabled displaying diagnostic information in web.config. Today problems occur again and here is what I found:
-----------
Server Error in '/' Application.
Could not load file or assembly 'Microsoft.Sharepoint.intl, Version=12.0.0.0, Culture = neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. Access is denied.
Description: An unhandled exception occured during the execution of the current web request.
------------
Maybe You know what does it mean ?
BTW Recycle of App pool solve this issue
Free Windows Admin Tool Kit Click here and download it now
November 17th, 2010 6:05pm
Hmmm could be some account is trying to access the GAC that doesn't have permissions.
http://blogs.msdn.com/b/miah/archive/2008/07/15/specifying-global-assembly-cache-permissions.aspx
It would be helpful to get ULS logs snippets from the timeframe this is occuring to get more details.
Bing search on this error pulled up quite a bit.
http://www.bing.com/search?q=%22Microsoft.Sharepoint.intl%22+%22Access+is+denied%22+%22Could+not+load%22&qs=n&sk=&first=11&FORM=POREFred Ellis - MSFT - "Please remember to click Mark as Answer on the post that helps you, and to click Unmark as Answer if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread."
November 17th, 2010 11:11pm
I am getting this error :
Server Error in '/' Application.
Could not load file or assembly 'Microsoft.Sharepoint.intl, Version=12.0.0.0, Culture = neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. Access is denied.
After the application pool recycles.
Any help is appreciated.
Thanks,
Sam
Free Windows Admin Tool Kit Click here and download it now
August 17th, 2011 7:51pm