Enable Enterprise Features Hung at Initializing
Two days ago, I upgraded the license on our SharePoint 2007 server from Standard to Enterprise. The upgrade completed ok. Afterwards, I selected the option to Enable Enterprise Features on existing sites. This process has been stuck saying Initializing for two days now. I followed the advise of another Microsoft Social thread and restarted the Application Pools but this made no difference. The SharePoint databases contain very, very little information as this is a pilot system running on a TechNet license while we decide on whether to use SharePoint or another product for our ECM. The WFE is running on Server 2008 R2 and the SQL Server is Server 2008 R2 with SQL Server 2008. UPDATE: It appears that search is not functioning currently either, and it was working prior to the Enterprise upgrade. the crawler in the SSP is showing an error. The Event Logs on the WFE have no errors in them. Thanks Richard
August 12th, 2010 4:31pm

Can you check event logs, IIS Logs, and SharePoint Logs and LMK if you find logged events related to the problems you're facing?Tom Molskow SharePoint Architect
Free Windows Admin Tool Kit Click here and download it now
August 12th, 2010 8:08pm

Can you check event logs, IIS Logs, and SharePoint Logs and LMK if you find logged events related to the problems you're facing? Tom Molskow SharePoint Architect Application and System event logs are totally clean. IIS logs show nothing but HTTP GET and POST requests. I've loaded the logs for this week into SharePoint LogViewer, however it's like looking for a needle in a haystack. Do you have anything that could help me narrow the search - Am I looking for a particular log level (Critical, High, Warning) or will the log come from a particular process or category? Thanks Richard
August 13th, 2010 11:40am

I had an interesting thought just a moment ago. I've got two content databases. One called SSPContent_DB and one called MySitesContent_DB. The SSPContent_DB has all the sites it requires created, so I marked the database offline in Application Management and then added the MySitesContent_DB and marked it online so that all the MySites will be created here. I had a thought that the Enterprise Features may not be completing because it can't make the requires changes to the offline database, so in Application Management I tried to mark it online and I recieved this error: "An update conflict has occurred, and you must re-try this action. The object SPContentDatabase Name=VnetSSPContent_DB Parent=SPDatabaseServiceInstance is being updated by VOCERA\rgreen, in the w3wp process, on machine SHAREPOINT. View the tracing log for more information about the conflict." In the tracing log I got the following message repeated about 50-75 times: "The previous instance of the timer job '', id '{32E33507-1E5E-4C7A-B2BF-C8C35A92797F}' for service '{00000000-0000-0000-0000-000000000000}' is still running, so the current instance will be skipped. Consider increasing the interval between jobs."
Free Windows Admin Tool Kit Click here and download it now
August 13th, 2010 12:44pm

OK, at this point I would do this: Kill the old job (you may have to stop the timer service OWSTimer.exe) run IISREST After Reset check the App Pools, Event Logs, etc. (just to be thorough) Bring all SP databases online Retry the upgrade I hope that helps!Tom Molskow SharePoint Architect
August 13th, 2010 3:34pm

OK, at this point I would do this: Kill the old job (you may have to stop the timer service OWSTimer.exe) run IISREST After Reset check the App Pools, Event Logs, etc. (just to be thorough) Bring all SP databases online Retry the upgrade I hope that helps! Tom Molskow SharePoint Architect Thanks for the reply Tom. Just to confirm something before I start this. Can you explain what you mean by kill the old job and retry the upgrade? I've tried iisreset etc previously and when I log back into Central Administration, the upgrade is already running automatically. Thanks Richard
Free Windows Admin Tool Kit Click here and download it now
August 13th, 2010 5:36pm

Hey Richard, The timer service runs independent of IIS - you need to stop and star "Windows SharePoint Services Timer" - the timer service runs all timer jobs for SharePoint. I hope that helps! Tom Molskow SharePoint Architect
August 13th, 2010 5:59pm

Tom, On the WFE, I did the following: Stopped SharePoint Timer search and set startup type to Manual to prevent it auto-restarting. Competed iisreset. Opened Central Administration, and navigated to Application Management > Content Databases. Tried to online the SSPContent_DB database but recieved the same error. An update conflict has occurred, and you must re-try this action. The object SPContentDatabase Name=VnetSSPContent_DB Parent=SPDatabaseServiceInstance is being updated by VOCERA\rgreen, in the w3wp process, on machine SHAREPOINT. View the tracing log for more information about the conflict. What I don't understand is why it reports the database as being updated by VOCERA\rgreen (my domain account), as all of the SharePoint processes are running in their own service accounts (sa-sharepoint in the case of the worker processes). Checking Task Manager, there are no w3wp.exe processes running in my my user context. Could it be worth a shot at taking the SQL database offline to force this mysterious w3wp.exe process to terminate? Thanks Richard
Free Windows Admin Tool Kit Click here and download it now
August 13th, 2010 6:34pm

Yes, that would certainly be worth a try, also, you mentioned you took all these actions on the WFE, are there other SP Servers in your farm - like a seperate application server or a WFE dedicated for crawling? You may want to check those servers as well. But definately, if you have the luxury of stopping the Database server I would do that, and then see if that kills the pernicoius process bug.Tom Molskow SharePoint Architect
August 13th, 2010 6:40pm

Tom, The farm consists of the SQL Server, Exchange for Outbound Email and one WFE which is also acting as the Crawler. This is only a pilot, so more servers will be added to the farm if adopted. Took all of the SQL databases offline including the SharePoint_Config database, the search databases and more importantly the content databases. Brought all of the databases back online again. Restarted the SharePoint machine to totally reset IIS and the timer processes. Tried to onlnie the content database in Application Management > Content Databases but got the same error. Is there a way I can kill the Feature Enabling process/timer job using stsadm so that I get the option to restart it manually? Even though I'm killing processes and restarting things, the status is showing Initializing everytime without me starting the job back up which is what I assume it preventing me from onlining the database? Thanks Richard
Free Windows Admin Tool Kit Click here and download it now
August 13th, 2010 7:07pm

Hi, Been following your threads. typically on my end here when I run into jobs that get stuck, I force them to run using: stsadm -o execadmsvcjobs And I make sure that I run this command on all my servers except the SQL because MOSS isn't in there, and guess what? It always been working. I suggest you do that often if you haven't already been doing that. In fact, that seems like my most used command after every configuration that is farm wide. I hope this really helps you.You've only got one life; help as many people as you can, and enjoy it while it lasts.
August 14th, 2010 11:38pm

Hey newcreationxavier, It's certainly worth a try! VC - have you tried this?Tom Molskow SharePoint Architect
Free Windows Admin Tool Kit Click here and download it now
August 16th, 2010 4:37am

Hi, Been following your threads. typically on my end here when I run into jobs that get stuck, I force them to run using: stsadm -o execadmsvcjobs And I make sure that I run this command on all my servers except the SQL because MOSS isn't in there, and guess what? It always been working. I suggest you do that often if you haven't already been doing that. In fact, that seems like my most used command after every configuration that is farm wide. I hope this really helps you. You've only got one life; help as many people as you can, and enjoy it while it lasts. Thanks for the tip on this command. I ran it on the server and it came back with Opeation Completed Successfully, however the job is still stuck as Initializing and I am still getting the error that the database cannot be brought online because of the w3wp.exe process.
August 16th, 2010 11:51am

He Richard, It never hurts to try the easy stuff! Anyway, to be clear, you brought the problem content database back online before restarting IIS right? OK, let's try the stsadm route, see if you can see the timer job by using stsadm.exe. Here is the command to see the timer jobs: stsadm.exe -o enumssptimerjobs -title <mySSPname> Now, if you can finsd the name of the timer job, here is the command to delete a timer job: stsadm.exe -o deletessptimerjob -title <mytimerjob> -jobid <mytimerjobid> BTW, this can be done through the GUI too - from CA, Operations -> Gloabl Configuration -> Timer Job Status - this displays all the timer jobs From CA, Operations -> Global Configuration -> Timer Job Definitions - this let's you kill the timer job Tom Molskow SharePoint Architect
Free Windows Admin Tool Kit Click here and download it now
August 16th, 2010 3:23pm

Tom, As it stands, the content database is still offline and I'm still getting the error when trying to bring it online. Restarting IIS or the WFE as a whole doesn't make any difference to that currently. The output of the stsadm command is as follows: SSP Timer Job Id="e70d2a21-15c1-4250-9b58-1cbfa504c231" Display Name="Audience Compilation Job" SSP Timer Job Id="f78756a8-bea4-4544-93ea-1d121da97a76" Display Name="User Profile Incremental Import Job" SSP Timer Job Id="c2b87f75-1a61-4a04-bf88-1fba64335184" Display Name="User Profile Change Cleanup Job" SSP Timer Job Id="0b846dad-be1b-4c53-accb-3a9a839d52f5" Display Name="User Profile Full Import Job" SSP Timer Job Id="142b824e-24ae-4f11-8a52-3eca5d473d5a" Display Name="User Profile Change Job" SSP Timer Job Id="3be253e6-9c00-466f-9661-751c5be8cb10" Display Name="Distribution List Import Job" None of these jobs appear releated to the Enterprise Feature Enablement to me. With the Enterprise Features Enablement being an operation under CA and not SSP, is there an stsadm command to list the global timer jobs? Within the CA GUI, under Timer Job Definitions, I can see an Enable Features Job which has a Web Application of N/A and a Last Run Time of N/A.
August 16th, 2010 4:22pm

You could try killing that Enable Features job, just in case. it sounds like it never ran but maybe that's the output it gives for a hung job.Tom Molskow SharePoint Architect
Free Windows Admin Tool Kit Click here and download it now
August 16th, 2010 4:28pm

You could try killing that Enable Features job, just in case. it sounds like it never ran but maybe that's the output it gives for a hung job. Tom Molskow SharePoint Architect Will deleting the job prevent it from coming back if it needs to (AKA Will SharePoint be able to recreate the job as needed)? Thanks Richard
August 16th, 2010 5:00pm

Yes, it's just a timer job, meaning you are only deleting the job, not the feature/solution/ect. that will use the job. Then, using stsadm, you can create the job again. Chances are, with a web application of N/A and last ran of N/A it's never actually been run (so your current production build doesn't require it).Tom Molskow SharePoint Architect
Free Windows Admin Tool Kit Click here and download it now
August 16th, 2010 5:29pm

Ok, I deleted the timer job, then I did iisreset and restart the owstimer.exe process. The good news is that the Enable Enterprise Features job is no longer automatically kicking itself off when I look in CA > Enable Features on Existing Sites. The bad news is that the content database is still reporting that it is in use by a w3wp.exe process. I'm going to restart the box and reset all of the SQL databases as we did before now that the job is no longer running to see if this resets everything, so I will update again shortly.
August 16th, 2010 6:16pm

Took the SQL databases offline, restarted the SharePoint machine and then brought the databases back online. Reset the owstimer.exe service and ran the stsadm -o execadmsvcjobs command to try and flush any kind of queuing but still getting the error trying to bring the database online. Viewing the SPLog, I see two messages - One attempting the topology change and the other for the exception. Updating SPPersistedObject SPContentDatabase Name=VnetSSPContent_DB Parent=SPDatabaseServiceInstance. Version: 12124 Ensure: 0, HashCode: 30369223, Id: a3b04cad-0461-4fdb-954f-a6147e9de3ba, Stack: at Microsoft.SharePoint.Administration.SPPersistedObject.Update() at Microsoft.SharePoint.Administration.SPDatabase.Update() at Microsoft.SharePoint.ApplicationPages.ExistingContentDBPage.BtnSubmit_Click(Object sender, EventArgs e) at System.Web.UI.WebControls.Button.OnClick(EventArgs e) at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest() at System.Web.UI.Page.ProcessRequest(HttpContext context) at ASP._admin_oldcntdb_aspx.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error) at System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData) at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr) at System.Web.HttpRuntime.ProcessRequestNoDemand(HttpWorkerRequest wr) at System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType) UpdatedConcurrencyException: The object SPContentDatabase Name=VnetSSPContent_DB Parent=SPDatabaseServiceInstance was updated by another user. Determine if these changes will conflict, resolve any differences, and reapply the second change. This error may also indicate a programming error caused by obtaining two copies of the same object in a single thread. Previous update information: User: VOCERA\rgreen Process:w3wp Machine:SHAREPOINT Time:February 02, 2010 09:46:48.0000 Current update information: User: VOCERA\rgreen Process:w3wp Machine:SHAREPOINT Time:August 16, 2010 08:36:07.8999 Being nosy, I was just looking at the tables in the VnetSSPContent_DB on the SQL Server and there is a table called dbo.TimerLock with one record which shows SHAREPOINT has a lock from 2010-08-10 13:14:59.107. I manually deleted the row and did an iisreset, however trying to bring the database back online still gives me that same error. I feel like it's getting closer now, and this is just going to be something silly that's got a lock on the DB or something now.
Free Windows Admin Tool Kit Click here and download it now
August 16th, 2010 6:53pm

Hey Richard, A couple tools I use to help decipher SP problems are the SharePoint ULS Log Viewer - http://ulsviewer.codeplex.com/ and the SharePoint Diatgnostic Tools - which is part of the SharePoint Administration Toolkit - which can be download here - http://www.microsoft.com/downloads/en/confirmation.aspx?familyId=cd2d09a7-1159-4d40-be1c-8efab1345381&displayLang=en. These two tools will save you a lot of time looking through logs. So for clarity and to save my eyes, what error is still being reported now?Tom Molskow SharePoint Architect
August 16th, 2010 7:19pm

I have found the source of the lock errors, but the problem still exists - Albeit in a different form now. A while back I was annoyed that the SharePoint_Config database couldn't be renamed to match my naming convention for my other databases so I followed some steps I found online to get this completed so that my config database is now called VnetFarmConfig_DB. This caused me some problems so I reverted back to the SharePoint_Config database. The VnetSSPContent_DB was locked by the alternate config database! I guess we need to start over with this issue. As it stands now I guess I should start over and explain where I'm at. Using VnetFarmConfig_DB as my config database. Search services are now working again. Both content databases are Ready and online. The problem I have now, is that if I select Enable Enterprise Features in CA Operations, it shows Standard. I enter the Enterprise CAL key, and it processes for sometiem before quitting saying that a problem occured. I followed some of the basic tips on the web to restart the owstimer.exe process but it doesn't solve the problem.
Free Windows Admin Tool Kit Click here and download it now
August 16th, 2010 7:30pm

Hi, Still watching these threads from the sidelines, I'll chip in anything I find useful. But when we get to the end of this problem, you will have to blog it well Vocera, and with details and make sure Bing and Google crawls it well for someone in the not-too-far future who might need it as a saving grace, maybe even me. Tom asked for the logs, let see what you've got. Your last log almost broke my eyes, it extended far into the right -----> Cheers!You've only got one life; help as many people as you can, and enjoy it while it lasts.
August 16th, 2010 7:35pm

As I said in my last post, search and various other things broken by the previous config database are now working again (such as search) which is excellent. I think I now now the source of the problem it's just going to be fixing it which is where I need your help :) In CA > Operations > Servers in Farm, I see at the top, it lists the following: Version: 12.0.0.4518 Configuration Database Server: SQLSERVER.x.x Configuration Database Name: SharePoint_Config When I try to perform the Enable Enterprise Features job by entering the Enterprise CAL, I believe that it is trying to write these changes to the SharePoint_Config database, however the config database is actually VnetFarmConfig_DB. I ran SharePoint Products and Technologies Configuration Wizard and that correctly shows the configuration database as VnetFarmConfig_DB as does the DSN registry key. I believe that if I can force CA to show the correct database name then this will allow Enable Enterrpise Features to work. Thanks and sorry to have led us down this wonky and uneven path and my dodgy configuration database history! Richard
Free Windows Admin Tool Kit Click here and download it now
August 16th, 2010 7:50pm

There is a tool called psconfig which might be able to fix this issue, but before trying to use this, I would run the SharePoint Products and Technologies Wizard again. The SPT Wizard is located from Start -> All Programs -> Microsoft Office Server. I can't tell you the exact clicks to use once you run the wizard because I'm not experiencing that problem, but here is a great article about the tool: http://support.microsoft.com/kb/944267 Please LMK how that goes.Tom Molskow SharePoint Architect
August 16th, 2010 8:01pm

I read a TechNet blog that said the error I was getting was because the WFE had cached the old config database parameters and the cache needed clearing, however upon viewing the folder that should store the cache, it was empty. One of the comments on the thread said if it's empty that they resolved the problem by removing the server from the farm and re-adding it. I ran SharePoint Products and Technologies Wizard and selected remove this server from the farm. I ran SharePoint Products and Technologies Wizard for a second time and reconnected to the configuration database. The wizard completed, and after I enabled HTTPS on the CA site as per my design, I was able to see everything in CA correctly again. I brought both of the content databases online and ran the Enable Enterprise Features job. It took less than two minutes to complete - Told you I didn't have much data. As part of removing the server and re-adding the server into the farm, I did somehow loose the indexing role on the WFE, so I had to create a new search database and reconfigure the WFE as an indexer for the SSP, but everything is working perfectly today. Thanks for all of the help. Richard
Free Windows Admin Tool Kit Click here and download it now
August 17th, 2010 11:15am

Hooray Richard!!! I'm glad you got it going and I'm gald I could help!!! Tom Tom Molskow SharePoint Architect
August 17th, 2010 3:10pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics