I have  a two node DAG I have successfully updated the database copies no bad copies. The content catalog index on all copies shows as failed. Update-DatabaseCopy -catalogonly does not work. /have deleted thedatabase abd remounted it shows healthy and returns to failed catalog soon after have strict targets client needs HA. cant failover the database as a result. The content database is failed even on servers with one copy only on DAG member server. Help

If you need to move the databases while the catalogs are failed you can use the Skip Client Experience Checks parameter.

I need the content index to reflect healthy this is a client site and the infrastructure will be undergoing quality assurance how do I reset the index like it used to be done in 2010.

Even after moving all nodes from the DAG the database index still reflects failed

• Edited by Bright Murewanhema Monday, December 03, 2012 9:23 AM

I need the content index to reflect healthy this is a client site and the infrastructure will be undergoing quality assurance how do I reset the index like it used to be done in 2010.

Even after moving all nodes from the DAG the database index still reflects failed

• Edited by Bright Murewanhema Monday, December 03, 2012 9:23 AM

Hi BrightM,

Please run the Get-MailboxDatabaseCopyStatus databasename | fl *index* to check the detailed information of the error.

If possible, please create a new database and put it in a dedicated disk to check the status.

If you have any feedback on our support, please click here

The output I get is as follows:

Content Index State            :     Failed

ContentIndexErrorMessage  :     An internal error occurred for the database or its index.

I tried t create a blank database but I it went healthy and then went failed soon after.

Hi BrightM,

Two suggestions:

1. Restart the Microsoft Exchange Search Indexer service on both member servers.

2. Using Troubleshoot-CI.ps1 script.

.\Troubleshoot-CI.ps1 -Database "database name" -Action DetectAndResolve

Thanks,

Da Xu

Ok I have reduced the problem to this

I have a Mapi network replication allowed 192.168.101.0/24 Status UP for subnet and interfaces

Ialso have  replication network replication allowed 10.10.10.0/30 Status Partitioned for subnet and Unreachable for interfaces

On the replication network the transmission is good for sending and not receiving.

How can I set the relevant static routes to resolve this problem.

Hi BrightM,

If the two replication networks are on same subnet (10.10.10.0/30), there is no need to add static route.

Please make sure the replication network (10.10.10.0/30) has no gateway, lmhosts lookup disabled, no netbios, no dns servers, and is setup not to register the connection address in DNS.

Also, please check your firewall setting.

Da Xu

Da Xu

This is correct I have managed to do this and my replication network is up. I still have an issue with the content index on the active copy which still reflects failed.

Regards

Bright

Hi

.\Troubleshoot-CI.ps1 -Database "database name" -Action DetectAndResolve

Tried this I get an error The troubleshooter failed with error:System.InvalidOperationException: Couldn't read registry key:Software\Microsoft\ExchangeServer\V15\ContentIndex\CatalogHealth\{GUID} At E:\Exchange Server 15\Scripts\CITSLibrary.ps1

Bright

Hi Bright,

Please try to manually rebuild the full-text index catalog:

1. Stop the Microsoft Exchange Search Service on the Exchange server on which the problematic database is mounted.

2. Delete the full-text index catalog directory

3. Start the Microsoft Exchange Search Service

Then, please reseed the catalog of the database copy again.

Thanks,

Da Xu

• Marked as answer by Da XuMicrosoft employee, Moderator Monday, December 17, 2012 2:58 AM

Hi Bright,

Please try to manually rebuild the full-text index catalog:

1. Stop the Microsoft Exchange Search Service on the Exchange server on which the problematic database is mounted.

2. Delete the full-text index catalog directory

3. Start the Microsoft Exchange Search Service

Then, please reseed the catalog of the database copy again.

Thanks,

Da Xu

• Marked as answer by Da XuMicrosoft employee, Moderator Monday, December 17, 2012 2:58 AM

Hi Bright,

Is there any update about this issue?

Thanks,

Da Xu

Still no resolution.

1. Where is the content index directory for Exchange 2013 databases.

Hi,

Check C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database....\

Following also will help you to understand.

How to Manually Rebuild the Full-Text Index Catalog
http://www.msexchangegeek.com/2009/09/09/how-to-manually-rebuild-the-full-text-index-catalog/

How to Rebuild the Full-Text Index Catalog
http://technet.microsoft.com/en-us/library/aa995966(v=exchg.80).aspx

ecsword

I wanted to follwup on this information.

For Exchange 2013, the service name is Microsoft Exchange Search Host Controller that holds the locks on the indexing files.

Also, I have this issue on all databases in two separate forests and cannot get the databases to failover, copy, or have a healthy content index status.

When I follow the steps above I continue to get the failed content index status.

Any additional information on how to resolve this issue would be great.

If I need to start a new thread please let me know.

I believe that the issue is that the location of

HKLM\Software\Microsoft\ExchangeServer\V15\ContentIndex\CatalogHealth\{GUID}

has moved to this location in exchange 2013

HKLM\Software\Microsoft\ExchangeServer\V15\Search\IndexStatus\{GUID}

The issue is that Troubleshoot-CI.ps1 calls upon CITSLibrary.ps1 and CITSLibrary.ps1 is looking in the registry for the old location. Correct me if I'm wrong but I dont think Microsoft got around to fixing the scripts. I looked at CITSLibrary.ps1 briefly but I dont know PS that well. Maybe someone can help us out here.

Hi Jared

Move-ActiveMailBoxDatabase -SkipClientExperienceChecks

The move works but I am unable to connect to the databases after the move and the indexes are still failed.

Regards

Bright

• Edited by Bright Murewanhema Monday, January 14, 2013 1:44 PM

Hi Jared

Move-ActiveMailBoxDatabase -SkipClientExperienceChecks

The move works but I am unable to connect to the databases after the move and the indexes are still failed.

Regards

Bright

• Edited by Bright Murewanhema Monday, January 14, 2013 1:44 PM

Hi  Ecsworld

Ok these articles seem to refer to older versions of Exchange. 15 does not actually create the Catalog{GUID} folder. I don't know if this is the new design.

Regards

Bright

• Edited by Bright Murewanhema Monday, January 14, 2013 1:14 PM

Hi  Ecsworld

Ok these articles seem to refer to older versions of Exchange. 15 does not actually create the Catalog{GUID} folder. I don't know if this is the new design.

Regards

Bright

• Edited by Bright Murewanhema Monday, January 14, 2013 1:14 PM

Hi all

Thanks for all your helpful responses to date. Some additional clarification on the problem. The content index is on the active database and the index remains failed even when the database is removed from the DAG or if the DAG is deleted. And the server remains standalone. Exchange 2013/Windows 2012.

Regards

Bright

• Edited by Bright Murewanhema Monday, January 14, 2013 1:48 PM

Hi all

Thanks for all your helpful responses to date. Some additional clarification on the problem. The content index is on the active database and the index remains failed even when the database is removed from the DAG or if the DAG is deleted. And the server remains standalone. Exchange 2013/Windows 2012.

Regards

Bright

• Edited by Bright Murewanhema Monday, January 14, 2013 1:48 PM

Hi Bright and others running into this issue,

I am also experiencing the same behavior and am wondering if you are all running SCCM on your network?  Last night I re-installed all of my systems and was able to replicate until SCCM found the systems.  I'm currently working with the SCCM Admin to exclude the Exchange servers and will be re-installing once that has been completed.

If someone can test this theory before I get to it please let me know.

You just might  have something here let me try and remove sccm client agent from the dag members and see whether this might be the problem.

As a final and sad followup, I was able to have the SCCM server down for two days this week and re-installed the Exchange system.  Although I could manually replicate using "update-mailboxdatabasecopy db01\mbox1 -deleteexistingfiles" with out error after suspending the database, I was unable to get the content index state to "Healthy".

For the time being, since we cannot get the replication to be healthy we have decided not to move forward with Exchange 2013 in this domain.  We are going to wait for about 6 months and try again.  Hopefully some of the weir issues we have seen will be taken care of by then.

Let me know how it works out for you all.

Finally Solved

Got assistance from Microsoft Support. It seems there is a bug in the RTM its looking for a AD Group that does not exist will probably be fixed later. The fix is:

Create a new AD group named ContentSubmitters and grant Admistrators and NetworkService full access to it

Thereafter restart "Microsoft Exchange Search" & "Microsoft Exchange Search Host Controller" services

This worked for me and the index went from crawling to healthy on all databasecopies.

Thanks for all your help.

• Marked as answer by Bright Murewanhema Monday, January 28, 2013 10:37 AM

Finally Solved

Got assistance from Microsoft Support. It seems there is a bug in the RTM its looking for a AD Group that does not exist will probably be fixed later. The fix is:

Create a new AD group named ContentSubmitters and grant Admistrators and NetworkService full access to it

Thereafter restart "Microsoft Exchange Search" & "Microsoft Exchange Search Host Controller" services

This worked for me and the index went from crawling to healthy on all databasecopies.

Thanks for all your help.

• Marked as answer by Bright Murewanhema Monday, January 28, 2013 10:37 AM

Hi Bright,

When you say "grant Administrators and NetworkService full access to it", what object is it?  Is this the database and log folders?

Also do you add ContentSubmitters group to any object on the mailbox server?

Thank you.

When you create the active directory security group called ContentSubmitters right click the group so created and properties security tab add those two groups and give them full control to the group.

Ok - I've just done this is my POC environment (with CU1 applied) and it worked. My Journal Database now has a Healthy Status - What's going on, why are we having to create these groups and apply the relevant permissions.

Not very happy that these issues are still occurring post CU1!

Please advise if this was a fix in Exchange 2013 CU1 or there is something I'm missing...

Ryan

Thanks for passing that on!  It seemed to do the trick.

• Edited by Robert Tuck Friday, August 02, 2013 4:56 PM

Thanks for passing that on!  It seemed to do the trick.

• Edited by Robert Tuck Friday, August 02, 2013 4:56 PM

Hello,

   I've been following this due to the exact issue.   However applied the ContentSubmitters group to no avail.   Are there any other solutions available?

  We have 2012 standard.  Two servers, one for DB's and one for CAS.   The databases are intermittent, one will fail today a different one will fail tomorrow.  A reboot always fixes the issue for about an hour.

Please share any ideas!  Thanks!

Hello,

We're having this same issue. I've tried adding the "ContentSubmitters" AD group to no avail. The "Microsoft Exchange Search Host Controller" service appears to be disabled on our server. Is this by design?

We're running Exchange 2013 CU2 on a W12 server.

Thanks!

Hello,

We're having this same issue. I've tried adding the "ContentSubmitters" AD group to no avail. The "Microsoft Exchange Search Host Controller" service appears to be disabled on our server. Is this by design?

We're running Exchange 2013 CU2 on a W12 server.

Thanks!

The service should be set to startup type Automatic on a Multi-role Exchange 2013 server.

I was having the same issue. The NIC's configured in my DAG were bound in the wrong order so my Cluster Heartbeat network was first rather than my Exchange network. To resolve the issue I had to open my Network and Sharing Center from Control Panel. Click "Change Adapter Settings". Hit the "Alt" key to bring up the "File, Edit, View, Tools, Advanced, Help Menu. Click on "Advanced" and select "Advanced Settings". Set the correct Nic order. After this I restarted the "Microsoft Exchange Search" and the "Microsoft Exchange Search Host Controller" services from the services applet.  Now my content index shows healthy for my DAG.

Hope this helps.

First off Bright, thank you soo much for this info.  MS can't seem to pull their head out of their posteriors long enough to actually look in to issues like this.   

Second, this is still an issue with SP1.  Unbelievable. 

I believe that the issue is that the location of

HKLM\Software\Microsoft\ExchangeServer\V15\ContentIndex\CatalogHealth\{GUID}

has moved to this location in exchange 2013

HKLM\Software\Microsoft\ExchangeServer\V15\Search\IndexStatus\{GUID}

The issue is that Troubleshoot-CI.ps1 calls upon CITSLibrary.ps1 and CITSLibrary.ps1 is looking in the registry for the old location. Correct me if I'm wrong but I dont think Microsoft got around to fixing the scripts. I looked at CITSLibrary.ps1 briefly but I dont know PS that well. Maybe someone can help us out here.

Correct - The Troubleshoot-CI script doesn't work on 2013.

BTW, everyone else, please see this relevant KB:

KB 2807668

Content Index status of all or most of the mailbox databases in the environment shows "Failed"

 Print  Email

Hi Bright,

not sure how you got to create a AD group and add Admistrators and NetworkService to it as members.

my DC won't allow you to add those, just Domain Admins.

Great info, one question I have is, whom do you put into the security group ContentSubmitters?

Thanks 

No, I think he meant to give them "full access" to the AD object, not make them members of the group.

Still, is this supposed to be a "Global" group, "Universal?"

Does it need to reside where the other Exchange groups reside, or can it live anywhere in AD?