Mailbox moves extremely slow and stopping with StalledDueToCI status
I am migrating mailboxes from Exchange 2007 to Exchange 2013.  I have an Exchange 2013 DAG.  Previously I was running Exchange 2013 CU2 and had no issues with migrations, they were migrating fast and had no issues.  After upgrading Exchange 2013 to SP1 I have had very slow migrations and they eventually halt with the StalledDueToCI status.  I was originally doing batch migrations, I dropped it down to a single move request and even migrating a single mailbox from one database to another on the same Exchange 2013 server and I still have the same issue.  I've also tried moving a mailbox to a database that is not part of the DAG and I still have the same issue.  I have the ContentSubmitters group so that's not the issue.  Once it stalls I have to restart the MRS and it kicks off again but then it stalls again with the same error.  ALL of my databases have a "Healthy" status for the content index.  Any help would be appreciated.  Thanks.
April 12th, 2014 1:28am

are you getting any repeated events in the event logs?
Free Windows Admin Tool Kit Click here and download it now
April 12th, 2014 6:45am

These two are repeated:

Log Name:      Application
Source:        MSExchangeFastSearch
Date:          4/13/2014 1:57:13 AM
Event ID:      1009
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Mailbox Server
Description:
The indexing of mailbox database DB14 encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Fast.FastTransientDocumentException: The Content Submission Service returned failure for the document: The Content Submission Service returned failure for the document: The item could not be indexed successfully because one or more of the index partitions was not available for indexing.
Failed to distribute content group to all fragments
.. ---> Microsoft.Exchange.Search.Fast.FastTransientDocumentException: The Content Submission Service returned failure for the document: The item could not be indexed successfully because one or more of the index partitions was not available for indexing.
Failed to distribute content group to all fragments
.
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Search.Fast.FastFeeder.EndSubmitDocument(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.DocumentCompleteCallback(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SearchFeedingController.ExecuteComplete(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeFastSearch" />
    <EventID Qualifiers="32772">1009</EventID>
    <Level>3</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-04-13T05:57:13.000000000Z" />
    <EventRecordID>1240922</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Mailbox Server</Computer>
    <Security />
  </System>
  <EventData>
    <Data>DB14</Data>
    <Data>Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---&gt; Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---&gt; Microsoft.Exchange.Search.Fast.FastTransientDocumentException: The Content Submission Service returned failure for the document: The Content Submission Service returned failure for the document: The item could not be indexed successfully because one or more of the index partitions was not available for indexing.
Failed to distribute content group to all fragments
.. ---&gt; Microsoft.Exchange.Search.Fast.FastTransientDocumentException: The Content Submission Service returned failure for the document: The item could not be indexed successfully because one or more of the index partitions was not available for indexing.
Failed to distribute content group to all fragments
.
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Search.Fast.FastFeeder.EndSubmitDocument(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.DocumentCompleteCallback(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SearchFeedingController.ExecuteComplete(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)</Data>
  </EventData>
</Event>   Log Name:      Application
Source:        MSExchange Mid-Tier Storage
Date:          4/13/2014 2:00:05 AM
Event ID:      6002
Task Category: ResourceHealth
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Mailbox Server
Description:
Ping of mdb '2d6186e1-9e7f-470b-90a1-9958855a44cb' timed out after '00:00:00' minutes.  Last successful ping was at '4/13/2014 6:00:05 AM' UTC.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchange Mid-Tier Storage" />
    <EventID Qualifiers="32768">6002</EventID>
    <Level>3</Level>
    <Task>6</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-04-13T06:00:05.000000000Z" />
    <EventRecordID>1240948</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Mailbox Server</Computer>
    <Security />
  </System>
  <EventData>
    <Data>2d6186e1-9e7f-470b-90a1-9958855a44cb</Data>
    <Data>00:00:00</Data>
    <Data>4/13/2014 6:00:05 AM</Data>
  </EventData>
</Event>


April 14th, 2014 12:32am

I already mentioned that ContentSubmitters is there.  I already disabled all the chimney stuff previously.  No thanks on the third party app.  Anyone else have any suggestions?  Thanks.
Free Windows Admin Tool Kit Click here and download it now
April 14th, 2014 2:49pm

Hi,

I recommend that you run the following command and post the output:

Get-MailboxDatabaseCopyStatus  | ft name, status, contentindexstate,contentindexerrormessage
Thanks.

April 15th, 2014 9:59am

Sure, I've attached the output in a screenshot.  Right now all DBs are running on the same server except DB12, I haven't rebalanced them yet after some reboots except for moving one DB.  Thanks.


Free Windows Admin Tool Kit Click here and download it now
April 15th, 2014 11:05am

Problem solved.  There is a bug in Exchange 2013 SP1 that Microsoft is currently working on.  The temporary workaround is to disable content indexing on the DB while you move your mailboxes by running:

Set-MailboxDatabase DB -IndexEnabled:$False

  • Marked as answer by JRedmond7 Wednesday, April 16, 2014 12:13 AM
April 16th, 2014 12:13am

Works perfectly for me, thank you!
Free Windows Admin Tool Kit Click here and download it now
May 4th, 2015 8:01pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics