We recently had an issue with our Crawler hanging and traced it back to the indexing processes. After some attempts to resolve with Premier support, they eventually recommended we rebuild the SSA, which we did and we're now back in action.

While looking into this I noticed in we have some performance monitor counters for search "indexing", "index router", "journal" and "journal shipper". In our case we were queuing up on the "PendingDocuments" counter of the "Search IndexRouter".

My question is, what are these 4 processes, what is the information flow between them, is anyone aware of any documentation out there that covers the search index process at this level of detail?

Hello,
 
Thank you for your question.

We are currently looking into this issue and will give you an update as soon as possible.
 
Thank you for your understanding and support.

Victoria

Forum Support

Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

Hi,

As my understanding,  the "index router" might be a component between content processing and the indexing core. The processes about such detailed information are quite limited. Here are some related articles, and hope they are useful for you.
Search Architecture with SharePoint 2013
https://searchunleashed.wordpress.com/2013/12/13/271/
PPT: Discussion of Search internals
http://www.google.com.hk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&cad=rja&uact=8&ved=0CDkQFjAE&url=%68%74%74%70%3a%2f%2f%76%69%64%65%6f%2e%63%68%39%2e%6d%73%2f%73%65%73%73%69%6f%6e%73%2f%73%70%63%2f%32%30%31%34%2f%53%50%43%33%37%35%5f%50%65%6e%64%65%72%67%72%61%73%73%2e%70%70%74%78&ei=6irKVObsKoG5oQSAmIH4DQ&usg=AFQjCNG_6oVz0ngniX36q3z87zaoJfMLAg&bvm=bv.84607526,d.dGY
Manage the index component in SharePoint Server 2013
https://technet.microsoft.com/en-us/library/jj862355(v=office.15).aspx
Technical diagrams for SharePoint 2013
https://technet.microsoft.com/en-us/library/cc263199(v=office.15).aspx

Thanks,
Reken Liu

Unfortunately, I don't think you'll find in-depth information about this topic out there yet.  It's a bit difficult to explain information flow between these 4 modules and how to troubleshoot them without having to go into much more detail of Content Processing and Indexing internals.

To address your question, these are not separate processes but are all modules within an Index Component.  Specifically, each primary Index Component in a Partition has an Index Router running, which receives groups of documents from Content Processing Component.  This activity can be tracked from "Search Content Router" perfmon category.  Using a procedure managed by the Admin Component, each Index Component attempts to de-queue document groups from its Index Router(in memory) to its Journal (on disk).  If this de-queuing was not happening on your system, it would suggest some problems writing to the Journal.  Some typical offenders would be index corruption, anti-virus, etc.

Some notes:

 - You can think of a Journal as sort of an RDBMS Transaction log that can be replayed in case of a failure. 

 - Journal Shipper - Journal entries are shipped to all replicas in an index cell. 

 - Update Groups -  All document properties(managed properties) are split into Update Groups to ensure that an update only impacts a specific Group and does not trigger reload for all managed properties.

Hope this helps.