InternalCatalogException in Reporting Services 2008 - no reports rendering
Hi all, We've had an unexpected outage on two of our reporting services server overnight and we're hoping someone can point us in the right direction. We have a Windows Server 2008 machine running about 5 instances of reporting services in standalone, non-IIS mode. Two of these instances have started hitting exceptions when rendering any reports. I include an excerpt from the log below. The first evidence of a problem in the log is an InternalCatalogException, of which I can find many unrelated hits when searching the forums so I don't think this is specific enough to trace. I do note a reference to "segmentedChunkStore" in the text which seems like a better clue. I've eliminated the idea of a machine/OS issue due to several instances not affected by this problem. There is some possibly related info which may help: -This machine is a HyperV VM which has been moved between hosts within the last few weeks. In the process, all Report Server databases were backed up and restored at the new site. - It uses a Custom Authentication extension authored by us. - Within the last few days, users of three RS instances advised that a role permission they had prior to the move (Consume Reports Task) was no longer granted. I had to re-grant this to the role. The two RS instances now experiencing the InternalCatalogException issue raised an error at the time about not being able to find a group when I granted the task. I added the group, granted again, and it granted without error so I thought nothing more of it. - These two RS instances have been working fine since the HyperV host move right up until last night. - The machine had several Windows Updates queued up which were installed last night. Since completion of the updates and subsequent reboot, these two Reporting Services instances now raise this InternalCatalogException error when trying to render any reports from Report Manager, Report Server, or previous from Report Viewer. At this point, I'm suspecting some sort of data corruption has occurred in the ReportServer databases, initially causing the loss of the Consume Reports permission and, after a reboot, these exeptions. But any suggestions on how to troubleshoot further greatly appreciated. Here is the excerpt from the log. library!ReportServer_0-1!8d4!09/19/2012-20:55:53:: Call to GetItemTypeAction(/Risk Manager/Dashboard Module/Medsys/IncConseqCountYTD). library!ReportServer_0-1!12f0!09/19/2012-20:55:53:: Call to GetItemTypeAction(/Risk Manager/Dashboard Module/Medsys/IncCount13Months). library!ReportServer_0-1!f98!09/19/2012-20:55:53:: Call to GetItemTypeAction(/Risk Manager/Dashboard Module/Medsys/my30DayActionList). library!ReportServer_0-1!10c0!09/19/2012-20:55:53:: Call to GetItemTypeAction(/Risk Manager/Dashboard Module/Medsys/org30DayStatusPanel). library!ReportServer_0-1!830!09/19/2012-20:55:54:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/org30DayStatusPanel') library!ReportServer_0-1!830!09/19/2012-20:55:54:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore; Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!830!09/19/2012-20:55:55:: i INFO: Exception InternalCatalogException dumped to: C:\Program Files\Microsoft SQL Server\MSRS10.TESTIMPACRS\Reporting Services\Logfiles flags= ReferencedMemory, AllThreads, SendToWatson chunks!ReportServer_0-1!830!09/19/2012-20:55:55:: w WARN: Rolling back shared chunk transaction for snapshot '49663ae9-6692-4ea2-8a5d-210de4729653', Permanent=False. library!ReportServer_0-1!830!09/19/2012-20:55:55:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!ff8!09/19/2012-20:55:55:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/my30DayStatusPanel') library!ReportServer_0-1!8d4!09/19/2012-20:55:55:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/triCountPanel') library!ReportServer_0-1!8d4!09/19/2012-20:55:55:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore; Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!ff8!09/19/2012-20:55:55:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore; Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. chunks!ReportServer_0-1!ff8!09/19/2012-20:55:55:: w WARN: Rolling back shared chunk transaction for snapshot '68d6456c-1ed0-46eb-8bec-37b04f94065a', Permanent=False. library!ReportServer_0-1!ff8!09/19/2012-20:55:55:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!8d4!09/19/2012-20:55:56:: i INFO: Exception InternalCatalogException dumped to: C:\Program Files\Microsoft SQL Server\MSRS10.TESTIMPACRS\Reporting Services\Logfiles flags= ReferencedMemory, AllThreads, SendToWatson chunks!ReportServer_0-1!8d4!09/19/2012-20:55:56:: w WARN: Rolling back shared chunk transaction for snapshot '464d6ec5-f256-4633-97d9-792c700931a0', Permanent=False. library!ReportServer_0-1!8d4!09/19/2012-20:55:56:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!b90!09/19/2012-20:55:56:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/my30DayActionList') library!ReportServer_0-1!b90!09/19/2012-20:55:56:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore; Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!1598!09/19/2012-20:55:57:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/IncCount13Months') library!ReportServer_0-1!b90!09/19/2012-20:55:57:: i INFO: Exception InternalCatalogException dumped to: C:\Program Files\Microsoft SQL Server\MSRS10.TESTIMPACRS\Reporting Services\Logfiles flags= ReferencedMemory, AllThreads, SendToWatson chunks!ReportServer_0-1!b90!09/19/2012-20:55:57:: w WARN: Rolling back shared chunk transaction for snapshot '94bb87a7-bece-454e-b418-78b18b9eacb1', Permanent=False. library!ReportServer_0-1!b90!09/19/2012-20:55:57:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!8d4!09/19/2012-20:55:57:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/IncConseqCountYTD') library!ReportServer_0-1!1598!09/19/2012-20:55:57:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore; Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!8d4!09/19/2012-20:55:58:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore; Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. chunks!ReportServer_0-1!8d4!09/19/2012-20:55:58:: w WARN: Rolling back shared chunk transaction for snapshot 'e254656f-6c22-49e6-b473-5ed986e0a730', Permanent=False. library!ReportServer_0-1!8d4!09/19/2012-20:55:58:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!ReportServer_0-1!1598!09/19/2012-20:55:58:: i INFO: Exception InternalCatalogException dumped to: C:\Program Files\Microsoft SQL Server\MSRS10.TESTIMPACRS\Reporting Services\Logfiles flags= ReferencedMemory, AllThreads, SendToWatson chunks!ReportServer_0-1!1598!09/19/2012-20:55:58:: w WARN: Rolling back shared chunk transaction for snapshot 'e15cc0b9-c8b1-446a-a3bf-da2ceba6273d', Permanent=False. library!ReportServer_0-1!1598!09/19/2012-20:55:58:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. library!WindowsService_0!14a0!09/19/2012-21:01:07:: i INFO: Call to CleanBatch() library!WindowsService_0!14a0!09/19/2012-21:01:13:: i INFO: Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings. Regards, Michael
September 19th, 2012 7:14am

Okay the issue is resolved. The original cause appears to be that the Reporting Services instance is a TEST clone of the LIVE instance that was renamed. This was done in order to copy the reports, model and folder structure of the live Reporting Services server into something we can test with. Referring to notes online from Microsoft, they stipulate that the Reporting Services database should not be renamed like this. One of the reasons being that it includes internal references to the original name, which become invalid after the database is renamed. I now recall having a similar issue over a year ago (although it wasn't as difficult to diagnose then) - and we followed a process to fix which I've documented below. I am not 100% why this old issue has re-occurred, however I note that the Reporting Services server had SQL Server 2008 Service Pack 3 installed very early yesterday morning so I suspect that this is the likely cause. If it occurs again, we can just follow this process to fix it: Steps to fix Diagnosed by switching on verbose tracing in ReportingServicesService.exe.config file, set to 4. <switches> <add name="DefaultTraceSwitch" value="4" /> </switches> <RStrace> ... <add name="Components" value="all:4" /> </RStrace> And found the following in the logfile C:\Program Files\Microsoft SQL Server\MSRS10.[your instance name]\Reporting Services\LogFiles ... chunks!ReportServer_0-2!1dc!09/20/2012-13:36:00:: v VERBOSE: Open Segmented Chunk (a74f44e4-6b2e-458c-aaff-cd1c77434d96, False, CompiledDefinition, '9', Update) chunks!ReportServer_0-2!1dc!09/20/2012-13:36:00:: v VERBOSE: Segmented Chunk 'CompiledDefinition' was not found. library!ReportServer_0-2!1dc!09/20/2012-13:36:00:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore; Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. ... Source of the problem is the report server is writing to a different (wrong) tempdb when preparing the job output, then attempting to read from the correct tempdb and therefore failing to find the temp data it prepared. Workaround is to modify the stored procedures that reference the wrong tempdb as follows: In SSMS: - Right-click the report server database Tasks - Generate Scripts - Tick Stored Procedures - Hit Select All - Script to New Query Window - Replace CREATE PROC with ALTER PROC (note one proc has a double space e.g. CREATE PROC) - Replace [Your old tempdb name] with [your new tempdb name] - stop the SSRS instance - execute the script - There is one trigger that needs fixing too (Schedule_UpdateExpiration), I did that by manually editing from SSMS to correct the tempdb name. - Restart the SSRS service - Remember to switch off the verbose tracing if you had it switched on. You're in business. I should add that my original suspicions about this issue being related to the move to a new HyperV host, and/or the errors I got when applying permissions seem to be unfounded. I should listen to my own advice and avoid jumping to conclusions! Michael
Free Windows Admin Tool Kit Click here and download it now
September 19th, 2012 11:19pm

Hi Michael, I am glad to hear that your issue have been solved. Thank you for your sharing. Regards, Charlie Liao
September 20th, 2012 1:47am

Hi Michael, I am glad to hear that your issue have been solved. Thank you for your sharing. Regards, Charlie Liao
Free Windows Admin Tool Kit Click here and download it now
September 21st, 2012 7:31am

Awesome!! Run into this hole some months ago and have not been able to solve it til now. To make things easier for anyone coming across this: - to find out what is the name of the wrong temp db, have a look at stored procedure TempChunkExists, in my case it is ReportServerTempDB As a side note: unbelievable Microsoft hardcoding db names into its code!!!!!!!!!!!
September 28th, 2012 3:40am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics