Hi, I'm running a set of SSIS packages from command line DTExec. Every week or so (sometimes more frequently), I get an APPCRASH where DTExec fails and takes down the .NET Runtime dll. Server Manager also crashes and I can no longer run any DTexec commands without rebooting the server. I have this happening on 2 different servers. There are 4 DTSX packages that are scripted to run every morning (6am on one server and 7am on another): (script is a batch file that calls each dtexec if the previous one is successful) 1. Loads data from a 64-bit oracle database, sorts it, saves it to MSSQL Server database. 2. Loads data from MSSQL Server (loaded in step 1), does lookup and slowly changing dimensions, saves the results back to MSSQL. 3. Loads data from MSSQL, lookup SCD from (2) adds to data and transforms datatypes, saves back to MSSQL. 4. Process an SSAS cube. Notes: Step 4 was added later, so it's something with 1-3. There is no custom code, it is all built in functions. When it happens: ----------------- Most of the event logs are recorded as Step 1 is initiated (the very first thing in each package is to send an Email to user staying "started", which doesn't happen). From this, I figure the DTexec has actually crashed sometime before step 1 or from the previous run, but doesn't really crash until sometime later. I have been on the server (via Remote Desktop) after step 3 has finished and Management Console will randomly fail, which records the below 2 events and prevents any DTExec from running. I cannot recreate this by running again, and it takes up to a week before the problem will happen again. In addition, it is random between my 2 sets of servers (QA and DEV) which run against the exact same input data. Potential problem spots: ------------------------ - I read somewhere that the Sort steps inside SSIS packages are not good to do, I have several of them. - Could be that the servers are VM machines and SSIS might not like that? Events: --------- It creates Event ID 1000, and 1023 at the same time... described below: Log Name: Application Source: Application Error Date: 1/28/2011 9:39:38 AM Event ID: 1000 Task Category: (100) Level: Error Keywords: Classic User: N/A Computer: INLBSAS02QA-DM Description: Faulting application DTExec.exe, version 2007.100.4000.0, time stamp 0x4c92e8ab, faulting module mscorwks.dll, version 2.0.50727.4206, time stamp 0x4bf4c181, exception code 0xc0000005, fault offset 0x0000000000180b4e, process id 0x%9, application start time 0x%10. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Level>2</Level> <Task>100</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2011-01-28T14:39:38.000Z" /> <EventRecordID>50541</EventRecordID> <Channel>Application</Channel> <Computer>INLBSAS02QA-DM.ad.bbm.ca</Computer> <Security /> </System> <EventData> <Data>DTExec.exe</Data> <Data>2007.100.4000.0</Data> <Data>4c92e8ab</Data> <Data>mscorwks.dll</Data> <Data>2.0.50727.4206</Data> <Data>4bf4c181</Data> <Data>c0000005</Data> <Data>0000000000180b4e</Data> </EventData> </Event> Log Name: Application Source: .NET Runtime Date: 1/28/2011 9:39:38 AM Event ID: 1023 Task Category: None Level: Error Keywords: Classic User: N/A Computer: INLBSAS02QA-DM.ad.bbm.ca Description: .NET Runtime version 2.0.50727.4206 - Fatal Execution Engine Error (000007FEF9C665EA) (80131506) Servers Are VM machines: ---------------------------- Windows Server 2008 Standard 6.0.6002 SP2 64-bit Dual Core AMD Processor 275, 2195 Mhz. Memory: 4.00GB From the dump files: --------------------- Windows NT Version 6.0 Build: 6002 Service Pack 2 Product (0x7): Windows Server (R) 2008 Standard Edition: ServerStandard BuildString: 6002.18267.amd64fre.vistasp2_gdr.100608-0458 Flavor: Multiprocessor Free Architecture: X64 LCID: 1033

On the surface it looks like there is some sort of a conflict on the machine. At this stage I advocate to continue with the discovery phase by enabling extensive logging from within the package to capture as much information as possible. Then please post your findings.Arthur My Blog

I'll enable it, but a quick question... My packages all finish successfully, do you think will the extensive logging will cause it to fail instead of working?

The extensive logging (I would use the logging to file) may add a little overhead, but ideally all packages should have at least some basic logging set, but not necessarily enabled by default. Please let me know if I have clarified your concern.Arthur My Blog

I have basic logging already enabled. It gives me log files with OnInformation, OnPreValidate, OnPostValidate, OnPreExecute, OnPostExecute, Diagnostic message types. I have error files that are empty. I've read through them and can't see anything special (1.6 Mb worth). I have submitted my crash reports to MS, but I thought I would see if anyone else has seen this. Here is a sample of one of the log files: #Fields: event,computer,operator,source,sourceid,executionid,starttime,endtime,datacode,databytes,message PackageStart,INLBSAS02QA-DM,BBM\dpayne,PPM_HOME_LIFECYCLE_F_i,{EDEEE080-20FD-4E24-BAAE-04347E912F9D},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:06:59 AM,1/31/2011 6:06:59 AM,0,0x,Beginning of package execution. OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Set up Auditing Metadata,{11BA3C4B-7FD7-45FD-8D89-1292DEA19E07},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:06:59 AM,1/31/2011 6:06:59 AM,0,0x,(null) OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Insert Audit Job Record,{67D6C162-8E89-4196-8F88-E7E35F6001FF},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:06:59 AM,1/31/2011 6:06:59 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Insert Audit Job Record,{67D6C162-8E89-4196-8F88-E7E35F6001FF},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:06:59 AM,1/31/2011 6:06:59 AM,0,0x,(null) OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Get Initial RowCount,{38E3CE8D-115C-4507-90B6-C5847390CBC6},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:06:59 AM,1/31/2011 6:06:59 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Get Initial RowCount,{38E3CE8D-115C-4507-90B6-C5847390CBC6},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:07:16 AM,1/31/2011 6:07:16 AM,0,0x,(null) OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Insert Audit Table Record,{70BDFD41-DA47-4EBF-A6B6-60D936617C82},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:07:16 AM,1/31/2011 6:07:16 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Insert Audit Table Record,{70BDFD41-DA47-4EBF-A6B6-60D936617C82},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:07:16 AM,1/31/2011 6:07:16 AM,0,0x,(null) OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Insert Audit Dim Record,{495214D6-ECAB-497C-8C9C-BC7B529556E7},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:07:16 AM,1/31/2011 6:07:16 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Insert Audit Dim Record,{495214D6-ECAB-497C-8C9C-BC7B529556E7},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:07:16 AM,1/31/2011 6:07:16 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Set up Auditing Metadata,{11BA3C4B-7FD7-45FD-8D89-1292DEA19E07},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:07:16 AM,1/31/2011 6:07:16 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Get last date,{b32cdac7-ef98-4518-84eb-f7764e0db2c6},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:07:17 AM,1/31/2011 6:07:17 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Get distinct dates,{2547FB29-322D-4661-9AD2-51208B33FD82},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:07:17 AM,1/31/2011 6:07:17 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Create Home Life Cycle for this date,{258DD1C8-EFA0-427A-93C7-5783DC371EC4},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:04 AM,1/31/2011 6:08:04 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Foreach Loop Container,{6077258F-5F34-4557-A31F-AB3ABF74D3B9},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:04 AM,1/31/2011 6:08:04 AM,0,0x,(null) OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Update Audit Metadata,{207CEBCF-2A68-4CD1-B8CA-B290904191FB},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:04 AM,1/31/2011 6:08:04 AM,0,0x,(null) OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Get Final Row Count,{204EDCB0-3081-4A3A-B224-4336F3DF27F2},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:04 AM,1/31/2011 6:08:04 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Get Final Row Count,{204EDCB0-3081-4A3A-B224-4336F3DF27F2},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:07 AM,1/31/2011 6:08:07 AM,0,0x,(null) OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Update Audit Table Record,{BC951B4F-FC95-4FC6-9AB3-8B74E81A5DF3},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:07 AM,1/31/2011 6:08:07 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Update Audit Table Record,{BC951B4F-FC95-4FC6-9AB3-8B74E81A5DF3},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:07 AM,1/31/2011 6:08:07 AM,0,0x,(null) OnPreExecute,INLBSAS02QA-DM,BBM\dpayne,Update Audit Job Record,{4E5FFDD1-ED90-4366-8725-B9182B6031E1},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:07 AM,1/31/2011 6:08:07 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Update Audit Job Record,{4E5FFDD1-ED90-4366-8725-B9182B6031E1},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:07 AM,1/31/2011 6:08:07 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,Update Audit Metadata,{207CEBCF-2A68-4CD1-B8CA-B290904191FB},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:07 AM,1/31/2011 6:08:07 AM,0,0x,(null) OnPostExecute,INLBSAS02QA-DM,BBM\dpayne,PPM_HOME_LIFECYCLE_F_i,{EDEEE080-20FD-4E24-BAAE-04347E912F9D},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:07 AM,1/31/2011 6:08:07 AM,0,0x,(null) PackageEnd,INLBSAS02QA-DM,BBM\dpayne,PPM_HOME_LIFECYCLE_F_i,{EDEEE080-20FD-4E24-BAAE-04347E912F9D},{E64C831E-36DC-4BAD-84B0-4C863C2331E7},1/31/2011 6:08:07 AM,1/31/2011 6:08:07 AM,0,0x,End of package execution.

No, these would not be helpful. I recall we also had issues on Virtual Machines, to fix we resorted to installing missing patches, do you have any KBs/SPs to apply that make sense?Arthur My Blog

All patches are up to date. My next step is to open a ticket. Thanks for the responses Arthur.

Issue ended up being VMWare drivers on the hardware side not being updated by our Infrastructure group. I needed to stop searching with "DTExec". http://kb.vmware.com/kb/1028440