Hey Guys, I have a 3 server Exchange 2010 SP2 RU2 DAG. All three servers yesterday had store.exe crash on them repeatedly. Yesterday was a holiday here, so there wasn't much going on so I'm trying to figure out what could've triggered this. The only entries in the System logs are "Exchange Information service terminated uenxpectedly" entries that happened every few minutes for 2 hours with a 30 minute gap at one point. The application logs show that store.exe crashed: Faulting application name: store.exe, version: 14.2.283.3, time stamp: 0x4f14dc0c Faulting module name: VSAPIScanSourceX64.dll, version: 7.0.1425.0, time stamp: 0x4ee7f77f There's not much to go on that was happening prior to these issues. The first time it happened, there were a couple of warnings 3 minutes prior to the crash: "folder is being scoped by a search which no longer exists" "An exception occurred and was handled by Exchange ActiveSync. This may have been caused by an outdated or corrupted Exchange ActiveSync device partnership. " Otherwise not much going on, DB health checks all passed 5 minutes prior to the crash. The only thing I can assume is that the faulting module (FSAPIScanSourceX64.dll) suggests there's yet another issue with McAfee GroupShield for Exchange. I'm mostly posting this in hopes of finding someone else with the same issue since maybe this is caused by some new bug. Still, if you have any suggestions on how to further track down the cause, please let me know. I found one post with similar errors in the migration forum, except in our case this had nothing to do with migrations. http://social.technet.microsoft.com/Forums/hi-IN/exchangesvrmigration/thread/95ab6f87-6766-4791-a30f-10cf38f3b37c. I will try the suggestion in there of running Test-MRSHealth to see if it reports back similar errors. Thanks, Matt

Please find below article for your reference https://kc.mcafee.com/corporate/index?page=content&id=KB52898 As per above article VSAPIScanSourceX64.dll pointing to Mcafee GroupShield 7.0.2 Stop all GroupShield services and check the status of "Exchange Information service also suggest you to consult with Mcafee support team for above issue Run chkdsk read only mode on all drives to check any disk problems on server. Exclude exchange database from file level scanning in antivirus Girishp

Matt, Did you get anywhere with this error? Our Exchange 2010 SP2 RU3 server has just started logging a very similar error. Seems to be crashing every 10 minutes or so, no warnings prior to the crash. DB health checks all report ok also. Here's our eventlog: Log Name: Application Source: Application Error Date: 6/21/2012 9:12:57 PM Event ID: 1000 Task Category: (100) Level: Error Keywords: Classic User: N/A Computer: emailserver.domain.xxx Description: Faulting application name: store.exe, version: 14.2.309.1, time stamp: 0x4fb19456 Faulting module name: VSAPIScanSourceX64.dll, version: 7.0.1425.0, time stamp: 0x4ee7f77f Exception code: 0xc000000d Fault offset: 0x000000000000b24c Faulting process id: 0x2c60 Faulting application start time: 0x01cd4fea0f2b572c Faulting application path: D:\Program Files\Microsoft\Exchange Server\V14\bin\store.exe Faulting module path: C:\Program Files (x86)\McAfee\GroupShield for Exchange\bin\VSAPIScanSourceX64.dll Report Id: 7d4148fc-bbdd-11e1-9f87-00237d579a0a Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Level>2</Level> <Task>100</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2012-06-21T20:12:57.000000000Z" /> <EventRecordID>1020504</EventRecordID> <Channel>Application</Channel> <Computer>emailserver.domain.xxx</Computer> <Security /> </System> <EventData> <Data>store.exe</Data> <Data>14.2.309.1</Data> <Data>4fb19456</Data> <Data>VSAPIScanSourceX64.dll</Data> <Data>7.0.1425.0</Data> <Data>4ee7f77f</Data> <Data>c000000d</Data> <Data>000000000000b24c</Data> <Data>2c60</Data> <Data>01cd4fea0f2b572c</Data> <Data>D:\Program Files\Microsoft\Exchange Server\V14\bin\store.exe</Data> <Data>C:\Program Files (x86)\McAfee\GroupShield for Exchange\bin\VSAPIScanSourceX64.dll</Data> <Data>7d4148fc-bbdd-11e1-9f87-00237d579a0a</Data> </EventData> </Event> On the McAfee link Girish posted I noticed there isn't an entry for v 7.0.1425.0 of VSAPIScanSourceX64.dll.... We've not updated McAfee recently nor installed any Windows updates, the only possible change could be a DAT update but why wouldn't we have been affected when you were if that was the case....

Matt, Did you get anywhere with this error? Our Exchange 2010 SP2 RU3 server has just started logging a very similar error. Seems to be crashing every 10 minutes or so, no warnings prior to the crash. DB health checks all report ok also. Here's our eventlog: Log Name: Application Source: Application Error Date: 6/21/2012 9:12:57 PM Event ID: 1000 Task Category: (100) Level: Error Keywords: Classic User: N/A Computer: emailserver.domain.xxx Description: Faulting application name: store.exe, version: 14.2.309.1, time stamp: 0x4fb19456 Faulting module name: VSAPIScanSourceX64.dll, version: 7.0.1425.0, time stamp: 0x4ee7f77f Exception code: 0xc000000d Fault offset: 0x000000000000b24c Faulting process id: 0x2c60 Faulting application start time: 0x01cd4fea0f2b572c Faulting application path: D:\Program Files\Microsoft\Exchange Server\V14\bin\store.exe Faulting module path: C:\Program Files (x86)\McAfee\GroupShield for Exchange\bin\VSAPIScanSourceX64.dll Report Id: 7d4148fc-bbdd-11e1-9f87-00237d579a0a Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Level>2</Level> <Task>100</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2012-06-21T20:12:57.000000000Z" /> <EventRecordID>1020504</EventRecordID> <Channel>Application</Channel> <Computer>emailserver.domain.xxx</Computer> <Security /> </System> <EventData> <Data>store.exe</Data> <Data>14.2.309.1</Data> <Data>4fb19456</Data> <Data>VSAPIScanSourceX64.dll</Data> <Data>7.0.1425.0</Data> <Data>4ee7f77f</Data> <Data>c000000d</Data> <Data>000000000000b24c</Data> <Data>2c60</Data> <Data>01cd4fea0f2b572c</Data> <Data>D:\Program Files\Microsoft\Exchange Server\V14\bin\store.exe</Data> <Data>C:\Program Files (x86)\McAfee\GroupShield for Exchange\bin\VSAPIScanSourceX64.dll</Data> <Data>7d4148fc-bbdd-11e1-9f87-00237d579a0a</Data> </EventData> </Event> On the McAfee link Girish posted I noticed there isn't an entry for v 7.0.1425.0 of VSAPIScanSourceX64.dll.... We've not update McAfee recently nor installed any Windows updates, the only possible change could be a DAT update but why wouldn't we have been affected when you were if that was the case....

Hey, No, sadly our solution to this problem was to uninstall McAfee because after my post it randomly happened again a few weeks later. Store.exe kept crashing every few minutes just as you had experienced. No idea why, seems completely unprovoked. We also didn't have any McAfee updates or Windows updates to link this problem to. We're going to try reinstalling it again but this time disable the VSAPI scanning and only leave transport scanning enabled. We didn't have these problems until we switched to having all roles installed on each of our 3 servers in the DAG. Prior to that our Transport servers were separate. What's your current setup? Matt

We've only got the 1 server so like you all roles are on 1 machine. It seems pretty sporadic, it seemed to happen approx every 10 minutes for 3 hours on Friday but today for example it's done it twice, once at 6am and then again around 9:34. There goes my idea that it might have been load related.... Let me know how you get on and I'll do likewise. For now I've just disabled VSAPI Scanning but with it being so sporadic it's going to be a while before we can say it's "fixed". Strange how there's hardly anyone else out there posting this message. What OS are you running on the 3 boxes? Ours is 2008R2 Standard, SP1. Josh

Hey Josh, 2008 R2 SP1 Enterprise. And yea I agree, I was expecting to see more of these posts.. glad to have some company though since at least now we both know it's not an isolated issue. Our issues were also completely random, not load related or backup or anything. One happened middle of the day on a long weekend with practically 0 load. What kind of server is yours on? Ours are all IBM x3650 M3's. Maybe we can narrow this down to something unique to us. Matt

Hey Matt, Yeah good to have some company on this! Our server is a HP DL380 G5. The McAfee setup is, GroupShield for Exchange v. 7.0.1406.110, SP2, Hotfixes: HF648829;GSE7SP2Rollup2;Patch1;HF726815 & Virus Scan Console: 8.8. Since I disabled VSAPI scanning yesterday it hasn't happened, but it's gone away for at least a day before so I'm going to leave it disabled over the weekend see how that goes, re-enable it on Monday see if it comes back. Josh