High CPU usage on FIM Sync server when no sync activity - mmsscrpt.exe process
What I was seeing with 4.0.3531.2 (RTM +update1) is the miiserver.exe using 98% of the CPU (on all cores). And indeed restarting the FIM Synchronization serivce fixes this. On one of our Sync servers (dedicated to GAL Sync) we installed
4.0.3555.2 (a hotfix prior to 4.0.3566.2), and ever since we haven't seen that issue return.
So this might be something new you discovered.
Regards,
Thomashttp://setspn.blogspot.com
November 16th, 2010 12:08pm
I'm using FIM Version 4.0.3561.2 which I understand addressed a number of issues including high CPU usage observed on export. Whether or not it's new to this patch I'm unsure, but I have a situation now whereby there is a constant CPU usage on the
FIM Sync server of around 98% attributable to the mmsscrpt.exe process. There is no sync activity presently running. Can anyone explain this? When I restart the FIM Sync service the CPU usage drops back to next to nothing ...Bob Bradley, www.unifysolutions.net (FIMBob?)
Free Windows Admin Tool Kit Click here and download it now
November 16th, 2010 12:44pm
Are you doing Exchange 2010 provisioning?
Are you using an ECMA with password sync/change code?
What kind of configuration do you have? traditional? codeless? mix?
November 16th, 2010 5:07pm
Paul - it's Exchange 2007 provisioning, no ECMA and no pwd sync. It's 100% "codeless" at the moment :).Bob Bradley, www.unifysolutions.net (FIMBob?)
Free Windows Admin Tool Kit Click here and download it now
November 16th, 2010 6:14pm
The plot thickens - there may be a problem with the scripting interface ... as the problem surfaces immediately after running the following VBS (via cscript.exe) from the command prompt:
Option Explicit
Const PktPrivacy = 6
Const MA_AD = "{F2B5ABA4-1998-4A27-BC31-A2F335E83A73}"
Const MA_STAGING = "{3AED062D-DEB0-4327-B489-5D6F0EDBC64A}"
Const MA_FIM = "{43A4B3CA-D406-4DCD-83B4-9E10B4E4A09F}"
Dim Locator, Service
Set Locator = CreateObject("WbemScripting.SWbemLocator")
Set Service = GetObject("winmgmts:{authenticationLevel=PktPrivacy}!root/MicrosoftIdentityIntegrationServer")
ExecRunProfile MA_AD, "Delta Import Delta Synch"
ExecRunProfile MA_STAGING, "Delta Import Delta Sync"
ExecRunProfile MA_FIM, "Delta Import Delta Synchonization"
ExecRunProfile MA_FIM, "Export Delta Import Delta Sync"
ExecRunProfile MA_AD, "Export"
ExecRunProfile MA_AD, "Delta Import Delta Synch"
ExecRunProfile MA_FIM, "Export Delta Import Delta Sync"
ExecRunProfile MA_FIM, "Export Delta Import Delta Sync"
Set Service = Nothing
Set Locator = Nothing
Sub ExecRunProfile (ByVal MAGuid, ByVal RunProfile)
Dim MASet, MA
Set MASet = Service.ExecQuery("select * from MIIS_ManagementAgent where Guid = '" + MAGuid + "'")
for each MA in MASet
WScript.Echo "Running " + MA.name + ".Execute(""" + RunProfile + """)..."
WScript.Echo "Run completed with result: " + MA.Execute("" + RunProfile + "")
next
Set MASet = Nothing
End Sub
Can anyone spot anything in the above that might be causing the problem? The above is based on the generated VBS files from the run profile "Script" button, and it is designed for calling by the Windows Task Scheduler ... I was thinking of posting
the above as a sample FIM delta sync cycle for a 3 MA configuration anyhow :).
Bob Bradley, www.unifysolutions.net (FIMBob?)
November 17th, 2010 7:47am
Bob, what about trying old runMA.vbs from MS inside basic .cmd without 'for each' scripting?
Free Windows Admin Tool Kit Click here and download it now
November 17th, 2010 9:18am
I've had a quick look internally and the known issue is around Exchange 2010 and PS not being a seperate process. This will change in the next update (update 2).
A couple of guys hit some problems with ECMAs and password reset. The underlying issue *might* be a .NET bug to do with unloading. You might want to consider looking at this as a possibility:
http://support.microsoft.com/?id=981574
http://support.microsoft.com/?id=971988
The former might supercede the latter.
However in answer to your question I'm running PS scripts that do pretty much what you're doing above against 4.0.3558.2 without any issue. I have one Oracle MA, two SQL MA, one AD MA and one FIM MA. Everything's "codeless" except the AD deprovisioning
action which is a basic switch statement that does either a disconnect or a delete, depending on objectClass.
November 17th, 2010 5:12pm
I've had a quick look internally and the known issue is around Exchange 2010 and PS not being a seperate process. This will change in the next update (update 2).
A couple of guys hit some problems with ECMAs and password reset. The underlying issue *might* be a .NET bug to do with unloading. You might want to consider looking at this as a possibility:
http://support.microsoft.com/?id=981574
http://support.microsoft.com/?id=971988
The former might supercede the latter.
However in answer to your question I'm running PS scripts that do pretty much what you're doing above against 4.0.3558.2 without any issue. I have one Oracle MA, two SQL MA, one AD MA and one FIM MA. Everything's "codeless" except the AD deprovisioning
action which is a basic switch statement that does either a disconnect or a delete, depending on objectClass.
Free Windows Admin Tool Kit Click here and download it now
November 17th, 2010 5:12pm
Thanks Paul - I think you're on the money there ... because I don't have the problem when I simply comment out the EXPORT run profile to AD ...
Thanks Evgeniy ... I tried the default generated VBS for one of the run profiles, and there was no problem ... then I tried commenting selected "ExecRunProfile" lines out of my above script, and also found there was no problem ... so I figured it wasn't
my script. I also removed the "Option Explicit" directive (hate not having this set!), still ran OK.
I've now proven that the above script works 100% OK provided I do one of the following:
comment out the EXPORT TO AD step; OR comment out ALL the FIM MA run profiles
If I try to combine the FIM MA run profiles with the AD Export run profile (other AD run profiles no problem), THAT'S when I get the 99% CPU problem!!! So it APPEARS that there's a conflict between the FIM MA run profiles (any of them - not just export) and
the AD Export run profile.
I'll now take a look at the links you published there Paul ...Bob Bradley, www.unifysolutions.net (FIMBob?)
November 17th, 2010 6:43pm
Thanks Paul - I think you're on the money there ... because I don't have the problem when I simply comment out the EXPORT run profile to AD ...
Thanks Evgeniy ... I tried the default generated VBS for one of the run profiles, and there was no problem ... then I tried commenting selected "ExecRunProfile" lines out of my above script, and also found there was no problem ... so I figured it wasn't
my script. I also removed the "Option Explicit" directive (hate not having this set!), still ran OK.
I've now proven that the above script works 100% OK provided I do one of the following:
comment out the EXPORT TO AD step; OR comment out ALL the FIM MA run profiles
If I try to combine the FIM MA run profiles with the AD Export run profile (other AD run profiles no problem), THAT'S when I get the 99% CPU problem!!! So it APPEARS that there's a conflict between the FIM MA run profiles (any of them - not just export) and
the AD Export run profile.
I'll now take a look at the links you published there Paul ...Bob Bradley, www.unifysolutions.net (FIMBob?)
Free Windows Admin Tool Kit Click here and download it now
November 17th, 2010 6:43pm
Oh - one more thing ... about every 2nd or 3rd time I execute the AD export run profile there's a delay of about 4-5 seconds - even when there's no activity ... a problem with Exchange integration and "unloading" sounds consistent with this
behaviour.Bob Bradley, www.unifysolutions.net (FIMBob?)
November 17th, 2010 6:47pm
Oh - one more thing ... about every 2nd or 3rd time I execute the AD export run profile there's a delay of about 4-5 seconds - even when there's no activity ... a problem with Exchange integration and "unloading" sounds consistent with this
behaviour.Bob Bradley, www.unifysolutions.net (FIMBob?)
Free Windows Admin Tool Kit Click here and download it now
November 17th, 2010 6:47pm
I was having an issue around ECMAs and CPU going to the top and the above (KB 981574) seemed to solve the problem.My Book - Active Directory, 4th Edition
My Blog - www.briandesmond.com
November 17th, 2010 10:47pm
Bob, if you're doing only Exch2007 provisioning and not Exch2010 - can you temporary disable exchange provisioning option in AD MA and check for AD MA Export profile run and CPU load?
what I meant for 'old MS runMA.vbs':
when you script you MA run profile it looks like this (using 'select * from MIIS_MA where'):
Set Locator = CreateObject("WbemScripting.SWbemLocator")
Set Service = GetObject("winmgmts:{authenticationLevel=PktPrivacy}!root/MicrosoftIdentityIntegrationServer")
Set MASet = Service.ExecQuery("select * from MIIS_ManagementAgent where Guid = '{B71F58B1-60AD-454B-81D5-95F22258C0BC}'")
for each MA in MASet
WScript.Echo "Running " + MA.name + ".Execute(""export - !!!"")..."
WScript.Echo "Run completed with result: " + MA.Execute("export - !!!")
next
old Microrosoft runMA.vbs is using wmiService.Get( "MIIS_ManagementAgent.Name='" & managementagentName & "'")
which does the same thing but may have sense for your environment
set wmiService = wmiLocator.ConnectServer(server, "root/MicrosoftIdentityIntegrationServer")
Set managementagent = wmiService.Get( "MIIS_ManagementAgent.Name='" & managementagentName & "'")
runResult = managementagent.Execute(profile)
'cause I have exchange 2007 provisioning turned on and no problems with CPU load - but I have miiserver.exe version 4.0.3531.2
if your version 4.0.3561.2 has this issues with CPU load - I would not update then :)
I wanted to update to 4.0.3561.2
to fix my Recycle Bin issues but seems like its better to wait a little more...
Free Windows Admin Tool Kit Click here and download it now
November 18th, 2010 2:28am
Bob, if you're doing only Exch2007 provisioning and not Exch2010 - can you temporary disable exchange provisioning option in AD MA and check for AD MA Export profile run and CPU load?
what I meant for 'old MS runMA.vbs':
when you script you MA run profile it looks like this (using 'select * from MIIS_MA where'):
Set Locator = CreateObject("WbemScripting.SWbemLocator")
Set Service = GetObject("winmgmts:{authenticationLevel=PktPrivacy}!root/MicrosoftIdentityIntegrationServer")
Set MASet = Service.ExecQuery("select * from MIIS_ManagementAgent where Guid = '{B71F58B1-60AD-454B-81D5-95F22258C0BC}'")
for each MA in MASet
WScript.Echo "Running " + MA.name + ".Execute(""export - !!!"")..."
WScript.Echo "Run completed with result: " + MA.Execute("export - !!!")
next
old Microrosoft runMA.vbs is using wmiService.Get( "MIIS_ManagementAgent.Name='" & managementagentName & "'")
which does the same thing but may have sense for your environment
set wmiService = wmiLocator.ConnectServer(server, "root/MicrosoftIdentityIntegrationServer")
Set managementagent = wmiService.Get( "MIIS_ManagementAgent.Name='" & managementagentName & "'")
runResult = managementagent.Execute(profile)
'cause I have exchange 2007 provisioning turned on and no problems with CPU load - but I have miiserver.exe version 4.0.3531.2
if your version 4.0.3561.2 has this issues with CPU load - I would not update then :)
November 18th, 2010 2:28am
If I turn off the Exchange 2007 provisioning option on the AD MA I no longer have a problem :) ... I guess that's fairly difinitive then. I'll be applying the patch recommended by Paul once we know that we're not rolling back to pre-update 1 due
to the bug with the ResolveGrammarActivity :)Bob Bradley, www.unifysolutions.net (FIMBob?)
Free Windows Admin Tool Kit Click here and download it now
November 18th, 2010 3:09am
just to be sure, try to manually run update-recipient in Exchange management shell against that user. and check for CPU
So we'll see if its either powershell or .net problem
I wish it would be .net problem... and kb971988 fixes it.
November 18th, 2010 3:16am
Evgniy - there's no user in context, so I don't know what the equivalent activity would be for an export run with no pending exports :). However, I am already convinced that the problem is to do with the .Net namespace issue referred to by Paul,
and Brian has already confirmed this fixed the problem for him. That's enough for me ... but I won't close this until I can prove it's definitely the fix I want. Haven't got any more time to spend on this one ...Bob Bradley, www.unifysolutions.net (FIMBob?)
Free Windows Admin Tool Kit Click here and download it now
November 18th, 2010 3:21am
ok, I have plenty of time - will try it now with .net hotfix
November 18th, 2010 3:57am
upgraded to 4.0.3561.2, .net hotfix is not installed
recycle bin issue is not fixed, still have export-change-not-reimported for membership and manager values;
no high cpu load on AD Export profile run
Free Windows Admin Tool Kit Click here and download it now
November 18th, 2010 5:45am
upgraded to 4.0.3561.2, .net hotfix is not installed
recycle bin issue is not fixed, still have export-change-not-reimported for membership and manager values;
no high cpu load on AD Export profile run
November 18th, 2010 5:45am
Regarding:
recycle bin issue is not fixed, still have export-change-not-reimported for membership and manager values;
The FIM PG is still writing a fix for this. My case is still open. So indeed this is not fixed yet.
http://setspn.blogspot.com
Free Windows Admin Tool Kit Click here and download it now
November 18th, 2010 5:54am
Regarding:
recycle bin issue is not fixed, still have export-change-not-reimported for membership and manager values;
The FIM PG is still writing a fix for this. My case is still open. So indeed this is not fixed yet.
http://setspn.blogspot.com
November 18th, 2010 5:54am
.Net (namespace) hotfix fixed my issue, so thanks guys - pretty conclusive.Bob Bradley, www.unifysolutions.net (FIMBob?)
Free Windows Admin Tool Kit Click here and download it now
November 24th, 2010 6:09am