Hi, I've seen notes that performance with processing large groups was a problem in FIM RC builds and was supposed to be addressed in RTM. How is it now? I'm currently trying to see how FIM Synchronization Service deals with large groups and how to design it properly. For testing I was trying to set up a simple scenario similar to MIIS/ILM and GroupBuilder - groups are built in SQL and synchronized to AD. I'm testing with a 100k member group, and just the delta import step from SQL (importing just the one group) took several hours. I'm running the delta sync now, and few hours passed already ... I'm wondering how the export to AD will be. My question is, bottom line, is there any hope in trying to use the standard MAs (SQL and AD) and FIM mechanisms to manage very large groups (like 100k-200k+ members)? Or it's a waste of time, and I need to be creative here. Piotr

The import process for the built-in SQL MA is really slow with multivalue fields, unfortunately. I haven't tried this with FIM but it was definetely difficult with MIIS/ILM. Processing reference attributes is the other place where this really seems to slow down. One solution that another MVP showed me involved calculating the group memberships and storing them as strings in the metaverse versus references. That sped things up substantially. The export to AD should be pretty quick, though.My Book - Active Directory, 4th Edition My Blog - www.briandesmond.com

I know Joe Stepongzi has done a lot of work with managing very large groups. He was promising to published an optimized SQL XMA at one point, and I know he prefered to use LDIF on the AD side, because it's possible to add and remove members without having to resync the whole member attribute. But I don't think he ever posted any solutions. You could try contacting him through his blog <cite>www.microsoftidm.com</cite> <cite></cite>http://www.wapshere.com/missmiis

I know Joe Stepongzi has done a lot of work with managing very large groups. He was promising to published an optimized SQL XMA at one point, and I know he prefered to use LDIF on the AD side, because it's possible to add and remove members without having to resync the whole member attribute. But I don't think he ever posted any solutions. You could try contacting him through his blog <cite>www.microsoftidm.com</cite> <cite></cite>http://www.wapshere.com/missmiis

Thanks Brian and Carol, at this point I just wanted to confirm that using the default MAs will never work for managing really large groups. I will definitely contact Joe to get his insight on this matter. I also have several ideas on my own how to make this work properly, looks like a lot of testing ahead of me. Would appreciate if anyone else has any experience or examples how to solve this. Piotr

Thanks Brian and Carol, at this point I just wanted to confirm that using the default MAs will never work for managing really large groups. I will definitely contact Joe to get his insight on this matter. I also have several ideas on my own how to make this work properly, looks like a lot of testing ahead of me. Would appreciate if anyone else has any experience or examples how to solve this. Piotr

Piotr - I would be looking at what you can do with set transitions now, using the SQL MA to perform direct import flows only into FIM to drive recalculation of dynamic groups - importantly avoiding reference value types and multivalue attributes in the SQL MA (as mentioned above). Of course you may find yourself having to write custom activities, or resort to rules extensions, as the FIM portal needs to ultimately work with reference attributes for dynamic groups to work, but the trick will be to minimise the load on the SQL MA import.Bob Bradley, www.unifysolutions.net (FIMBob?)

Piotr - I would be looking at what you can do with set transitions now, using the SQL MA to perform direct import flows only into FIM to drive recalculation of dynamic groups - importantly avoiding reference value types and multivalue attributes in the SQL MA (as mentioned above). Of course you may find yourself having to write custom activities, or resort to rules extensions, as the FIM portal needs to ultimately work with reference attributes for dynamic groups to work, but the trick will be to minimise the load on the SQL MA import. Bob Bradley, www.unifysolutions.net (FIMBob?) It's still going to spend alot of time processing the references on the FIM and AD syncs. I haven't tried this in FIM (vs ILM) so it's possible this has gotten better since then.My Book - Active Directory, 4th Edition My Blog - www.briandesmond.com

This is more painful on the SQL MA side to be honest, especially on the Import. The preferred approach here is to use one of the File MAs to import the data from a SQL source. Rebecca Croft will be presenting on this topic at TEC this year and showing off how Ensynch translates SQL tables into LDIF for fast imports as well as produces deltas. While this example does not use multi-valued attribues like a group would, we have scaled this approach up to millions of identities and the LDIF file import is far superior and it could be extended to manage multivalued attributes. As Carol points out, there are significant advantages to using LDIF as the file format for expressing changes in multivalued attributes like a Group. In Joe's examples, he avoids reference processing all the way up to the point at which the group is exported to AD, thus his use of XMA's for AD group exports as well. I can say that reference processing in FIM is far better than it was, especially now that FIM requires the SQL Native client. Instead of using the Group Populator approach, have you modeled this using Sets instead? You should be able to express most of the GP scenarios within the FIM Service, the only one I haven't seen tried is generating the groups themselves dynamically. This would bypass the SQL import stage altogether and move the prolem to the FIM Service. In this case you'll want to get to know the timeout values on whichever service your portal or admin portal is pointing to: http://blogs.msdn.com/b/darrylru/archive/2010/02/02/extending-fim-timeouts.aspxBrad Turner, ILM MVP - Ensynch, Inc - www.identitychaos.com [If a post helps to resolve your issue, please click the "Mark as Answer" or "Helpful" button at the top of that post. By marking a post as Answered or Helpful, you help others find the answer faster.]

Guys, my SQL MA with 8000 objects (4000 users, 3500 groups + contacts and PFs) does 'full import + full sync' run for 8 minutes. this includes multivalued attributes like membership. all FIM components runs inside one VM, even SQL server for FIM portal and FIM sync DBs are inside that VM. only membership is stored on external clustered SQL server. if you have to wait hours then I would look for memory usage by MIISServer.exe and by the SQL keeping membership. something is wrong there - until the latest release I had MIISServer issues with memory leakage. it could easily grow from 100MB to 2GB during full sync and started to swap...

Evgeniy ... do you mean your instance of "the SQL MA" or your own custom ECMA? If it is the native SQL MA and you're getting that sort of perfomrance I'm surprised ... because a full import involves looping through the primary view/table and doing a PK lookup on the second view (put a SQL trace on this MA to see what I mean if you haven't already). Only way I could get that sort of performance was to preload the main and multi-values fully into memory and do in-memory joins using the DataRelationship object.Bob Bradley, www.unifysolutions.net (FIMBob?)

Thanks Brad, I'll definitely attend Rebecca's session at TEC. I'm currently also looking into ldif files as one of the alternatives. As for using Sets/FIM Service, this instance is purely for back-end processing so I'm only using the FIM Sync Service here. I'll try to do a little performance comparison between using the ldif files and XMA's. As for export to AD, is the AD MA smart enough to only export changes to the multi-value attributes, or is it also refreshing the entire attribute (that's still the fact for SQL MA Export right?)? With the example from Evgeniy, it's possible ... ~8000 objects synced in 8 minutes, that's ~17 objects per second ... that's totally doable even with reference attributes. Anyway, this is not the scale we're talking about here, with 10k objects you don't have to care about anything, and could just run full syncs every 10 minutes. This becomes tricky if you have multiple MAs with close to a million objects in each, every millisecond spent on an object counts. Thanks everyone for the input, this was really helpful. Piotr

Evgeniy ... do you mean your instance of "the SQL MA" or your own custom ECMA? If it is the native SQL MA and you're getting that sort of perfomrance I'm surprised ... because a full import involves looping through the primary view/table and doing a PK lookup on the second view (put a SQL trace on this MA to see what I mean if you haven't already). Only way I could get that sort of performance was to preload the main and multi-values fully into memory and do in-memory joins using the DataRelationship object. Bob Bradley, www.unifysolutions.net (FIMBob?) I meant native SQL MA and almost no declarative rules, just old MV extension on vb.net which is way more faster than new declarative rules. this machine is running with 1 CPU and 4 gigs of RAM, 2 GB for SQL instance holding sync DB. I cannot see 100% CPU load during sync, so SQL/RAM is the bottleneck in my setup (notice 1 CPU as MIISServer.exe can't do things in parallel :) )

Guys, my SQL MA with 8000 objects (4000 users, 3500 groups + contacts and PFs) does 'full import + full sync' run for 8 minutes. this includes multivalued attributes like membership. all FIM components runs inside one VM, even SQL server for FIM portal and FIM sync DBs are inside that VM. only membership is stored on external clustered SQL server. if you have to wait hours then I would look for memory usage by MIISServer.exe and by the SQL keeping membership. something is wrong there - until the latest release I had MIISServer issues with memory leakage. it could easily grow from 100MB to 2GB during full sync and started to swap...

Evgeniy ... do you mean your instance of "the SQL MA" or your own custom ECMA? If it is the native SQL MA and you're getting that sort of perfomrance I'm surprised ... because a full import involves looping through the primary view/table and doing a PK lookup on the second view (put a SQL trace on this MA to see what I mean if you haven't already). Only way I could get that sort of performance was to preload the main and multi-values fully into memory and do in-memory joins using the DataRelationship object. Bob Bradley, www.unifysolutions.net (FIMBob?) I meant native SQL MA and almost no declarative rules, just old MV extension on vb.net which is way more faster than new declarative rules. this machine is running with 1 CPU and 4 gigs of RAM, 2 GB for SQL instance holding sync DB. I cannot see 100% CPU load during sync, so SQL/RAM is the bottleneck in my setup (notice 1 CPU as MIISServer.exe can't do things in parallel :) )