Idea to automate the clean-up of orphaned EREs
Hi again. I'm after some anecdotal/theoretical assessment on an idea I have successfully implemented in a lab just now to clean up 13000 orphaned EREs. I am considering implementing this idea in the Production environment, but I want to find out if there are any good reasons why NOT to do this ... particularly as it strikes me that if orphaned EREs were supposed to be deleted immediately they become orphaned then this would be OOTB FIM functionality!!! What I have done is created the following: a set (Orphaned EREs) defined as "Select expected rule entry that match all of the following conditions: Resource Parent not in All Objects" a workflow (Delete orphaned EREs - run on policy update = ON) invoking a custom activity to delete an /ExpectedRuleEntry[ObjectID='[//Target/ObjectID]'] a set transition MPR (Orphaned ERE objects are automatically deleted) to invoke the above workflow whenever an ERE falls into the above set The above worked like a charm, deleting all 13K orphaned EREs in an hour or two. Question is ... what are the implications (besides a little CPU overhead) on running such a "housekeeping" activity in a production environment? I was thinking of making the set temporal based on one of the many DateTime attributes on an ERE, but it is not clear which one to use ... but leaving this to run at say 1 am every night might actually be more of an overhead than running it as and when an ERE is orphaned. Thoughts anyone? TIABob Bradley, www.unifysolutions.net (FIMBob?)
December 15th, 2010 12:52am

Ignoring ERE cleanups for a moment, you hit the nail on its head - the main consideration in conjunction with these operations is the performance impact. As an alternative, you could also just schedule a script that runs during the night - comparable to a run profile automation. The tradeoff of an automated script is that a script can fail to do its job. In the case of a cleanup process, this is probably not too bad because the worst case is that you have unnecessary data; however, this kind of data doesn't prevent your system from working correctly. Regarding orphaned EREs, I belief that a solution designer should also look at how to avoid creating them. It means, you have deleted an object that has at least one active OSR - what is the impact of this on your environment? So, from that perspective, it might be better to prevent objects with active OSRs from being deleted. While you might be authoritative for FIM, you might not have the required permissions to do the required cleanups in the external systems. So, what I'm trying to get at is that there should be first a discussion on where the orphaned EREs came from and how to avoid them before looking into a cleanup solution - does this make sense? Cheers, Markus Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
December 15th, 2010 8:42am

Markus - thanks for the reply. Firstly, I am not a fan of an external scheduled task if I can avoid it as these tend to be "out of sight, out of mind" - however if this is considered "best practice" then maybe that's the way to go. I would prefer to continue to drive this from a set transition MPR, but make it a temporal set, which means I would need to drive it from a date. Is one of the date bindings for an ERE appropriate here? Secondly, the EREs I deleted were caused by a design change where object classes and their corresponding attribute flows were removed from the solution entirely ... but I'd read about the occurrance of these in a day-to-day context and figured this might be a good idea. I understand this would be band-aiding, though, if what you are saying is treat the cause and not the symptom ... I guess I was wondering if an orphaned ERE could occur through no fault of the design ...Bob Bradley, www.unifysolutions.net (FIMBob?)
December 15th, 2010 8:50am

Bob, from my point of view - MPR with a transition set is an easy solution to remove orphaned EREs... but! I would never do this in my production. the reason for this is a question: why do I have these EREs at all? I know if I would delete a group object with an active OSR it would be deleted from AD, SQL MA, Sharepoint MA and so on by the object deletion rule in MIIS. I do not care about that ERE. in this particular case. but in another scenario I would think of placing custom WF action as a first one in AuthZ stage on MPR allowing to delete a group. having your MPR to automatically delete such EREs can hide a problem with existing design or failed workflows... who knows. it really depends on design. as for me - I do use cmdlet from Markus and run it manually once a week.
Free Windows Admin Tool Kit Click here and download it now
December 15th, 2010 2:40pm

Bob, from my point of view - MPR with a transition set is an easy solution to remove orphaned EREs... but! I would never do this in my production. the reason for this is a question: why do I have these EREs at all? I know if I would delete a group object with an active OSR it would be deleted from AD, SQL MA, Sharepoint MA and so on by the object deletion rule in MIIS. I do not care about that ERE. in this particular case. but in another scenario I would think of placing custom WF action as a first one in AuthZ stage on MPR allowing to delete a group. having your MPR to automatically delete such EREs can hide a problem with existing design or failed workflows... who knows. it really depends on design. as for me - I do use cmdlet from Markus and run it manually once a week.
December 15th, 2010 2:40pm

Bob, from my point of view - MPR with a transition set is an easy solution to remove orphaned EREs... but! I would never do this in my production. the reason for this is a question: why do I have these EREs at all? I know if I would delete a group object with an active OSR it would be deleted from AD, SQL MA, Sharepoint MA and so on by the object deletion rule in MIIS. I do not care about that ERE. in this particular case. but in another scenario I would think of placing custom WF action as a first one in AuthZ stage on MPR allowing to delete a group. having your MPR to automatically delete such EREs can hide a problem with existing design or failed workflows... who knows. it really depends on design. as for me - I do use cmdlet from Markus and run it manually once a week.
Free Windows Admin Tool Kit Click here and download it now
December 15th, 2010 2:40pm

The simple scenario where you have an object deletion rule set to anything other than the FIM MA is going to cause these things to be orphaned. IMO it's not really a design flaw if you're using the product as intended but part of the product (EREs in this case) don't talk to the other part of the product. One of the MCS folks that posts here sometimes has a solution much like Evgeniy describes which uses an AuthZ WF that fires on delete.My Book - Active Directory, 4th Edition My Blog - www.briandesmond.com
December 16th, 2010 10:19am

Thanks for your replies guys ... makes sense to me, and Brian/Evgeniy - your last comments about OSRs and the object deletion rule explains why I am continuing to get these EREs under some circumstances (but this being a development environment I am aware that a lot of them have not arisen from realistic production scenarios ... such as schema cleanup or use case replay). I am curious to know more about the MCS idea, because the ERE in itself has very little useful info in it and wouldn't be much use to an admin trying to make a decision - so I presume you mean firing an AuthZ just before the ERE is orphaned?Bob Bradley, www.unifysolutions.net (FIMBob?)
Free Windows Admin Tool Kit Click here and download it now
December 16th, 2010 6:27pm

Thanks for your replies guys ... makes sense to me, and Brian/Evgeniy - your last comments about OSRs and the object deletion rule explains why I am continuing to get these EREs under some circumstances (but this being a development environment I am aware that a lot of them have not arisen from realistic production scenarios ... such as schema cleanup or use case replay). I am curious to know more about the MCS idea, because the ERE in itself has very little useful info in it and wouldn't be much use to an admin trying to make a decision - so I presume you mean firing an AuthZ just before the ERE is orphaned?Bob Bradley, www.unifysolutions.net (FIMBob?)
December 16th, 2010 6:27pm

Ohhhh - please don't get me wrong, I didn't mean to indicate a best practice here. To me, this is a perfect example for a scenario that doesn't have a "silver bullet" solution. One can either use an external cleanup script / process or something that is integrated into the FIM processing model. Like you have indicated it - "I'm not a fan of..." - is perfectly acceptable as long as an approach doesn't violate best practices - something like cleaning up a connector space by whipping the related table in the database :o) Just making sure, I'm not trying to defend external cleanup processes; I'm just trying to make sure that using them is not interpreted by anyone as a "second best approach". Whether or not an orphaned ERE could occur through no fault of the design is a tough one to answer - I wonder if there is an answer because it depends... When you delete an object with a populated ERL, you get "one" - is this a design flaw? Should the system automatically delete them? It might be OK to delete them and it might also be a design flaw if the system would do this. This is to me debatable. Cheers, Markus Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
December 16th, 2010 8:05pm

Ohhhh - please don't get me wrong, I didn't mean to indicate a best practice here. To me, this is a perfect example for a scenario that doesn't have a "silver bullet" solution. One can either use an external cleanup script / process or something that is integrated into the FIM processing model. Like you have indicated it - "I'm not a fan of..." - is perfectly acceptable as long as an approach doesn't violate best practices - something like cleaning up a connector space by whipping the related table in the database :o) Just making sure, I'm not trying to defend external cleanup processes; I'm just trying to make sure that using them is not interpreted by anyone as a "second best approach". Whether or not an orphaned ERE could occur through no fault of the design is a tough one to answer - I wonder if there is an answer because it depends... When you delete an object with a populated ERL, you get "one" - is this a design flaw? Should the system automatically delete them? It might be OK to delete them and it might also be a design flaw if the system would do this. This is to me debatable. Cheers, Markus Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
December 16th, 2010 8:05pm

Ohhhh - please don't get me wrong, I didn't mean to indicate a best practice here. To me, this is a perfect example for a scenario that doesn't have a "silver bullet" solution. One can either use an external cleanup script / process or something that is integrated into the FIM processing model. Like you have indicated it - "I'm not a fan of..." - is perfectly acceptable as long as an approach doesn't violate best practices - something like cleaning up a connector space by whipping the related table in the database :o) Just making sure, I'm not trying to defend external cleanup processes; I'm just trying to make sure that using them is not interpreted by anyone as a "second best approach". Whether or not an orphaned ERE could occur through no fault of the design is a tough one to answer - I wonder if there is an answer because it depends... When you delete an object with a populated ERL, you get "one" - is this a design flaw? Should the system automatically delete them? It might be OK to delete them and it might also be a design flaw if the system would do this. This is to me debatable. Cheers, Markus Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
December 16th, 2010 8:05pm

Thanks for your replies guys ... makes sense to me, and Brian/Evgeniy - your last comments about OSRs and the object deletion rule explains why I am continuing to get these EREs under some circumstances (but this being a development environment I am aware that a lot of them have not arisen from realistic production scenarios ... such as schema cleanup or use case replay). I am curious to know more about the MCS idea, because the ERE in itself has very little useful info in it and wouldn't be much use to an admin trying to make a decision - so I presume you mean firing an AuthZ just before the ERE is orphaned? Bob Bradley, www.unifysolutions.net (FIMBob?) AuthZ WF will fire before the Delete occurs so at that point you can walk the ERE/DRE links and remove the associated objects before the delete is processed.My Book - Active Directory, 4th Edition My Blog - www.briandesmond.com
December 16th, 2010 9:25pm

Thanks for your replies guys ... makes sense to me, and Brian/Evgeniy - your last comments about OSRs and the object deletion rule explains why I am continuing to get these EREs under some circumstances (but this being a development environment I am aware that a lot of them have not arisen from realistic production scenarios ... such as schema cleanup or use case replay). I am curious to know more about the MCS idea, because the ERE in itself has very little useful info in it and wouldn't be much use to an admin trying to make a decision - so I presume you mean firing an AuthZ just before the ERE is orphaned? Bob Bradley, www.unifysolutions.net (FIMBob?) AuthZ WF will fire before the Delete occurs so at that point you can walk the ERE/DRE links and remove the associated objects before the delete is processed.My Book - Active Directory, 4th Edition My Blog - www.briandesmond.com
Free Windows Admin Tool Kit Click here and download it now
December 16th, 2010 9:25pm

I've ended with prohibiting regular users to delete objects on a portal and removed these rights from all MPRs. instead I have a checkbox for lets say a 'role' named 'disable this role' that brings an object to 'To be deprovisioned' set and removes all OSRs from a role. and administrator has to manually delete objects once they appear in another set 'to be deleted' which is 30 days after all OSRs were removed. the second step was setting up MV object deletion rule by FIM MA only.
December 17th, 2010 2:48am

I've ended with prohibiting regular users to delete objects on a portal and removed these rights from all MPRs. instead I have a checkbox for lets say a 'role' named 'disable this role' that brings an object to 'To be deprovisioned' set and removes all OSRs from a role. and administrator has to manually delete objects once they appear in another set 'to be deleted' which is 30 days after all OSRs were removed. the second step was setting up MV object deletion rule by FIM MA only.
Free Windows Admin Tool Kit Click here and download it now
December 17th, 2010 2:48am

I've ended with prohibiting regular users to delete objects on a portal and removed these rights from all MPRs. instead I have a checkbox for lets say a 'role' named 'disable this role' that brings an object to 'To be deprovisioned' set and removes all OSRs from a role. and administrator has to manually delete objects once they appear in another set 'to be deleted' which is 30 days after all OSRs were removed. the second step was setting up MV object deletion rule by FIM MA only Wow, this is actually very cool! I like your approach with the disable checkbox a lot. Cheers, MarkusMarkus Vilcinskas, Knowledge Engineer, Microsoft Corporation
December 17th, 2010 3:03am

I've ended with prohibiting regular users to delete objects on a portal and removed these rights from all MPRs. instead I have a checkbox for lets say a 'role' named 'disable this role' that brings an object to 'To be deprovisioned' set and removes all OSRs from a role. and administrator has to manually delete objects once they appear in another set 'to be deleted' which is 30 days after all OSRs were removed. the second step was setting up MV object deletion rule by FIM MA only Wow, this is actually very cool! I like your approach with the disable checkbox a lot. Cheers, MarkusMarkus Vilcinskas, Knowledge Engineer, Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
December 17th, 2010 3:03am

Markus, for the perfect solution this checkbox should trigger Transition Out MPR and take object off the 'Enabled Roles' set and start Remove Sync Rules WF. same with object deletion - there must be another MPR that should raise object delete action. just have no time to do this :)
December 17th, 2010 5:06am

Markus, for the perfect solution this checkbox should trigger Transition Out MPR and take object off the 'Enabled Roles' set and start Remove Sync Rules WF. same with object deletion - there must be another MPR that should raise object delete action. just have no time to do this :)
Free Windows Admin Tool Kit Click here and download it now
December 17th, 2010 5:06am

It is possible that I have missed something - but why do you need a MPR for the object deletion? After removing an object from the scope of all OSRs, you just need to sync it. In other words, the deletion is imported from the synchronization service. Cheers, MarkusMarkus Vilcinskas, Knowledge Engineer, Microsoft Corporation
December 17th, 2010 5:31am

It is possible that I have missed something - but why do you need a MPR for the object deletion? After removing an object from the scope of all OSRs, you just need to sync it. In other words, the deletion is imported from the synchronization service. Cheers, MarkusMarkus Vilcinskas, Knowledge Engineer, Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
December 17th, 2010 5:31am

I have kind of protection from accident object deletion in external system. so MV object deletion rule is set up to fire only from FIM MA. once OSR is removed and object is deleted in external system is remains on FIM portal and in MV with all attributes for another 30 days (hand made recycle bin for an external MA :) ).
December 17th, 2010 5:35am

I have kind of protection from accident object deletion in external system. so MV object deletion rule is set up to fire only from FIM MA. once OSR is removed and object is deleted in external system is remains on FIM portal and in MV with all attributes for another 30 days (hand made recycle bin for an external MA :) ).
Free Windows Admin Tool Kit Click here and download it now
December 17th, 2010 5:35am

Ahhhh - very smart! Have you ever thought about documenting your solution? This is a great topic for a wiki article. Cheers, Markus Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
December 17th, 2010 6:42am

Ahhhh - very smart! Have you ever thought about documenting your solution? This is a great topic for a wiki article. Cheers, Markus Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
December 17th, 2010 6:42am

never thought about such things. I'll have 10 days off in January, so may be... actually it was caused by real life's practice: 1. imaging that you have an application role that grants access to something, e.g. 'HR manager' in Axapta. 2. one day you've migrated from Axapta to new Dynamics AX with its own roles 3. you don't need old 'HR manager' any more 4. to remove a role from business application catalog you have a proper workflow and everyone has approved that role removal (actually disable checkbox) 5. AD group is removed and remains in AD recycle bin. 6. in a week or two you get an incident that some 'smart' people from another department were using this group for something else. and they need its membership back. what would you do? restore a group from the recycle bin? enable it back on the FIM portal so the new group with another objectSID will be created? do you have a MV extension rule to delete all unnecessary objects in this particular OU? do you have 'do not recall object attributes on deprovision' checkbox enabled? I can continue to count these questions for couple more hours :D 7. finally, you're restoring that object from a recycle bin and enabling it back in FIM, join rule does its job and everyone is happy. for at least an hour :)
December 17th, 2010 6:55am

never thought about such things. I'll have 10 days off in January, so may be... actually it was caused by real life's practice: 1. imaging that you have an application role that grants access to something, e.g. 'HR manager' in Axapta. 2. one day you've migrated from Axapta to new Dynamics AX with its own roles 3. you don't need old 'HR manager' any more 4. to remove a role from business application catalog you have a proper workflow and everyone has approved that role removal (actually disable checkbox) 5. AD group is removed and remains in AD recycle bin. 6. in a week or two you get an incident that some 'smart' people from another department were using this group for something else. and they need its membership back. what would you do? restore a group from the recycle bin? enable it back on the FIM portal so the new group with another objectSID will be created? do you have a MV extension rule to delete all unnecessary objects in this particular OU? do you have 'do not recall object attributes on deprovision' checkbox enabled? I can continue to count these questions for couple more hours :D 7. finally, you're restoring that object from a recycle bin and enabling it back in FIM, join rule does its job and everyone is happy. for at least an hour :)
Free Windows Admin Tool Kit Click here and download it now
December 17th, 2010 6:55am

never thought about such things. I'll have 10 days off in January, so may be... 10 days would do the trick :o) Well, let me know if you need help with this. This could become something like "Methods to automate the clean-up of orphaned EREs". So, your implementation would be covered next to others including the discussion we had on this thread about pros and cons. Cheers, MarkusMarkus Vilcinskas, Knowledge Engineer, Microsoft Corporation
December 17th, 2010 1:04pm

never thought about such things. I'll have 10 days off in January, so may be... 10 days would do the trick :o) Well, let me know if you need help with this. This could become something like "Methods to automate the clean-up of orphaned EREs". So, your implementation would be covered next to others including the discussion we had on this thread about pros and cons. Cheers, MarkusMarkus Vilcinskas, Knowledge Engineer, Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
December 17th, 2010 1:04pm

http://social.technet.microsoft.com/wiki/contents/articles/how-to-avoid-orphaned-expectedruleentry-objects.aspx my English is less than perfect, so feel free to edit it :)
December 28th, 2010 9:45am

Wonderful, Evgeniy, thanks a lot! Could you please do one more thing: How do I post a Wiki article announcement I want to make sure that the community can find your article. Let me know if you have questions about the announcement article. Cheers + Thanks, MarkusMarkus Vilcinskas, Knowledge Engineer, Microsoft Corporation
Free Windows Admin Tool Kit Click here and download it now
December 28th, 2010 9:54am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics