Exchange 2013 w/2007 co-existence/ CU2 v1 Uninstall removes Public Folders, Exchange Security Groups Items, then prepareAD duplicates Exchange Security Group items

Written by Noel Dorobek August 2013

I spent 7 weeks on the phone with MS regarding these issues. I ultimately resolved myself. Here are the steps I took to resolve all of the above issues. I have written this up to save anyone else the fumbling. No two environments are alike, but hopefully this information is helpful to someone.

I Installed exchange 2013 RTM, Then applied CU1, and the first release of Cu2. During the CU2 upgrade the process failed. Couldnt continue. According to technet the issue I faced could easily be resolved if I uninstalled Exchange and Installed Exchange directly from CU2. I uninstalled Exchange & noticed that our public folders, meeting rooms, and the like were removed. (Verified by looking at the Default naming context\ms exchange All Exchange Security Groups in AD were also removed. Running co-existence with Exchange 2007 so without the security groups it is now down, and as AD permissions are now hosed, the services wont start, with access denied errors, and missing guids errors in exchange gui.

**Note - All the following steps assume healthy DCs & AD replication, and that you have sufficient permissions to perform the steps. (rule of trade run as admin FTW)

Exchange System Objects gone after Exchange 2013 CU2 v1 Uninstall.

1.        Get on a DC (fsmo role schema master preferred), and set it to stop incoming replication. (repadmin /options DCname +DISABLE_INBOUND_REPL)
2.        Drop to Safe Mode & select Directory Restore Mode. I used a Symantec backup to restore AD.
3.        Do a authoritative restore on the Default Naming Context\Microsoft Exchange System Objects. This will get your public folders, calendars, resources restored.
4.        Bring the DC back up, and repadmin /syncall. Wait a good 10minutes before going forward while everything syncs. (verified by checking the restored AD object on other DCs and verifying your removed items are back)

Exchange Security Groups Gone after Exchange 2013 CU2v1 Uninstall.

Now we have to correct the missing Exchange Security Groups & Permissions in AD. Before Exchange 2007 will work & before we can go back to Exchange 2013. AD now has permissions and item attributes corrupt/missing/wrong.

1.        First open ADSIEdit & connect to the default Naming context.
2.        Go to the properties of the MS Exchange Security groups & MS Exchange System Objects. Make sure you have no unknown (guid/ssid) items. If you have any remove them. Make sure any inherited unknown items are removed from the location they are inherited from. Make sure to check every folder from MS Exchange * Down.
3.        Now in ADSI edit connect to the Configuration Container\Services\MS Exchange\*enterprise name*\Administrative Groups\*admingroup name*\servers\*exchange server name*. Make sure you have no unknown (guid/ssid) items. If you have any remove them. Make sure any inherited unknown items are removed from the location they are inherited from. Make sure to check every folder from MS Exchange * Down.

That last step will leave orphaned attributes in the MS Exchange Security Group object. You cannot edit these attributes in adsiedit. You will have to use ldp.exe (built into windows). -= CAUTION: LDP.exe is not to be played with, you can get in real trouble using this tool if not very conscious and cautious=-

*NOTE - Exchange prepareAD is what will give us all our exchange security group items back, but exchange perpareAD will choke and wont complete a run complaining of the following missing items items unless removed first. 

1.        Open ldp.exe on the same DC the above steps were performed. Connect to the local DC, Bind with credentials, View select Tree & give the BaseDN of your default naming context. (DC=domain,DC=suffix)
2.        Locate then double click the MS Exchange Security Groups object & find the otherWellKnownObjects attribute. You will see the deleted Group Objects listed out here. These all have to be removed. Copy them out and make a list.
3.    Right click the MS Exchange Security Groups object & select Modify.

In the Edit Entry Attribute:  type otherWellKnownObjects In the Values Field you will paste the objects one at a time here. In the Operation make sure you click the Delete button.

Example: B:32:C262A929D691B74A9E068728F8F842ED:CN=Organization Management1\0ADEL:ddf8a0b2-7683-4af5-9533-fa003645c879,CN=Deleted Objects,DC=Domain,DC=suffix

1.    Now copy the listed objects (example above) one by one into the Values Field, and click RUN. This will remove the attribute.
2.    Open Admin CMD prompt & repadmin /syncall (wait about 10min to do next steps, if a lot of DCs or slow replication, wait longer.)

Now all remnants of the old deleted groups are gone from AD. We will now propagate the Exchange Security Groups again.

1.    On Exchange 2013 from an Admin CMD Prompt Run

setup.exe /prepareAD /iacceptexchangeserverlicenseterms

setup.exe /prepareschema /iacceptexchangeserverlicenseterms

1.    This will give you back your Exchange 2007 & 2013 Security Groups. Likely we just duplicated groups. Example: Exhange Organization Admin, and an Exchange Organization Admin1 will simultaneously exist, and so on down the list. I actually had 3 of each group.
2.    If your exchange services have been down. The server can now be brought up. Restart the Exchange AD Topology Service on the Exchange server. You may need to mount your store manually depending how bad it has been.

Exchange Security Groups are Duplicated after Exchange perpareAD

After this step I had all my exchange security group duplicate example - Ex Org Admin, Ex Org Admin1, and Ex Org Admin2. No matter how many are loaded, the last will usually be the one exchange will use (Admin2 from my example above). You can verify this by launching your current exchange server, and looking in the ORG Management Permissions tab, which groups are associated with Exchange.

Now to remove the duplicates & get in position to do 2013 install from CU2 version 2. Unfortunately we will have to perform the above steps again, for the unused exchange security group removal.

1.    Check what group current Exchange is using from above.
2.    Open AD Users & Computers. Delete the Groups Exchange is NOT using. Rename the groups if it is admin1 and so on back to the default with no number, if you like. I did for cosmetics.
3.    Now we have both unknown permission accounts, and orphaned object attributes from above (steps 5-12). So we need to run through steps 5-12, making sure during the ldp.exe that we only remove the accounts we deleted in step17 from the otherWellKnownObjects.

Your current server is now healthy, but we have some work to do for the reinstall of Exchange 2013 CU2v2.

1.    Install Exchange 2013 Cu2 version 2

**NOTE - Unfortunately we will likely get MANY errors (You dont have sufficient permissions opening exchange shell) due to RBAC being torn up now. The install of Exchange 2013 will fail likely on step 11 of 15 due to AD item attribute issues in the arbitration mailboxs of Exchange 2013.

Correct Arbitration Mailbox issues

1. Likely the Exchange 2013 install has installed the mail database before you receive these errors. You can verify by checking adsiedit Configuration\Services\ Microsoft Exchange\*enterprisename*\Administrative Groups\*exchange administrative group name*\Databases\ If you have one it did. Copy that whole string backwards from DB up. Example:  CN=Mailbox Database 0947437051,CN=Databases,CN=Exchange Administrative Group (blah),CN=Administrative Groups,CN=ENTERPRISE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=suffix
   1.    Now we need to locate and correct the arbitration mailboxs in AD Users & Computers, make a list of them. Arbitration Mailbox example. (these can be found in default naming context\rootdomain\Users by default.) FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95gb182 SystemMailbox bb558c35-97f1-4cb9-8ff7-hg741dc9289 SystemMailbox*****and so on DiscoverySearchMailbox******and so on Migration******and so on Healthmailbox*****and so on
   2.    In ADSIEdit connect to the Default Naming Context, and open the Users container. Locate the account you collected in the last step. Right click them and select properties.
   3.    Scroll through the attributes tab till you find the HomeMDB Attribute. The value should be your exchange 2013 database path from above (step20). If it is blank or not pointing at the proper database. Correct that now for all the arbitration mailboxs.

Correct RBAC

1.    Run a powershell session as admin
2. Get-ManagementRoleAssignment
3. Add-PSSnapin *setup
4. Install-CannedRbacRoleAssignments -InvocationMode Install Verbose
5. Remove-PSSnapin *setup
6. Get-ManagementRoleAssignment

RBAC is now healthy, Group Permissions are now Correct.

You should now be able to open the exchange management shell & complete the install of Exchange 2013.

Issues should now all be resolved.

Great  ...... !!

Very Helpful ...

Thanks !!!

I hoped it would be helpful to someone, thanks!

Noel