Introducing the Lingering Object Liquidator

Hi all, Justin Turner here —it's been a while since my last update. The goal of this post is to discuss what causes lingering objects and show you how to download, and then use the new GUI-based Lingering Object Liquidator (LOL) tool to remove them. This is a beta version of the tool, and it is currently not yet optimized for use in large Active Directory environments.

This is a long article with lots of background and screen shots, so plug-in or connect to a fast connection when viewing the full entry. The bottom of this post contains a link to my AD replication troubleshooting TechNet lab for those that want to get their hands dirty with the joy that comes with finding and fixing AD replication errors.

Overview of Lingering Objects

Lingering objects are objects in AD than have been created, replicated, deleted, and then garbage collected on at least the DC that originated the deletion but still exist as live objects on one or more DCs in the same forest. Lingering object removal has traditionally required lengthy cleanup sessions using tools like LDP or repadmin /removelingeringobjects. The removal story improved significantly with the release of repldiag.exe. We now have another tool for our tool belt: Lingering Object Liquidator. There are related topics such as “lingering links” which will not be covered in this post.

Lingering Objects Drilldown

The dominant causes of lingering objects are

1. Long-term replication failures
While knowledge of creates and modifies are persisted in Active Directory forever, replication partners must inbound replicate knowledge of deleted objects within a rolling Tombstone Lifetime (TSL) # of days (default 60 or 180 days depending on what OS version created your AD forest). For this reason, it is important to keep your DCs online and replicating all partitions between all partners within a rolling TSL # of days. Tools like REPADMIN /SHOWREPL * /CSV, REPADMIN /REPLSUM and AD Replication Status should be used to continually identify and resolve replication errors in your AD forest.

2. Time jumps
System time jump more than TSL # of days in the past or future can cause deleted objects to be prematurely garbage collected before all DCs have inbound replicated knowledge of all deletes. The protection against this is to ensure that :

    1. your forest root PDC is continually configured with a reference time source (including following FSMO transfers
    2. All other DCs in the forest are configured to use NT5DS hierarchy
    3. Time rollback and roll-forward protection has been enabled via the maxnegphasecorrection and maxposphasecorrection registry settings or their policy-based equivalents.

The importance of configuring safeguards can't be stressed enough. Look at this post to see what happens when time gets out of whack.

3. USN Rollbacks

USN rollbacks are caused when the contents of an Active Directory database move back in time via an unsupported restore. Root causes for USN Rollbacks include:

  • Manually copying previous version of the database into place when the DC is offline
  • P2V conversions in multi-domain forests
  • Snapshot restores of physical and especially virtual DCs. For virtual environments, both the virtual host environment AND the underlying guest DCs should be Virtual Machine Generation ID capable. Windows Server 2012 or later. Both Microsoft and VMWARE make VM-Generation ID aware Hyper-V host.

Events, errors and symptoms that indicate you have lingering objects
Active Directory logs an array of events and replication status codes when lingering objects are detected. It is important to note that while errors appear on the destination DC, it is the source DC being replicated from that contains the lingering object that is blocking replication. A summary of events and replication status codes is listed in the table below:


Event or Error status

Event or error text

Implication

AD Replication status 8606

"Insufficient attributes were given to create an object. This object may not exist because it may have been deleted."

Lingering objects are present on the source DC (destination DC is operating in Strict Replication Consistency mode)

AD Replication status 8614

The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime.

Lingering objects likely exist in the environment

AD Replication status 8240

There is no such object on the server

Lingering object may exist on the source DC

Directory Service event ID 1988

Active Directory Domain Services Replication encountered the existence of objects in the following partition that have been deleted from the local domain controllers (DCs) Active Directory Domain Services database.

Lingering objects exist on the source DC specified in the event

(Destination DC is running with Strict Replication Consistency)

Directory Service event ID 1388

This destination system received an update for an object that should have been present locally but was not.

Lingering objects were reanimated on the DC logging the event

Destination DC is running with Loose Replication Consistency

Directory Service event ID 2042

It has been too long since this server last replicated with the named source server.

Lingering object may exist on the source DC

A comparison of Tools to remove Lingering Objects

The table below compares the Lingering Object Liquidator with currently available tools that can remove lingering objects

Removal method

Object / Partition & and Removal Capabilities

Details

Lingering Object Liquidator

Per-object and per-partition removal

Leverages:

  • RemoveLingeringObjects LDAP rootDSE modification
  • DRSReplicaVerifyObjects method

  • GUI-based.
  • Quickly displays all lingering objects in the forest to which the executing computer is joined.
  • Built-in discovery via DRSReplicaVerifyObjects method
  • Automated method to remove lingering objects from all partitions
  • Removes lingering objects from all DCs (including RODCs) but not lingering links.

Repldiag /removelingeringobjects

Per-partition removal

Leverages:

  • DRSReplicaVerifyObjects method

  • Command line only
  • Automated method to remove lingering objects from all partitions
  • Built-in discovery via DRSReplicaVerifyObjects
  • Displays discovered objects in events on DCs
  • Does not remove lingering links. Does not remove lingering objects from RODCs (yet)

LDAP RemoveLingeringObjects rootDSE primative (most commonly executed using LDP.EXE or an LDIFDE import script)

Per-object removal

  • Requires a separate discovery method
  • Removes a single object per execution unless scripted.

Repadmin /removelingeringobjects

Per-partition removal

Leverages:

  • DRSReplicaVerifyObjects method

  • Command line only
  • Built-in discovery via DRSReplicaVerifyObjects
  • Displays discovered objects in events on DCs
  • Requires many executions if a comprehensive (n * n-1 pairwise cleanup is required. Note: repldiag and the Lingering Object Liquidator tool automate this task.

The Repldiag and Lingering Object Liquidator tools are preferred for lingering object removal because of their ease of use and holistic approach to lingering object removal.

Why you should care about lingering object removal

Widely known as the gift that keeps on giving, it is important to remove lingering objects for the following reasons

  • Lingering objects can result in a long term divergence for objects and attributes residing on different DCs in your Active Directory forest
  • The presence of lingering objects prevents the replication of newer creates, deletes and modifications to destination DCs configured to use strict replication consistency. These un-replicated changes may apply to objects or attributes on users, computers, groups, group membership or ACLS.
  • Objects intentionally deleted by admins or application continue to exist as live objects on DCs that have yet to inbound replicate knowledge of the deletes.

Once present, lingering objects rarely go away until you implement a comprehensive removal solution. Lingering objects are the unwanted houseguests in AD that you just can't get rid of.

Mother in law jokes… a timeless classic.

We commonly find these little buggers to be the root cause of an array of symptom ranging from logon failures to Exchange, Lync and AD DS service outages. Some outages are resolved after some lengthy troubleshooting only to find the issue return weeks later.
The remainder of this post, we will give you everything needed to eradicate lingering objects from your environment using the Lingering Object Liquidator.

Repldiag.exe is another tool that will automate lingering object removal. It is good for most environments, but it does not provide an interface to see the objects, clean up RODCs (yet) or remove abandoned objects.

Introducing Lingering Object Liquidator

 More:

Lingering Object Liquidator automates the discovery and removal of lingering objects by using the DRSReplicaVerifyObjects method used by repadmin /removelingeringobjects and repldiag combined with the removeLingeringObject rootDSE primitive used by LDP.EXE. Tool features include:

  • Combines both discovery and removal of lingering objects in one interface
  • Is available via the Microsoft Connect site
  • The version of the tool at the Microsoft Connect site is an early beta build and does not have the fit and finish of a finished product
  • Feature improvements beyond what you see in this version are under consideration

How to obtain Lingering Object Liquidator

1. Log on to the Microsoft Connect site (using the Sign in) link with a Microsoft account:

http://connect.microsoft.com

Note: You may have to create a profile on the site if you have never participated in Connect.

2. Open the Non-feedback Product Directory:

https://connect.microsoft.com/directory/non-feedback

3. Join the following program:

AD Health

Product Azure Active Directory Connection Join link

4. Click the Downloads link to see a list of downloads or this link to go directly to the Lingering Objects Liquidator download. (Note: the direct link may become invalid as the tool gets updated.)

5. Download all associated files

6. Double click on the downloaded executable to open the tool.

Tool Requirements

1. Install Lingering Object Liquidator on a DC or member computer in the forest you want to remove lingering objects from.

2. .NET 4.5 must be installed on the computer that is executing the tool.

3. Permissions: The user account running the tool must have Domain Admin credentials for each domain in the forest that the executing computer resides in. Members of the Enterprise Admins group have domain admin credentials in all domains within a forest by default. Domain Admin credentials are sufficient in a single domain or single domain forest.

4. The admin workstation must have connectivity over the same port and protocol required of a domain-joined member computer or domain controller against any DC in the forest. Protocols of interest include DNS, Kerberos, RPC, LDAP and ephemeral port range used by the targeted DC See TechNet for more detail. Of specific concern: Pre-W2K8 DCs communicate over the “low” ephemeral port between 1024 and 5000 while post W2K3 DCs use the “high” ephemeral port range between 49152 to 65535. Environments containing both OS version families will need to enable connectivity over both port ranges.

5. You must enable the Remote Event Log Management (RPC) firewall rule on any DC that needs scanning. Otherwise, the tool displays a window stating, "Exception: The RPC server is unavailable"

6. The liquidation of lingering objects in AD Lightweight Directory Services (AD LDS / ADAM) environments is not supported.

Lingering Object Discovery

To see all lingering objects in the forest:

1. Launch Lingering Objects.exe.

2. Take a quick walk through the UI:

Naming Context:

Reference DC: the DC you will compare to the target DC. The reference DC hosts a writeable copy of the partition.

Note: ChildDC2 should not be listed here since it is an RODC, and RODCs are not valid reference DCs for lingering object removal.

 More:

The version of the tool is still in development and does not represent the finished product. In other words, expect crashes, quirks and everything else normally encountered with beta software.

Target DC: the DC that lingering objects are to be removed from

3. In smaller AD environments, you can leave all fields blank to have the entire environment scanned, and then click Detect. The tool does a comparison amongst all DCs for all partitions in a pairwise fashion when all fields are left blank. In a large environment, this comparison will take a great deal of time as the operation targets (n * (n-1)) number of DCs in the forest for all locally held partitions. For shorter, targeted operations, select a naming context, reference DC and target DC. The reference DC must hold a writable copy of the selected naming context.

During the scan, several buttons are disabled. The current count of lingering objects is displayed in the status bar at the bottom of the screen along with the current tool status. During this execution phase, the tool is running in an advisory mode and reading the event log data reported on each target DC.

Note: The Directory Service event log may completely fill up if the environment contains large numbers of lingering objects and the Directory Services event log is using its default maximum log size. The tool leverages the same lingering object discovery method as repadmin and repldiag, logging one event per lingering object found.

When the scan is complete, the status bar updates, buttons are re-enabled and total count of lingering objects is displayed. The log pane at the bottom of the window updates with any errors encountered during the scan.
Error 1396 is logged if the tool incorrectly uses an RODC as a reference DC.
Error 8440 is logged when the targeted reference DC doesn't host a writable copy of the partition.

 Note:

Lingering Object Liquidator discovery method

  • Leverages DRSReplicaVerifyObjects method in Advisory Mode
  • Runs for all DCs and all Partitions
  • Collects lingering object event ID 1946s and displays objects in main content pane
  • List can be exported to CSV for offline analysis (or modification for import)
  • Supports import and removal of objects from CSV import (leverage for objects not discoverable using DRSReplicaVerifyObjects)
  • Supports removal of objects by DRSReplicaVerifyObjects and LDAP rootDSE removeLingeringobjects modification

The tool leverages the Advisory Mode method exposed by DRSReplicaVerifyObjects that both repadmin /removelingeringobjects /Advisory_Mode and repldiag /removelingeringobjects /advisorymode use. In addition to the normal Advisory Mode related events logged on each DC, it displays each of the lingering objects within the main content pane.

Details of the scan operation log in the linger.log.txt file in the same directory as the tool's executable.

The Export button allows you to export a list of all lingering objects listed in the main pane into a CSV file. View the file in Excel, modify if necessary and use the Import button later to view the objects without having to do a new scan. The Import feature is also useful if you discover abandoned objects (not discoverable with DRSReplicaVerifyObjects) that you need to remove. We briefly discuss abandoned objects later in this post.

Removal of individual objects

The tool allows you to remove objects a handful at a time, if desired, using the Remove button:

1. Here I select three objects (hold down the Ctrl key to select multiple objects, or the SHIFT key to select a range of objects) and then select Remove.

The status bar updates with the new count of lingering objects and the status of the removal operation:

Logging for removed objects

The tool dumps a list of attributes for each object before removal, and logs this along with the results of the object removal in the removedLingeringObjects.log.txt log file. This log file is in the same location as the tool's executable.

C:\tools\LingeringObjects\removedLingeringObjects.log.txt

the obj DN: <GUID=0bb376aa1c82a348997e5187ff012f4a>;<SID=010500000000000515000000609701d7b0ce8f6a3e529d669f040000>;CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com

objectClass:top, person, organizationalPerson, user;
sn:Schenk ;
whenCreated:20121126224220.0Z;
name:Dick Schenk;
objectSid:S-1-5-21-3607205728-1787809456-1721586238-1183;primaryGroupID:513;
sAMAccountType:805306368;
uSNChanged:32958;
objectCategory:<GUID=11ba1167b1b0af429187547c7d089c61>;CN=Person,CN=Schema,CN=Configuration,DC=root,DC=contoso,DC=com;
whenChanged:20121126224322.0Z;
cn:Dick Schenk;
uSNCreated:32958;
l:Boulder;
distinguishedName:<GUID=0bb376aa1c82a348997e5187ff012f4a>;<SID=010500000000000515000000609701d7b0ce8f6a3e529d669f040000>;CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com;
displayName:Dick Schenk ;
st:Colorado;
dSCorePropagationData:16010101000000.0Z;
userPrincipalName:Dick@root.contoso.com;
givenName:Dick;
instanceType:0;
sAMAccountName:Dick;
userAccountControl:650;
objectGUID:aa76b30b-821c-48a3-997e-5187ff012f4a;
value is :<GUID=70ff33ce-2f41-4bf4-b7ca-7fa71d4ca13e>:<GUID=aa76b30b-821c-48a3-997e-5187ff012f4a>
Lingering Obj CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com is removed from the directory, mod response result code = Success
———————————————-
RemoveLingeringObject returned Success

Removal of all objects

The Remove All button, removes all lingering objects from all DCs in the environment.

To remove all lingering objects from the environment:

1. Click the Remove All button. The status bar updates with the count of lingering objects removed. (the count may differ to the discovered amount due to a bug in the tool-this is a display issue only and the objects are actually removed)

2. Close the tool and reopen it so that the main content pane clears.

3. Click the Detect button and verify no lingering objects are found.


Abandoned object removal using the new tool

None of the currently available lingering object removal tools will identify a special sub-class of lingering objects referred to internally as, "Abandoned objects".

An abandoned object is an object created on one DC that never got replicated to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition.

The lingering object liquidator tool does not currently discover abandoned objects automatically so a manual method is required.

1. Identify abandoned objects based on Oabvalidate and replication metadata output.

Abandoned objects can be removed with the LDAP RemoveLingeringObject rootDSE modify procedure, and so Lingering Objects Liquidator is able to remove these objects.

2. Build a CSV file for import into the tool. Once, they are visible in the tool, simply click the Remove button to get rid of them.

a. To create a Lingering Objects Liquidator tool importable CSV file:

Collect the data in a comma separated value (CSV) with the following data:

FQDN of RWDC

CNAME of RWDC

FQDN of DC to remove object from

DN of the object

Object GUID of the object

DN of the object's partition

3. Once you have the file, open the Lingering Objects tool and select the Import button, browse to the file and choose Open.

4. Select all objects and then choose Remove.

Review replication metadata to verify the objects were removed.

Resources

For those that want even more detail on lingering object troubleshooting, check out the following:

To prevent lingering objects:

  • Actively monitor for AD replication failures using a tool like the AD Replication Status tool.
  • Resolve AD replication errors within tombstone lifetime number of days.
  • Ensure your DCs are operating in Strict Replication Consistency mode
  • Protect against large jumps in system time
  • Use only supported methods or procedures to restore DCs. Do not:
    • Restore backups older than TSL
    • Perform snapshot restores on pre Windows Server 2012 virtualized DCs on any virtualization platform
    • Perform snapshot restores on a Windows Server 2012 or later virtualized DC on a virtualization host that doesn't support VMGenerationID

If you want hands-on practice troubleshooting AD replication errors, check out my lab on TechNet Virtual labs. Alternatively, come to an instructor-led lab at TechEd Europe 2014. "EM-IL307 Troubleshooting Active Directory Replication Errors"

For hands-on practice troubleshooting AD lingering objects: I'll be presenting instructor-led lab sessions at TechEd Europe 2014. "EM-IL400 Troubleshooting Active Directory Lingering Objects"

Finally, if you would like access to a hands-on lab for in-depth lingering object troubleshooting; let us know in the comments.

Thank you,

Justin Turner and A. Conner


Ask the Directory Services Team

{ Comments on this entry are closed }

New update available for Azure Backup for Microsoft Azure Recovery Services Agent

September 15, 2014

Today we released an article describing an update for the Microsoft Azure Recovery Services Agent that is used both by Microsoft Azure Backup and the Microsoft Azure Site Recovery service to transport data to Azure. For details regarding the new features and reliability issues addressed in this update, please see the following: KB2997692 – Update […]

Read the full article →

Miracast in Enterprise Environments

September 15, 2014

This blog is intended to document our learning's about Miracast technology and explain things to consider when implementing a solution using Miracast. It is primarily about Windows 8.1 tablets and Windows Phone 8.1 as the authors work for Microsoft in the Worldwide Modern Devices Centre of Excellence (CoE). We work on a program called First […]

Read the full article →

First set of Android apps coming to a Chromebook near you

September 15, 2014

Chromebooks were designed to keep up with you on the go—they’re thin and light, have long battery lives, resume instantly, and are easy to use. Today, we’re making Chromebooks even more mobile by bringing the first set of Android apps to Chrome OS: Duolingo – a fun and free way to learn a new language […]

Read the full article →

A Tale Of Two Apps

September 14, 2014

 Sometimes multiple apps adopt similar ideas and designs as part of some peculiar cultural zeitgeist. But things get murkier when ideas become shared and adopted by long-time friends riffing on similar concepts. That appears to be what happened to Kevin Rose and Danny Trinh, who built separate photo-sharing apps that ended up looking eerily similar. […]

Read the full article →

Windows 9′s new Start menu demonstrated on video

September 14, 2014

Microsoft may have demonstrated its new Start menu earlier this year, but thanks to a recent “Windows 9″ leak we’re now seeing every single part of the company’s plans for bringing back this popular feature. German site WinFuture has posted a two-minute video that demonstrates how the Start menu works in the next major release […]

Read the full article →

New VM Images Optimized for Transactional and DW workloads in Azure VM Gallery

September 14, 2014

We are delighted to announce the release of new optimized SQL Server images in the Microsoft Azure Virtual Machines Gallery. These images are pre-configured with optimizations for transactional and Data Warehousing workloads respectively by baking in our performance best practices for running SQL in Azure VMs. What preconfigured VM images are available? The following four […]

Read the full article →

Miracast in Enterprise Environments – FAQ

September 14, 2014

Intro This blog is intended to document our learning's about Miracast technology and explain things to consider when implementing a solution using Miracast. It is primarily about Windows 8.1 tablets and Windows Phone 8.1 as the authors work for Microsoft in the Worldwide Modern Devices Centre of Excellence (CoE). We work on a program called […]

Read the full article →

Q&A With Fabian Uhse, Program Manager for Work Folders in Windows Server 2012 R2

September 13, 2014

Hi Folks – A few months back I wrote a blog article on Work Folders, one of the new “hero” features in Windows Server 2012 R2 and Windows Storage Server 2012 R2. In this post, I’ll share the perspective of Fabian Uhse, a Program Manager for Work Folders at Microsoft. Fabian Uhse Q: Why did […]

Read the full article →

FAQ – Azure IaaS workload protection using Data Protection Manager

September 13, 2014

Last week we announced the support for System Center Data Protection Manager (DPM) running in an Azure IaaS VM, for protecting workloads in Azure. The overwhelming positive response has been accompanied by deep-dive questions and a request for more details about certain aspects. This blog post is a list of Frequently Asked Questions to help […]

Read the full article →