I look after a relatively small server 2003 based domain (3 Sites / 3 DCs with around 20 clients at each site). Now on one of my DCs drive space on C: is getting very tight (<300Mb). I have noticed that the NTDS.DIT file is over 4.3Gb, surely that is larger than expected? Though saying that my other 2 DCs have large (>4Gb) files themselves.I have done some googling and have seen people saying theirs is under 100Mb with far far more clients and objects than my domain.Is there any easy way to reduce the size of it. I have seen some mention of performing an offline defragmentation but to be honest I am an interested amatuer with AD rather than a fully trained profesional and would like an easier option if at all possible. I manage OK on a day to day basis but am not that confident messing with major that I really dont understand at all.Could I for example just dcpromo the DC away from being a DC and then promote it straight back again to force a rebuild of the file? Or is that too simplistic.Thanks

that's a pretty large database for your size environment. to be able to tell whether you are going to get that down with an offline defrag or a dcpromo (both valid methods to recover whitespace) we have to know how much whitespace there is in the DIT to begin with. to find this out, set 'garbage collection' to '1' (http://technet.microsoft.com/en-us/library/cc787136(WS.10).aspx). at the next online defrag, a 1646 event will be generated in the DS event log telling you how much whitespace you have. the fact that you are up around 4 GB makes me think that something unorthodox is being stored in the directory and i wouldn't be surprised to see that there isn't as much whitespace as we'd like. if that's the case, we can perform an inventory of our object classes to see what we have. hth. /richhttp://cbfive.com/blog

I agree with Rich. I would tend to think there are something unusual being stored in the directory as well based on the size of this environment unless there was alot of testing done in this domain prior to it going production where scripts were used to create tens of thousands of object and then deleted.....hmmm... probably not. Visit my blog: anITKB.com, an IT Knowledge Base.

Thanks for the replys guys.I will have a look at seeing how much whitespace there is when I am back in work on Monday. The domain has had a rather checkered history and as far as I know was originally a NT 4 domain before being upgraded or migrated to 2k3. I suspect there is a lot of unused space though as every year I have to promote a server on a fourth site to a DC for one month only before I demote it again after everyone leaves. It only takes around an hour or 2 to pull all the AD info over a 1Mb leased line so cant believe that one is >4Gb also. (I do this because for 11 months of the year no one is there and the link to it is cut, one time I forgot some tombstone life had expired which caused me problems getting the server back up again on the domain)Overall though, assuming there is a lot of whitespace, what would be the easiest (and safest method). The offline defrag or a couple of dcpromo down then up again? bear in mind that this DC is my master DC and GC. Getting downtime is no problem as at the moment there are only around 4 users at the site so time taken for each task is not an issue for me.

I would prefer the offline defragmentation process. Its safe and easy. The DCPROMO process could fail during the process causing other issues that may require you to go back and clean up the NTDSutil file. Visit my blog: anITKB.com, an IT Knowledge Base.

let me say first off that if you were to consult msft, i am almost positive that the majority of the engineers that you would talk with would recommend that 1) you perform regular offline defrags to recover white space (say once a year) and 2) that you should perform an offline defrag to recover this space. so i think that it is inline with Jorge's recommendation. for me personally, i tell my customers to avoid offline defrags whenever possible. it is a low-level database operation that carries (minimal) risk of failure. wherever there is risk, i think that you have to ask, what is the reward. and the answer to that is only the recovery of disk space. so, the only time that i personally feel the risk-reward balance is in favor of offline defrags is when you're almost out of space on the database volume. since that happens to be your condition. i guess we have to recover space (if it's there). if i were recovering that space, i think i would consider a dcpromo (and may likely go that route). that's because i have seen more failures in my experience with offline defrags than with dcpromo. but there is overhead to dcpromo and to be fair, i have done an incalculable number of dcpromos and have built up some kind of technical muscle memory that reminds me of exactly what to look for to validate that it is working the way it should. so, you are probably better off using ntdsutil to do an offline defrag as there is a lot of good guidance on how to do that. http://support.microsoft.com/kb/232122 also, just so you can get an idea of sizing per object i included the link below. you may get a chuckle out of it. http://technet.microsoft.com/en-us/library/cc961779.aspx hth /rich http://cbfive.com/blog

Very well described Rich. In summary, both operations carry risk and both provide benefit. Not sure if anyone has calculated which operation is more likely to cause a problem for you, but you just may want to think about it in the case where you do run into an issue, which are you more likely to be able to handle?Remember, we are only introducing possibilities that really rarely happen. I have run both types of operations countless times and the majority of the time, each run without incident. I only mention that there could be an issue because I have experienced failure on both operations in the past, its just rare.There is alot of documentation out there about these procedures so I wouldnt get hung up and loose sleep over which decision to make.The end result is that you need to recover disk space. Failure to act will probably cause an interruption of service so, get to it... Visit my blog: anITKB.com, an IT Knowledge Base.

completely agree. well said.http://cbfive.com/blog

Looks like I have some work on next week then. I will get it done early in the week before I get an influx of users arriving back on Wed.Reading up on it I think I will go with the defrag (if after checking whitespace its required).One question though. If the defrag goes pear shaped can I just copy back the original copy of the large database that I save just prior to defrag? or would I have to perform a system restore. A restore is no problem (its backed up daily anyway) but it would be easier just to drag a copy of the DIT straight back from another volume if it all goes wrong.Thanks for all you help. If all goes well next week it will be good. If not I may be back to these forums asking for help on recovering a corrupt database.......

Well looks like I wont be bothering with an offline defrag anyway.The results in from garbage collection logging as follows:Total size = 4320free = 4Somehow I dont think I will bother to do anything to recover 4Mb!I have no idea what could be taking up so much space though in such a small domain. Is there anyway to see what is in there?

Do you have any enterprise applications that may be leveraging AD for storage? A third-party password manager? How do your group policy objects look like, do you have embeded login scripts that are not being stored in the sysvol and should be?Visit my blog: anITKB.com, an IT Knowledge Base.

We do have an 3rd party password manager (HP Protect Tools). Which I am sure probably does leberage AD to store its own data.But apart from that I am pretty sure we have no other applications that would use AD to store any data. The users basically use word, outlook, powerpoint and excel and a few locally installed analytical tools such as mtstat or matlab.I checked the whitespace on the other DCs as well with similar results all were around 4Gb size and all had less than 10Mb free space.Strange that when I dcpromo a server on a rarely used site it only takes only a couple of hours to complete over a 1Mb link. If all 4Gb of data was going over the link it should take a couple of days. Or does most of that information get created locally?I read something a few minutes ago (while googling) and a guy said the largest NTDS.DIT he had ever seen on 2k3 was around 8Gb on a GC for 8 seperate domains, with over 250k users, 300k contacts etc etc. Compared to my single domain with around 200 users and 10k contacts!Edit: I just remotely logged into a completely seperate domain of a parent organisation of ours and checked the NTDS.DIT size on one of their DCs. Theirs was a little over 6Gb. They have around 800 clients with around 1200 users but apart fromthat the application loadouts etc are basically identical including the use of Hp Protect Tools. It looks like I will just have to stick with my large database and instead of shrinking it maybe think about moving it onto a volume with more space.

oh man, this is fun stuff. btw, for starters, i have (and have had) multiple customers will databases 4 GBs and above. A few of them have had databases in excess of 32 GBs so don't stress about that piece. AD is very capable of handling it. okay, so let's figure out what taking up your space. you'll want the 2003 SP2 support tools. in the support tools is a program called dsastat which is kind enough to give object counts and sizes. what i'd like for you to do is run that utility with this syntax: dsastat -s:<name of PDC> -loglevel:info -output:file -sort:TRUE. it's going to start with a lot of gobbly gook but what i'd like for you to postback to the forum is everything that starts after DSAStat Diagnostics (or something like that). basically, i want the two tables at the end that give you a count of objects on the server and the bytes allocated per object class. you can also run this in the parent domain if you'd like but i am guessing the culprit between the two will be very obvious and very similar (fingers crossed). on the dcpromo question, rpc is an efficient network protocol so it isn't that surprising that it is able to DCPROMO within several hours. since we are talking about slow DCPROMOs, out of curiosity, how big is your SYSVOL? thx /richhttp://cbfive.com/blog

RichOK am running the dsastat command now. Its been running for an hour or so already and I am guessing it will not finish before I head home in around 30 mins. So far its at around 370k entries and counting.I will leave it run overnight and post the results tomorrow morning.My SYSVOL is around 197Mb (87MB size on disk). Thats almost all policys apart from a couple of hundred Kb of scripts.One other thing that I just noticed. The size of the NTDS.DIT, on the DC with little space left, grew by almost 100Mb between Garbage Collections. It went from something like 3220 with 97Mb free then 12 hours later was at 3320 with 4Mb free. If it keeps growing like this (when i am making no network changes and not adding / deleting users) it wont be long before I am in trouble. The other DC's never grew in that time period so I am at a loss why that one did. That said I did manage to free up a couple of Gb of space on the volume by running MSIZAP Q! to delete any orphaned install files so I have at least a little time left!Also In answer to Jorge's query earlier (which I ommited to answer earlier). There are no embeded scripts outside of the SYSVOL (that I am aware of anwyay!)

I thought I was strange for thinking this stuff is fun as well.... I am very eager and curious (as I imagine others following this thread) on what is taking so much space in your db file. You must continue posting!Visit my blog: anITKB.com, an IT Knowledge Base.

Have no fear Jorge, I will continue posting hopefully until we get to the bottom of why the DB is so large.BTW my dsastat had processed around 500k entries just before I left for home and was still going strong!

OK looks like Rich was right. The culprit is very very obvious! I very much suspect that the cause of it all have been multiple corrupt Global Address Lists that have been sent to us (an external organisation sends them to us) in the last couple of weeks. This has necessitated in multiple deletions of all contacts followed by a re-import of the contacts again (at 40k Contacts each time). I guess there are now a ton of tomb stoned contacts waiting to fall out of the DB whenever they expire. We import a new contacts list weekly anyway so I guess even best case we will have a large amount of space taken up by contacts.I will check for whitespace in a couple of months and see if the deleted contacts have fallen out. If / when they do I can then perform a offline defrag to recover the space.Here are the results of the dsastat: -=>>|*** DSA Diagnostics ***|<<=- Objects per server:Obj/Svr site7-rm213-dc01 Total builtinDomain 1 1classStore 3 3computer 53 53contact 1362437 1362437container 177 177dfsConfiguration 1 1dnsNode 33 33dnsZone 2 2domainDNS 1 1domainPolicy 1 1fTDfs 2 2fileLinkTracking 1 1foreignSecurityPrincipal 4 4group 127 127groupPolicyContainer 49 49infrastructureUpdate 1 1ipsecFilter 2 2ipsecISAKMPPolicy 3 3ipsecNFA 8 8ipsecNegotiationPolicy 6 6ipsecPolicy 3 3linkTrackObjectMoveTable 1 1linkTrackVolEntry 28 28linkTrackVolumeTable 1 1lostAndFound 1 1msDFSR-Connection 30 30msDFSR-Content 5 5msDFSR-ContentSet 5 5msDFSR-GlobalSettings 1 1msDFSR-LocalSettings 7 7msDFSR-Member 15 15msDFSR-ReplicationGroup 5 5msDFSR-Subscriber 15 15msDFSR-Subscription 15 15msDFSR-Topology 5 5msDS-QuotaContainer 1 1msExchSystemObjectsContainer 1 1nTFRSMember 3 3nTFRSReplicaSet 1 1nTFRSSettings 4 4nTFRSSubscriber 3 3nTFRSSubscriptions 5 5organizationalUnit 101 101packageRegistration 1 1printQueue 8 8publicFolder 18 18rIDManager 1 1rIDSet 3 3rpcContainer 1 1samServer 1 1secret 4 4serviceConnectionPoint 4 4user 130 130---Total: 1363339 1363339 . . . . . . . . . . . . . . Bytes per object:Object BytesbuiltinDomain 165classStore 591computer 22130contact 453480508container 28935dfsConfiguration 179dnsNode 4778dnsZone 334domainDNS 1537domainPolicy 183fTDfs 268fileLinkTracking 164foreignSecurityPrincipal 756group 150631groupPolicyContainer 13408infrastructureUpdate 181ipsecFilter 680ipsecISAKMPPolicy 801ipsecNFA 2243ipsecNegotiationPolicy 2092ipsecPolicy 1178linkTrackObjectMoveTable 210linkTrackVolEntry 6032linkTrackVolumeTable 193lostAndFound 200msDFSR-Connection 6570msDFSR-Content 775msDFSR-ContentSet 839msDFSR-GlobalSettings 193msDFSR-LocalSettings 1323msDFSR-Member 3165msDFSR-ReplicationGroup 883msDFSR-Subscriber 3285msDFSR-Subscription 3345msDFSR-Topology 795msDS-QuotaContainer 204msExchSystemObjectsContainer 247nTFRSMember 1050nTFRSReplicaSet 214nTFRSSettings 756nTFRSSubscriber 1101nTFRSSubscriptions 903organizationalUnit 22169packageRegistration 255printQueue 2474publicFolder 17056rIDManager 157rIDSet 417rpcContainer 168samServer 157secret 804serviceConnectionPoint 1475user 219187 . . . . . . . . . . . . . . Bytes per server:Server Bytessite7-rm213-dc01 454008344 . . . . . . . . . . . . . .Checking for missing replies... No missing replies!INFO: Server sizes are equal.*** Identical Directory Information Trees *** -=>> PASS <<=-

good news that you have discovered the cause and have a plan.Visit my blog: anITKB.com, an IT Knowledge Base.

that is great news. looks like you 432 MB of contacts currently in the database, of which the non-deleted portion of the database is about 433 MB. i am guessing, as you are, that we have several gigs of tombstoned contacts in the deleted objects container. by default, dsastat will not show tombstoned objects. so one more thing that you can do if you'd like to get an idea of how much of this is in the deleted objects container is adjust the dsastat query like so: dsastat -s:<name of PDC> -loglevel:info -output:file -sort:TRUE -filter:"(&(objectClass=contact)(isDeleted=TRUE))" this query should run a little bit faster than yesterday's and should give you an idea of how much is hanging out in Deleted Objects. in the end, you likely know what it's going to roughly look like but it may be interesting data to have. btw, what's your TSL? /rich http://cbfive.com/blog

RichOut of interest I will run the extra dsastat. Like you say though I can pretty much guess what the result will be.I have not changed the default TSL. I am running 2k3 SP2 so presumably its set to 180 days? To be honest though I dont know how to check that for sure, presumably with a tool like adsiedit but I woudnt know how to!Regardless of the outcome though I must thank you guys again for helping me figure it all out.

my pleasure.yep, you can check it out in ADSIEdit. just look at the value of the tombstoneLifetime attribute on the 'CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,DC=<domain>,DC=<tld>' object. /richhttp://cbfive.com/blog

Hmm not exactly sure the deleted contacts are taking up as much space as I thought. The dsastat showed around 1300000 deleted contact objects (which is around 40k less than the original dsastat and looks right as I have around 40k active contacts) but only shows 402Mb space taken up by those deleted contacts. I am probably just reading it wrongly but was expecting more like 4Gb not 400Mb. I have had to perform two more contacts delete / re-import this week and have seen my DB grow by another 200+MB on top of its already large size so still think that tombstoned contacts are the cause of the large DB.My TSL is 'not set' by the way. Does that means its at its default (180 days for 2k3 r2 SP2) presumably it doesnt means there is no value set at all and nothing is ever deleted?Regards

Was definately the tombstoned contacts causing my problem. I have changed the way we import new contacts which has now reulted in my database falling below 1GB (from a peak of 4.4Gb). I have now performed an offline defrag to recover space on the DC that had almost no space left. It probably will get even smaller over time but I have enough space on the volume now to not worry about it. Anyway I know I have dragged this up from a while ago but wanted to thank Rich and Jorge for their previous help

my pleasure. i'm glad to hear that you found the root cause. and i'll same i'm even happier to hear that you're changing the process. that should help quite a bit.http://cbfive.com/blog