Hi,

I noticed that many files were missing when searching for them on my Server 2012 Fileserver.

After some troubleshooting I noticed that the ones missing, had a Size on Disk of 4KB and the "SparseFile" and "ReparsePoint" (PL) Flags set.

So it looks like they were processed by the enabled Data Deduplication.

Am I missing something here or is it really the case that deduplicated files cannot be indexed by windows search?

• Changed type Shaon ShanMicrosoft contingent staff, Moderator Wednesday, March 13, 2013 7:08 AM by design

Hi,

It seems that Junction Points will not be indexed, so I'm thinking if Data Deduplication is supported by Windows Search... Will do more research on this...

Hi,

It seems that Junction Points will not be indexed, so I'm thinking if Data Deduplication is supported by Windows Search... Will do more research on t

Hi Shaon.

I am also interested in this.

In last couple of years - I have "obsoleted" use of drive letter. That's why we got mount point's in the first place to mount and virtualize those file hierarchies. I.e. most of us are not using drive letters and "network-based" harddrives - but networked file hierarchies using shares.

So I just ran into the same issue here on a test system. Made a share "...\Shares" and mounted those file hierarcies below. 5 resparse points.

Windows 7, 8 libraries advertised the relocated shares could not be added to file liberaries unless indexed.

So let's go add those folders.

Reminder and bad news: Indexing complete in 1 sec. Do not follow reparse points.

Very bad if this is true. It can not be we still have to use volumes mounted to drive letters only! Especially, now Windows Storage Spaces are really encouraging the use of volumes. I just recall why I had trouble using volumes in the first place ... Windows Search not working with resparse points. Because I can not just keep assigning drive letters to volumes. Simply running out ... THIS SOULD BE A HIGH PRIORITY ISSUE BECAUSE WINDOWS STORAGE SPACE AND VOLUMES HAVE COME ARROUND NOW. Before you would may be have a few very large volumes to deal with this issue of bad support in Windows Desktop Search for file and storage virtualization in Windows.

We can not use liberaries in Windows 8 without Windows Search ON THE SERVER having indexed those remote files. File hierarchies in the Windows client are functionally dependent on Windows Desktop Search to work.

• Edited by Computermensch Friday, March 08, 2013 12:52 PM

Just tested and can confirm resparse points does not work in Search indexing whether hard links, junctions or symbolic links ...

mklink /d test \\?\Volume{1cf32c52-15c0-497f-91a0-7321ebadbd0a}\

mklink /j test \\?\Volume{1cf32c52-15c0-497f-91a0-7321ebadbd0a}\

... can not browse in WDS: Control Panel>Index>Modify>browse folders ...

• Edited by Computermensch Friday, March 08, 2013 9:04 AM

BTW! "... and mounted those file hierarcies below. 5 reparse points" so I my case I had consolidated the users Space into 5 overall concepts on 1 volume each. In that way I had already reduced my use of volumes.

So for now I can add a drive letter to each for trying to make Windows Search work.

But guess I am going to stumble into the same issue then with Windows Search not supporting resparse points again ....because some of the volumes are deduped.

Great experience as ever working with file virtualization in the Windows client, i.e. Liberaries. Just wonder why the heck MS did not improove that experience ...

Right now, with Server 2012, find myself "hacking" again this year 2013 ... to try and make Windows work on some basic issues. All the Flex is starting to go away now ... here comes reality ... NOT WORKING.

NTFS resparse points, long file names, etc. Great Flex! However in theory only. For real when everything is pulled together and the system receives its more or less complete configuration ... unsupported in Windows. Go hack it.

Please stop doing just components test ... and do complete flight tests also. More hollistic tests please across constraints! With a focus in each test ... from 0 to a fully configured system.

How can no support in Windows Search for resparse points in NTFS be a miss during introduction of libraries (Windows 7/2008 R2) and now during introduction of data deduplication (Windows 8/2012) ... during just internal tests at MS? And we all have to use libraries (thus a remote Windows Search index on remote shares in a domain setup) ... and now Windows 8 WinRT is even more dependent on libraries? How can that be overlooked - or such a basic thing receive no priority - and just get skipped? How can that happen?

• Edited by Computermensch Friday, March 08, 2013 12:52 PM

Now go and tell the WDS team to fix this compatibility with resparse points getting more and more in heavy use. If no mount points or similar flex in ReFS - then ReFS is a no go for me ... it's reliability AND FLEXIBILITY.

Windows Search works ... but this built in dependency on Windows Search i.e. Windows client, WinRT and libraries ... and then no support reparse points. It's a no go! IT HAS TO GET FIXED.

Or switch the search engine. An intermediary here has to be FULLY compatible with NTFS! This is not 3rd party development.

• Edited by Computermensch Friday, March 08, 2013 12:52 PM

• Edited by Computermensch Friday, March 08, 2013 12:51 PM

And BTW:

DFS still does not work with Windows Search:

So problems with using Windows Search has now extended to:

1) No DFS support

2) No Reparse points support (i.e. no mount points)

3) Thus no deduplication support (implied by 2) (NEW)

... and Windows Search is required for using File Libraries on Windows 8

4) WinRT applications has to use file liberaries (NEW) - which has to use Windows Search ... which cannot use deduplication, reparse points or DFS.

What a mess. Please fix Windows Search.

• Edited by Computermensch Friday, March 08, 2013 12:51 PM

We reported the indexing / deduplication problem to Microsoft at the end of February.
They got back to us confirming that it is a known problem related to design.
The windows search service is not able to index deduplicated data. The following steps can be used as a workaround to get all data indexed:
1. Deactivate deduplication (Server Manager)
2. PS:> Start-DedupJob -Volume X: -Type unoptimization
3. Rebuild Index
4. Activate deduplication (Server Manager)
5. PS:> Start-DedupJob -Volume X: -Type optimization
Pay attention: The above mentioned steps decrease the server performance. Unoptimization additionally requires more space on the volume. Check with the PowerShell command "Get-DedupStatus" if you dispose over enough available space on the volume.
PS: Microsoft told us that if a lot of customers report this issue, they will fix it. So feel free to report to Microsoft accordingly.

In Febraruary you wrote:

We reported the indexing / deduplication problem to Microsoft at the end of February.
They got back to us confirming that it is a known problem related to design.
The windows search service is not able to index deduplicated data.

Any news?

Can we hope Microsoft will solve this problem?

Deduplication should be useful for data storages but if it brokes indexing and searching..... nobody can use it :(

I too am interested in MS fixing this issue.  My index is not finding much of what is in our file system.  I'm a little unclear if the workaround removes dedupe, indexes, and then RE-dedupes the data, or if it simply unoptimizes the deduped volume and reindexes.   The point being, indexing has to occur with dedupe off, but will remain intact (indexed) if you turn on dedupe after the fact.

Thanks for any help.

If I do the initial indexing that way - what is happening with new files, or old moved
files ... are they indexed correctly than?

Regards

Martin

Same issue here.

For testing we moved some files that were not searchable and the Index was unable to find to a new location. Once we moved the files they Indexed immediately and we were able to search them successfully.

It looks like you have to index your files first before you dedup.

Currently have a case open with Microsoft, I will report back if they find anything.

Thanks to those who opened a case.

Since we have OEM licenses for Windows Server (not Volume licensing), we would have to pay >200 $ just to report the issue. THEIR issue, as it clearly seems. Another lesson of "Just get big enough, then you don't have to bother anymore".

Sorry, but I'm quite upset that those (well advertised) features do not work together, even in Server 2012R2.

Regards,

Andreas

Is there some news about this issue?

Regards, Lorenzo.

This is ridiculous. We just came across this too. Our 2012 R2 file server search is now useless because all volumes are deduped.

This is critical functionality that has to be fixed.  Nowhere is any dedupe documentation or reviews or write-ups or blogs have I ever seen it mentioned that dedup breaks indexing.

I'm having the same issue.
I don't want to disable deduplication because that would need me to extend my disk volume which I won't be able to shrink afterwards.
Think the idea of deduplication is to use less diskspace isn't it... :-)

Microsoft, please fix this or provide us with a tool to create the initial correct windows search indexing database.

Regards,

Kristof

confirmed still not working on 2012R2 with latest windows updates as of June 2015 :(

this really needs to be fixed!!