Some of our users want to manually convert word .doc files to the new .docx Format for specific documents. What they do is download a copy of the document, open it in word, use the convert button, save as .docx. Then upload the file to the document library with Option "overwrite existing documents" set. It seems that SharePoint recognizes the .doc file in the library as the file to be replaced (it Shows me exactly the metadata set for the existing file). However if save is completed, there are two documents in the library. Deleting the .doc is not an Option because Version history is lost in this case. Any idea how to get this work without coding?

so, long and short is that what you are attempting will ALWAYS require two steps.

First, the PREVIOUS file (.doc) needs to be renamed to .docx

Then, the new file needs to be uploaded in its place.

The best way to do this is with a single checkout (though that's not always enabled).

The reason the metadata is maintained is because it's stored in the DOC/DOCX file. Not because SP is doing any sort of linking. Any files uploaded that do not replace a file are considered completely separate to all other files... just like a DOC file and a TXT file would be unrelated.