The publishing site has anonymous access enabled. I was not able to open any XLSX files (even a blank one) in any document libraries. Also it is not tied to this site. I have another test farm and the same thing happens. The login prompts. and if I login with a user who doesn't have any permissions on the site, I got access denied. XLS document has no problem to open. If I edit serverfilesExcelServer.xml and comments the line of < Mapping FileExtension="xlsx" RedirectUrlTemplate= "/_layouts/xlviewer.aspx?id=|0" NoGetRedirect="TRUE"/> then the problem disappears. however i am not sure that is the correct solution or not.

Make sure that the files are checked in and published.I hope this helps and answers your question-Jeff DeVerter, MCSERackspaceblog:http://www.social-point.comtwitter: http://www.twitter.com/jdeverter

Jeff, Thanks for the reply. I double checked the file is checked in. The versioning is not even enabled.

It seems like it's trying to open the Excel file via Excel Services? Try checking the settings on the document library, under advanced settings check that it is set to 'Open in client application'. Also check you anonymous access settings. Under the site settings go to advanced permissions. Under the settings menu go to 'Anonymous Access' and make sure it's set to 'Entire web site'.

Yes, it is set to "Open in Client Application" and the anonymouse Access was set to "Entire Site". Actually I tested this on three different farm and I got same results: Login required when opening up a XLSX file off anonymous enabled Publishing Site. So none of you has experienced the same issue before? it is really weird.

Try this: http://xinspace.com/sharepoint-prompts-login-when-opening-a-xlsx-file-on-a-publishing-site/

I know this thread has been out there for a while, but I would advise not to make any modifications to the XML files within the 12 or 14 Hive on your SharePoint Servers. This could put you into an unsupported state and we are not 100% certain of any other impact this may have. While I do agree removing this line or commenting it out from the ServerFiles.xml file will resolve your issue with being able to access "xlsx" files anonymously within a library, I would not do it from a supportability stand point. This is indeed a by design behavior in the way that SharePoint handles and maps "xlsx" and "xlsb" files and it tries to render these files within XlViewer.aspx by default. There are 2 supported options to resolve this: 1) break permissions on each library and grant anon users access to those libraries.. this works 2) add this base permissions to the anonymouspermmask64 to allow the "OpenItems" permission When you try to directly open an "xlsx" on an anonymous library, it fails because SharePoint will try to pass the file to xlviewer.aspx and will require the additonal permission of "OpenItems". In the ULS logs, you will see a Permissions Mask Check Failure indicating it needs an additonal permission. My 2 Cents...