This has been working for a while, not sure when it stopped. The last CU and SP1 did no difference.

We have two servers: 2012R2, SCCM2012R2 SP1

- Primary site server with application catalog website point, application catalog web service point + mp, sup, dp, endpoint, reporting services, migration. mixed mode. HTTP
- Site system server (internet facing) with application catalog website point, enrollment point & proxy, sup, dp. HTTPS

The application catalog works from intranet zone with NetBIOS name or FQDN DNS in domain.

SCCM-SRV1.domain.com/CMApplicationCatalog
SCCM-SRV1/CMApplicationCatalog
SCCM-SRV2.domain.com/CMApplicationCatalog
SCCM-SRV2/CMApplicationCatalog

Now, the internet fqdn:

SCCM.ANOTHERDOMAIN.COM/CMApplicationCatalog

does only work in IE on non-domain computers. Domain joined computers just show a plain white page (nothing). If I remove the internet address form intranet site (IE security zone) it will ask for credentials. If the correct credentials are supplied, one will just get a new credential prompt a few times before ending with a "HTTP Error 401. The requested resource requires user authentication." if I supply the wrong credentials I immidiatly get a "HTTP Error 401.1 - Unauthorized".

Any suggestions, we have tried different things like reinstalled the application catalog website point, 

That sounds like something is getting in the way of the traffic like a reverse proxy or some other network device. Have you verified in IIS (in the logs) that the request is indeed hitting the system hosting the Internet facing App catalog?