Some e-mail that is send to our exchange 2013 server is delivered with a lot of delay (like 15 minutes).

A message was sent to 3 e-mail addresses, of those, 2 people received the e-mail without delay, but only one person had 15 minutes delay. All 3 mailboxes are on the same Exchange server, to the issue is probably something internally.

When I check the logs, I see 'HAREDIRECTFAIL' (no suitable shadow servers). We didn't configure exchange in HA mode.

Anyone has any pointers on where to look?

HI 

HAREDIRECTFAIL indicates that a shadow message failed to be created. This means that the transport service failed at trying to make the specific message redundant (while in transit) at another Exchange 2013 mailbox server (transport service) for whatever reason, probably stated in the source-context field.

So the above message will not affect the delay of the emails in any ways.
Please check the affected mailbox scanning systems and see if there is  any delay 
Ask the affected user to check for the email in the webmail

As a last try move the user to a different DB and see the results