Do you have multiple web front ends? It's possible one of your WFEs is struggling and serving up pages slowly, which would account for seemingly random inconsistencies.
Hi Dauchnie
There are some known issues with the IE11 and SharePoint 2013 display. It causes some of the script to run longer than their usual response time. You could try to use Internet Explorer compatibility view and see if it loads just fine.
1. Tap or click the Tools button , and then tap or click Compatibility View settings.
2. Under Add this website, enter the URL of the site you want to add to the list, and then tap or click Add.
Also, go to IE -> Internet options -> advanced -> Uncheck use software rendering instead of GPU rendering and see if it helps.
AS You said that there are no issues with the Firefox and the issue is only with IE11.Did you check the issue with other versions of IE10,9? please check.There are some compatible issues with the IE11 with share point.Please check the link below that explains more in detail about the known issues.
http://expertsharepoint.blogspot.de/2014/08/issues-with-sharepoint-2013-and.html