So, I,ve been looking at the SQLPS.PSD1 and I see there is a SqlPsPostScript.ps1 in the nestedmodules.
NestedModules = @("Microsoft.SqlServer.Management.PSSnapins.dll","Microsoft.SqlServer.Management.PSProvider.dll", "SqlPsPostScript.ps1")
If I ommit SqlPsPostScript.ps1, all loads fast. I'm probably missing some functionality that way, but since I'm only using the module here to do some basic SQL qerying, this is fine for me.
Maybe someone has more insight in what's happening with that SqlPsPostScript.ps1 and why it's taking so much time to load?