If you want to develope a farm solution for SharePoint, we need install Visual Studio in a SharePoint server.
If you only want to develope a Client-Side project for SharePoint, we can install visual studio in our local PC and use some Client Object Model to access SharePoint.
About multiple developers for SharePoint project, here is a discussion for your reference:
The following articles about build a SharePoint development environment for your reference:
Start: Set up the development environment for SharePoint 2013
Setting up a SharePoint 2013 Development Environment 101
Set up an on-premises development environment for apps for SharePoint