You won't be able to apply conferencing policies to a physical site. You can apply them globally, to a Lync site (since you only have one Lync site, this would be the same as global for you), or to individual users.
To limit A/V from physical network sites, you should use Call Admission Control. You define your network subnets in Lync and associate those with Lync network sites. You can then create a CAC policy for those network sites that limit bandwidth
used for A/V. You can set limits for the entire bandwidth consumed by all users and per individual stream. I believe the lower limit for video is 100 kb for both the aggregate and the individual stream, so while you can't stop video entirely, you
can essentially limit it to 1 stream per site at any given instant. Everyone else at that site would be denied video.
I have a script that will translate AD sites/subnets into Lync sites/subnets. It isn't perfect, but its a good start: http://ucken.blogspot.ca/2013/04/automatically-creating-lync-sites-and.html