a Group Policy always "wins" for Software Updates. So, two ways:
1) Those fringe computers deserve a GPO which smacks down ConfigMgr and points to their WSUS server. GPO wins, always.
2) There are some things in ConfigMgr which, on a per-client basis, you have to use something called a "Local Policy Override", to tell the client, locally (not from a policy retrieved from the Management Point) to turn off a specific thing...
like Software Update agent component. Here's one person's example:
http://www.hasmug.com/2014/12/30/disable-software-update-agent-on-sccm-2012-client/
(Sorta related):
http://www.mnscug.org/blogs/sherry-kissinger/355-selectively-disable-software-distributions-and-application-deployments-on-clients
Since you said all you want is "inventory", you may want to deploy those ConfigItems (of course, that means that you'll have to break your own rule about "only inventory" and allow Compliance Settings and Baselines to be run). You
may want to also copy one of those configItems (probably the software Distribution Agent CI), and change it to do the Software Updates local policy override.