It seems like there should be some optional built in authentication method to prevent DOS on the service (intentional or accidental) and to conserve system resources for whiltelisted clients.
One option would be a whitelist and blacklist feature in the configuration file. This could parse the ICAP header X-Client-IP or default to the sourceIP if no X-Client-IP header is supplied. This would allow for a decent level of control with minimal performance impact.
I’m sure there are plenty of other options.