What Is an LLM Proxy?

What is an LLM proxy for?

Anything you want to happen to every model call, without changing every caller. Teams put a proxy in the path to hold provider keys centrally, to switch models without redeploying, to count tokens per team, and, increasingly, to take personal data and secrets out of prompts before they leave. The proxy is the one place every request has to pass.

The word covers two quite different deployments, and mixing them up causes confused buying decisions. The question to ask is: who points at the proxy, the application or the device?

Reverse proxy vs forward proxy

A reverse proxy is the easy case. Your service connects to it on purpose, so it sees the request in full. The AI gateway entry covers that side in detail.

How does a forward proxy catch AI traffic from employees?

Consider a sales manager on a Windows laptop who pastes a customer list into Claude on the web. The browser talks to the provider over HTTPS. A forward proxy that wants to mask the text has to be in that connection, which means the device must be told to send traffic for that host through the proxy, and the proxy must be able to read the content for those hosts.

Maskflare's desktop agent does the routing part narrowly. It routes only AI hosts through Maskflare; everything else, from banking to email, goes direct. Covered on the web: ChatGPT, Claude, Gemini, DeepSeek and Perplexity, plus apps calling the provider APIs that follow the system proxy. The agent runs on Windows (x64, ARM64, MSI), macOS (signed and notarized PKG) and Linux desktops with GNOME, and is deployed through Intune, Jamf or Group Policy with an enrollment token from the console.

What happens when the proxy is down?

Every proxy has to answer this. Fail open means traffic goes direct and unmasked when the proxy can't be reached. Fail closed means AI traffic stops until it's back. Fail open protects productivity; fail closed protects data. Maskflare's desktop agent fails closed by default, and that's configurable. Two related details matter for anyone who's run endpoint agents before: a watchdog re-applies the proxy setting within 10 seconds if a user or another tool changes it, and uninstalling needs a one-time code from the console.

Can you self-host an LLM proxy?

Yes. LiteLLM is a widely used open-source proxy that gives many providers one OpenAI-style API. It supports guardrail plugins, and Maskflare ships one in its Python package, so masking runs inside your LiteLLM proxy:

pip install "maskflare[litellm]"

# Guardrail class to reference in your LiteLLM config:
# maskflare.integrations.litellm.MaskflareGuardrail

Setup details: LiteLLM integration and the LiteLLM PII masking guide. Enterprise customers can also run Maskflare itself on premises.

What should an LLM proxy record?

A proxy sees every prompt, which makes it tempting to log everything. Resist that. A proxy that stores full prompts becomes the most sensitive database you own, and a breach of it exposes exactly the data you were trying to protect. What auditors and security teams usually need is metadata: which key or device, which provider and model, when, how many values were masked, which rule fired and what action it took.

That's Maskflare's approach. Prompts and responses are never logged or stored; records hold metadata and counts only. Audit records can be exported as NDJSON for up to 366 days, and signed webhooks can notify your own systems when a rule fires.

How do you deploy an LLM proxy for a team?

1. Split the problem: applications go through a reverse proxy or gateway, people go through a forward proxy.
2. For apps, change the base URL to https://<your-gateway>/v1/<provider>.
3. For people, create an enrollment token in the console and push the desktop agent with Intune, Jamf or Group Policy.
4. Start rules in shadow mode, then enforce.
5. Watch device states in the console: protected, unprotected, stale or revoked.

What can an LLM proxy not see?

Apps that ignore the system proxy, phones and unmanaged personal laptops, and AI features built into other SaaS tools that call models from the vendor's own servers. Maskflare's desktop agent covers ChatGPT, Claude, Gemini, DeepSeek, Perplexity, Mistral Le Chat and Microsoft 365 Copilot, native AI desktop apps and clipboard pastes into AI apps. In voice sessions, text events are masked but audio isn't. The heartbeat the agent sends every 5 minutes reports state and version only, never URLs or traffic, so the proxy isn't a browsing monitor either. AI app discovery records which AI apps are used, as metadata only.