LiteLLM: One Endpoint to Rule Them All

litellmgcpvertex-aiproxygeminiclaude

The natural endpoint of having multiple AI models — Claude on Vertex AI, Gemini via the Generative Language API, Gemma4 running locally on the RTX 5090 — is that every tool and script in the lab needs to know which endpoint to call, which authentication method to use, and which model name format each API expects. This is the kind of configuration sprawl that makes infrastructure engineers reach for the antacids.

The solution is a proxy. One endpoint. Every model behind it.

The Architecture

LiteLLM is an open-source proxy that presents a single OpenAI-compatible API and routes requests to any supported backend: Anthropic, Vertex AI, Gemini, local vLLM instances, and many others. Deploy it once, point everything at it, and add models by editing a YAML config rather than updating every client.

The lab’s LiteLLM deployment:

  • VM: litellm-proxy on GCP, e2-medium, australia-southeast2, IP 10.200.2.2
  • Stack: Docker Compose — LiteLLM + Redis 7
  • Internal port: 4000 (OpenAI-compatible)
  • External: nginx on 443, TLS via self-signed cert, reachable as https://10.200.2.2
  • Connectivity: Reachable from the home lab via a WireGuard VPN bridge

The GCP VM lives in the sdx-platform-llm-prod project. A second VM, wireguard-bridge (10.200.1.2) in sdx-lab-bridge-prod, terminates a WireGuard tunnel back to the OPNsense firewall at home. The tunnel itself is a /30 (10.99.0.1 ↔ 10.99.0.2); OPNsense advertises 172.16.0.0/16 across it and the bridge advertises 10.0.0.0/8 back. The result is a layer-3 overlay between home and cloud that makes the GCP proxy as reachable as any local host — at ~38ms latency, which is Queensland to Melbourne and back. WireGuard replaced an earlier Tailscale gateway in 2026; the kernel-space data path and explicit peer config are easier to reason about and run in Terraform.

The Model Roster

model_list:
  - model_name: claude-opus-4-7
    litellm_params:
      model: vertex_ai/claude-opus-4-7-20250514
      vertex_project: sdx-platform-llm-prod
      vertex_location: global

  - model_name: claude-sonnet-4-6
    litellm_params:
      model: vertex_ai/claude-sonnet-4-6-20250514
      vertex_project: sdx-platform-llm-prod
      vertex_location: global

  - model_name: claude-haiku-4-5
    litellm_params:
      model: vertex_ai/claude-haiku-4-5-20251001
      vertex_project: sdx-platform-llm-prod
      vertex_location: global

  - model_name: gemini-2.5-pro
    litellm_params:
      model: vertex_ai/gemini-2.5-pro
      vertex_project: sdx-platform-llm-prod
      vertex_location: us-central1

  - model_name: gemini-3-pro-preview
    litellm_params:
      model: gemini/gemini-3.1-pro-preview
      api_key: <redacted>

Versioned model names (claude-sonnet-4-6-20250514) are registered as explicit entries rather than aliases — LiteLLM’s model_group_alias only applies to the OpenAI-format /v1/chat/completions endpoint, not to Anthropic’s native /v1/messages format. Claude Code uses the Anthropic format. Aliases would silently fail; explicit entries work.

The nginx Layer

nginx sits in front of LiteLLM and adds TLS termination and one important routing exception:

location /v1beta/ {
    proxy_pass https://generativelanguage.googleapis.com/v1beta/;
    proxy_set_header x-goog-api-key "...";
    proxy_set_header Authorization "";
}

location / {
    proxy_pass http://127.0.0.1:4000;
}

The /v1beta/ route bypasses LiteLLM entirely and goes directly to Google’s Generative Language API. This exists because Vertex AI rejects the function_response.id field that Gemini CLI includes in tool call history — it’s valid in the Generative Language API but causes a 400 error on Vertex AI. Rather than modifying the Gemini CLI, nginx intercepts the request and routes it to the API that accepts it.

This was discovered after an afternoon of confusing 400 errors and a detailed reading of the Gemini CLI source code. The fix is three lines of nginx config. The debugging took significantly longer.

Redis Caching

LiteLLM’s Redis integration caches responses to identical prompts. For the lab’s usage patterns, this primarily helps with:

  • Repeated system prompts across sessions (high cache hit rate)
  • Tool descriptions sent with every agent invocation (constant across requests)
  • Testing scenarios where the same query is run multiple times

The cache TTL is set to match Anthropic’s prompt cache window (5 minutes for most contexts). This means quickly-repeated requests to the same model with the same system prompt often return cached responses in milliseconds rather than waiting for API round-trips.

Claude Code Configuration

Claude Code points at the LiteLLM proxy via ANTHROPIC_BASE_URL:

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://10.200.2.2",
    "ANTHROPIC_API_KEY": "<litellm-master-key>"
  },
  "model": "claude-opus-4-7"
}

The API key is LiteLLM’s master key, not an Anthropic key. LiteLLM handles the actual Vertex AI authentication using the GCP VM’s service account credentials — the home lab never holds GCP credentials directly.

The Failover Script

A pair of PowerShell scripts provide instant failover:

F:\use-proxy.ps1 — points Claude Code at the LiteLLM proxy, points Gemini CLI at litellm.sdx.local. F:\use-fallback.ps1 — switches Claude Code to direct Vertex AI (CLAUDE_CODE_USE_VERTEX=1), switches Gemini CLI to the public Google API with a real API key.

Running the fallback script takes seconds. Without it, a proxy outage would require manual environment variable editing under pressure, which is the kind of scenario that produces configuration drift and regret.

The Use Case: Every Agent, One Endpoint

After deploying LiteLLM, the configuration for every AI tool in the lab collapsed to a single endpoint. Claude Code, Gemini CLI, Cursor, Aider, and the LangGraph agents being built later all point at https://10.200.2.2 over the WireGuard tunnel. Switching models is a parameter change. Adding a new model is a YAML entry and a config reload.

The proxy also centralises usage visibility. LiteLLM’s built-in logging shows which models are being called, with what token counts, and at what cost. The total monthly Vertex AI spend is now visible in one place rather than scattered across multiple GCP billing dashboards.

The proxy went live on May 7, 2026. In the week following, approximately zero configuration changes were required on any client. That’s the point of a good proxy.

Update — August 2026

This post has aged the worst of the series, which is what happens when you write about the fastest-moving layer in the stack. There are two proxies now, the model list has roughly doubled, and the single most important thing that happened to this deployment was discovering it had been accepting anyone’s API key for weeks.

The authentication was failing open

On 3 July 2026 both proxies were found to be authenticating incorrectly. Any non-empty API key returned a 200 and a real completion. Only a missing key returned 401.

The cause was a single line in both config overlays: allow_requests_on_db_unavailable: true. Neither proxy runs a key database — deliberately, to avoid the operational weight of one — and with that flag set, LiteLLM interprets “no database” as “can’t check the key, let it through”. The flag had been added for a reason that made sense in isolation, and its interaction with a keyless deployment inverted the security property it was sitting next to.

The private proxy is only reachable over the WireGuard tunnel, so exposure was limited. The public one is on the internet. Removing the flag from both fixed it: master key returns 200, a wrong key returns a rejection, no key returns 401.

Two things worth taking from that. First, “is the auth actually enforced” is a test you write once and run forever, not a property you infer from configuration. Second, the WAF in front of the public proxy — the thing that looks like the security control in the architecture diagram — was completely irrelevant to this bug.

The WAF was also wrong

Cloud Armor was fronting the public proxy with the preconfigured SQL-injection and XSS rule sets. Both had to be dropped to preview mode, because they were blocking every completion request that mattered.

This is obvious in hindsight: an LLM API’s request bodies are arbitrary prose and code. Ask a model to help with a query and you have posted a SQL injection signature to your own endpoint. Signature-based body inspection is structurally mismatched to this traffic — it cannot distinguish an attack from the payload the service exists to receive. The real controls are the ones that don’t read the body: enforced key authentication, a 60-requests-per-minute-per-IP limit, and an ingress setting that closes the run.app URL entirely so the load balancer can’t be bypassed.

The current shape

  • Private proxy — GCE VM at 10.200.2.2, litellm.sdx.local, reachable over WireGuard. LiteLLM, Redis and nginx in Docker, config on local disk. 22 models, including gemma4-local routing back to the RTX 5090 at home.
  • Public proxy — Cloud Run behind an external HTTPS load balancer at chat.sequin.au, with a Google-managed certificate, Cloud Armor, and ingress restricted so only the load balancer can reach the service. 21 models.

They no longer share a master key — that was a real finding, fixed the same day as the auth bug, with separate per-proxy secrets and the public key rotated to a fresh value.

Switching between them is claude-mode.ps1 private|public|direct and the matching gemini-mode.ps1, which replaced the pair of scripts described above.

Claude 5, and what it costs to add a model

Claude Opus 5 and Sonnet 5 landed on both proxies on 28 July 2026. Opus 5 is the default on REX.

Adding a new Anthropic model to Vertex is no longer a YAML edit, which is the claim this post makes and can’t any more. Each new tier has picked up its own gates:

  • A per-model acceptance in Model Garden. Opus 5 has its own partner service entry — accepting the previous model’s terms does not carry over.
  • Data sharing must be enabled for the publisher, per location, or Vertex returns a 403 with a message that reads like a permissions problem.
  • The global endpoint only. A brand-new base model’s regional token-per-minute quota defaults to zero, so a correctly configured regional call returns an instant 429. A 429 there means access is fine and quota isn’t, which is a distinction worth knowing at 11pm.
  • An organisation policy allowlist for web search. vertexai.allowedPartnerModelFeatures is enumerated per model ID, so a new model silently loses web search with a 400 until it’s added. That policy is in Terraform now — and was briefly lost when it turned out to exist only on an unmerged branch, surviving purely because nothing had re-applied over it.

Worth its own note: Claude Code’s [1m] context suffix is a client-side token. Because the base URL points at LiteLLM, the client can’t probe upstream capability and silently falls back to a 200K window unless the model is written as claude-opus-5[1m]. The suffix is parsed and stripped before the request leaves — posting it to the proxy is a 400. So it belongs in the client config and must never appear in the model list.

Two operational corrections

The /v1beta/ nginx intercept described above no longer exists. LiteLLM handles Gemini’s native API surface directly through its gemini/* entries. The nginx config and deploy script that carried the intercept survived for months after the thing they configured had stopped being used — a reminder that stale infrastructure code is indistinguishable from live infrastructure code until you check.

The image is pinned by digest. The public proxy pulls through a remote Artifact Registry repository, which re-resolves main-latest on every new revision — so an unrelated environment-variable change quietly pulled a brand-new upstream LiteLLM build. Given that two releases in this project’s dependency chain were pulled as compromised, “an unrelated config change silently upgrades the binary” is not an acceptable property. Both proxies now pin a known-good 1.82.6, and upgrading is an explicit, deliberate act.

The Redis caching claim doesn’t hold up

The section above asserts that response caching helps with repeated system prompts and tool descriptions. Measured in July 2026, it essentially never fires for conversational traffic — exact-match caching requires an identical request, and agent turns are never identical. The cache demonstrably works when probed directly (10ms versus 1.9s on a repeated call), and the keyspace statistics suggesting a 21% hit rate are polluted by rate-limit and token-counter keys with five-second TTLs. The prompt caching that actually saves money is Anthropic’s own, and it passes through the proxy invisibly without any of this.