← All Posts POST #06 · · 8 min read

LiteLLM: One Endpoint to Rule Them All

Deploying a LiteLLM proxy on GCP to unify Claude, Gemini, and local Gemma4 behind a single OpenAI-compatible API. Also: why nginx and Redis are doing more work than you'd expect.

#litellm#gcp#vertex-ai#proxy#gemini#claude

Once you have a few different AI models in play - Claude on Vertex AI, Gemini via the Generative Language API, and Gemma4 running locally on the RTX 5090 - every script and CLI in the lab suddenly needs to know which URL to call, which auth header to send, and what model string each provider expects. That kind of config sprawl gets old very quickly.

The fix is a proxy. One endpoint, every model behind it.

The Architecture

LiteLLM is an open-source proxy that exposes a standard OpenAI- and Anthropic-compatible API on the front and routes to Vertex AI, Gemini, local vLLM, and dozens of other providers on the back. You deploy it once, point your tools at it, and add new models by editing a YAML file.

The original GCP deployment:

  • VM - litellm-proxy on GCE (e2-medium in australia-southeast2), IP 10.200.2.2
  • Stack - Docker Compose running LiteLLM + Redis 7
  • Internal port - 4000
  • TLS - nginx on 443 with a Step-CA cert, reachable at https://10.200.2.2 or https://litellm.sdx.local
  • Network - Connected back to the home lab over a site-to-site WireGuard VPN tunnel

The proxy VM lives in sdx-platform-llm-prod, while a small gateway VM in sdx-lab-bridge-prod terminates the WireGuard tunnel back to OPNsense at home. OPNsense advertises 172.16.0.0/16 across the /30 tunnel and the cloud side advertises 10.0.0.0/8 back. That gives me a clean layer-3 overlay between home and Melbourne at ~38ms latency. I switched from Tailscale to raw WireGuard earlier in 2026 because the kernel data path and static peer config are much easier to reason about in Terraform.

The Model Roster

model_list:
  - model_name: claude-opus-4-7
    litellm_params:
      model: vertex_ai/claude-opus-4-7-20250514
      vertex_project: sdx-platform-llm-prod
      vertex_location: global


  - model_name: claude-sonnet-4-6
    litellm_params:
      model: vertex_ai/claude-sonnet-4-6-20250514
      vertex_project: sdx-platform-llm-prod
      vertex_location: global


  - model_name: claude-haiku-4-5
    litellm_params:
      model: vertex_ai/claude-haiku-4-5-20251001
      vertex_project: sdx-platform-llm-prod
      vertex_location: global


  - model_name: gemini-2.5-pro
    litellm_params:
      model: vertex_ai/gemini-2.5-pro
      vertex_project: sdx-platform-llm-prod
      vertex_location: us-central1


  - model_name: gemini-3-pro-preview
    litellm_params:
      model: gemini/gemini-3.1-pro-preview
      api_key: <redacted>

I registered versioned model names (claude-sonnet-4-6-20250514) as explicit entries rather than model_group_alias mappings. LiteLLM’s alias feature only applies to OpenAI’s /v1/chat/completions route, not Anthropic’s /v1/messages route that Claude Code uses. Explicit entries work everywhere.

The nginx Layer

nginx sits in front of LiteLLM to handle TLS termination and one surgical routing bypass:

location /v1beta/ {
    proxy_pass https://generativelanguage.googleapis.com/v1beta/;
    proxy_set_header x-goog-api-key "...";
    proxy_set_header Authorization "";
}


location / {
    proxy_pass http://127.0.0.1:4000;
}

That /v1beta/ block bypassed LiteLLM and went straight to Google’s Generative Language API because Vertex AI rejected the function_response.id field that Gemini CLI included in tool call history. I found this after an afternoon of puzzling over 400 errors and reading the Gemini CLI source. The fix was three lines of nginx config - the debugging took a bit longer.

Redis Caching

LiteLLM’s Redis integration caches responses for identical prompts:

  • Repeated system prompts across sessions
  • Static tool definitions sent on agent calls
  • Test loops running the same prompt twice

I set the TTL to 5 minutes to match Anthropic’s prompt cache window, returning exact-match test calls in a few milliseconds.

Claude Code Configuration

Claude Code points at the proxy through ANTHROPIC_BASE_URL:

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://10.200.2.2",
    "ANTHROPIC_API_KEY": "<litellm-master-key>"
  },
  "model": "claude-opus-4-7"
}

The API key on the client is just the LiteLLM master key. LiteLLM handles the upstream Vertex AI auth using the GCP VM’s service account, so GCP credentials never have to sit on the home lab machine.

The Failover Script

Two small PowerShell scripts handle failover if the tunnel or proxy ever has a bad day:

  • F:\use-proxy.ps1 - Points Claude Code and Gemini CLI at litellm.sdx.local
  • F:\use-fallback.ps1 - Flips Claude Code to direct Vertex AI (CLAUDE_CODE_USE_VERTEX=1) and Gemini CLI to the direct Google API

Running the fallback script takes two seconds and saves me from editing env vars by hand when something is broken.

The Use Case: Every Agent, One Endpoint

With LiteLLM in place, Claude Code, Gemini CLI, Cursor, Aider, and custom Python scripts all talk to one URL over the WireGuard tunnel. Switching models is a one-line change, and Vertex AI spend stays visible in one place.

Update - August 2026

This post aged the fastest of the whole series, which is what happens when you write about the fastest-moving layer in the stack. There are two proxies now, the model list has roughly doubled, and the biggest lesson was discovering that both proxies had been accepting any non-empty API key for weeks.

Authentication was failing open

On 3 July 2026 I found out that both proxies were authenticating badly: any non-empty string in the API key header returned 200 OK and a real completion. Only a completely missing header returned 401.

The culprit was a single line in both config files: allow_requests_on_db_unavailable: true. Neither proxy runs a Postgres key database (on purpose, to keep them stateless and simple), and with that flag turned on, LiteLLM treated “no database configured” as “database is down - fail open and let the request through”.

The private proxy sits behind WireGuard, so exposure there was limited to my own network. The public proxy sits on the internet. Removing that one line fixed both immediately: the master key returns 200, a wrong key gets rejected, and a missing key returns 401.

Two takeaways from that one: first, “does auth actually reject a bad key” is a test you run in your deploy script, not something you assume from reading YAML. Second, the fancy Cloud Armor WAF in front of the public proxy didn’t help at all when the app behind it was politely waving everyone through.

The WAF rules were fighting the payload

Speaking of Cloud Armor, I originally attached the preconfigured SQL-injection and XSS rulesets to the public load balancer. I had to drop both to preview mode within days because they blocked almost every real coding prompt.

It’s obvious once you see it happen: an LLM endpoint’s request body is full of arbitrary prose, SQL queries, shell scripts, and HTML snippets. Ask the model to help optimise a SELECT statement and you’ve just sent a textbook SQLi signature to your own API. Signature-based body inspection doesn’t work when the payload is code. The controls that actually work here are the ones that don’t try to parse your prose: strict API key enforcement, per-IP rate limiting, and locking Cloud Run ingress to the load balancer so the run.app URL can’t be hit directly.

The August shape

  • Private proxy - GCE VM at 10.200.2.2 (litellm.sdx.local) over WireGuard, running LiteLLM, Redis, and nginx in Docker. Serves 22 models, including gemma4-local routed back over the VPN to the RTX 5090 at home.
  • Public proxy - Cloud Run behind an external HTTPS load balancer at chat.sequin.au with Cloud Armor and load-balancer-only ingress, serving 21 models.

Both proxies now use separate master keys, and claude-mode.ps1 private|public|direct (plus gemini-mode.ps1) replaced the older failover scripts.

Claude 5, and what it takes to add a model on Vertex

Claude Opus 5 and Sonnet 5 went live on both proxies on 28 July 2026, with Opus 5 as the default on REX.

Adding a new Anthropic model on Vertex AI is no longer just a two-line YAML edit. Each new release comes with four separate gates:

  • Per-model terms in Model Garden - Accepting the previous model’s terms doesn’t carry over to the new one.
  • Publisher data sharing - Must be enabled per location, otherwise Vertex returns a 403 that reads like an IAM error.
  • Global endpoint only on day one - Regional token-per-minute quota on a brand-new model defaults to zero, so regional calls return an instant 429 even when auth is fine.
  • Org policy allowlist for web search - vertexai.allowedPartnerModelFeatures lists each model ID explicitly, so a new model loses web search with a 400 until you update the Terraform policy.

One more quirk worth noting: Claude Code’s [1m] extended-context suffix is purely a client-side hint. When ANTHROPIC_BASE_URL points at a custom proxy, Claude Code assumes a 200k window unless you set the client model string to claude-opus-5[1m]. Claude Code strips the [1m] before sending the HTTP request - so the suffix belongs in your local client settings, never in LiteLLM’s YAML.

Two operational clean-ups

  • The /v1beta/ nginx bypass is gone - LiteLLM’s native gemini/* routing handles Gemini CLI properly now, so I deleted the old nginx intercept.
  • Container images are pinned by digest - Early on, changing an environment variable on Cloud Run caused Artifact Registry to re-resolve main-latest and silently pull a brand-new upstream LiteLLM build. Both proxies now pin a verified version (1.82.6 in August) by digest so a config tweak never sneaks in an unplanned upgrade.

Correction on Redis caching

Earlier in this post I claimed that Redis response caching saved time on repeated system prompts and tool schemas. When I actually measured it in July, exact-match response caching almost never fired on real agent traffic - because two conversational turns are never byte-for-byte identical. The cache that genuinely saves money and latency is Anthropic’s own upstream prompt caching, which passes straight through the proxy without Redis doing a thing.

Update - October 2026

  • The model catalogue nearly doubled again, with explicit <upstream>/<id> routing - Both proxies now run LiteLLM 1.99.0 (verified with cosign and mirrored into a standard Artifact Registry repo so multi-arch OCI digests stay immutable). They serve nearly forty models across Claude (claude-opus-5-5[1m] is now my default on REX), Gemini 3.x (with automatic AI Studio to Vertex AI failover), and partner open-weights models on Vertex.
  • Custom pre-call hooks smooth over upstream quirks - A small callback strips unsupported forced tool_choice parameters for reasoning models that reject them, and passes Claude Code’s auto-mode safety classifier headers and request fields through to Vertex AI while skipping Redis cache lookups on classifier probes.
  • Cloud Armor needed one more round of tuning - In late August, GET /health/liveliness on the public proxy returned 200 OK while every POST /v1/messages request failed with a 403, and rapid multi-turn coding sessions were triggering temporary IP bans. I scoped out the OWASP protocol-enforcement rules that choked on large JSON tool schemas and bumped the per-IP rate threshold to match the pace of real agentic loops.