LiteLLM: One Endpoint to Rule Them All

The natural endpoint of having multiple AI models — Claude on Vertex AI, Gemini via the Generative Language API, Gemma4 running locally on the RTX 5090 — is that every tool and script in the lab needs to know which endpoint to call, which authentication method to use, and which model name format each API expects. This is the kind of configuration sprawl that makes infrastructure engineers reach for the antacids.

The solution is a proxy. One endpoint. Every model behind it.

The Architecture

LiteLLM is an open-source proxy that presents a single OpenAI-compatible API and routes requests to any supported backend: Anthropic, Vertex AI, Gemini, local vLLM instances, and many others. Deploy it once, point everything at it, and add models by editing a YAML config rather than updating every client.

The lab’s LiteLLM deployment:

VM: litellm-proxy on GCP, e2-medium, australia-southeast2-a, IP 10.1.0.2
Stack: Docker Compose — LiteLLM + Redis 7
Internal port: 4000 (OpenAI-compatible)
External: nginx on 443, TLS via Step-CA cert, hostname litellm.sdx.local
Connectivity: Reachable from the home lab via Tailscale VPN

The GCP VM lives in the claude-code-endpoint project, connected back to the home lab via a Tailscale VPN gateway at 10.255.1.5. The home lab’s pfSense node advertises 172.16.0.0/16 into the Tailscale mesh; the GCP gateway advertises 10.0.0.0/8. The result is a full layer-3 overlay between home and cloud that makes the GCP VM as reachable as any local host — at ~38ms latency, which is Queensland to Melbourne and back.

The Model Roster

model_list:
  - model_name: claude-opus-4-7
    litellm_params:
      model: vertex_ai/claude-opus-4-7-20250514
      vertex_project: claude-code-endpoint
      vertex_location: us-east1

  - model_name: claude-sonnet-4-6
    litellm_params:
      model: vertex_ai/claude-sonnet-4-6-20250514
      vertex_project: claude-code-endpoint
      vertex_location: us-east5

  - model_name: claude-haiku-4-5
    litellm_params:
      model: vertex_ai/claude-haiku-4-5-20251001
      vertex_project: claude-code-endpoint
      vertex_location: us-east5

  - model_name: gemini-2.5-pro
    litellm_params:
      model: vertex_ai/gemini-2.5-pro
      vertex_project: claude-code-endpoint
      vertex_location: us-central1

  - model_name: gemini-3-pro-preview
    litellm_params:
      model: gemini/gemini-3.1-pro-preview
      api_key: <redacted>

Versioned model names (claude-sonnet-4-6-20250514) are registered as explicit entries rather than aliases — LiteLLM’s model_group_alias only applies to the OpenAI-format /v1/chat/completions endpoint, not to Anthropic’s native /v1/messages format. Claude Code uses the Anthropic format. Aliases would silently fail; explicit entries work.

The nginx Layer

nginx sits in front of LiteLLM and adds TLS termination and one important routing exception:

location /v1beta/ {
    proxy_pass https://generativelanguage.googleapis.com/v1beta/;
    proxy_set_header x-goog-api-key "...";
    proxy_set_header Authorization "";
}

location / {
    proxy_pass http://127.0.0.1:4000;
}

The /v1beta/ route bypasses LiteLLM entirely and goes directly to Google’s Generative Language API. This exists because Vertex AI rejects the function_response.id field that Gemini CLI includes in tool call history — it’s valid in the Generative Language API but causes a 400 error on Vertex AI. Rather than modifying the Gemini CLI, nginx intercepts the request and routes it to the API that accepts it.

This was discovered after an afternoon of confusing 400 errors and a detailed reading of the Gemini CLI source code. The fix is three lines of nginx config. The debugging took significantly longer.

Redis Caching

LiteLLM’s Redis integration caches responses to identical prompts. For the lab’s usage patterns, this primarily helps with:

Repeated system prompts across sessions (high cache hit rate)
Tool descriptions sent with every agent invocation (constant across requests)
Testing scenarios where the same query is run multiple times

The cache TTL is set to match Anthropic’s prompt cache window (5 minutes for most contexts). This means quickly-repeated requests to the same model with the same system prompt often return cached responses in milliseconds rather than waiting for API round-trips.

Claude Code Configuration

Claude Code points at the LiteLLM proxy via ANTHROPIC_BASE_URL:

{
  "env": {
    "ANTHROPIC_BASE_URL": "http://10.1.0.2:4000",
    "ANTHROPIC_API_KEY": "<litellm-master-key>"
  },
  "model": "sonnet"
}

The API key is LiteLLM’s master key, not an Anthropic key. LiteLLM handles the actual Vertex AI authentication using the GCP VM’s service account credentials — the home lab never holds GCP credentials directly.

The Failover Script

A pair of PowerShell scripts provide instant failover:

F:\use-proxy.ps1 — points Claude Code at the LiteLLM proxy, points Gemini CLI at litellm.sdx.local. F:\use-fallback.ps1 — switches Claude Code to direct Vertex AI (CLAUDE_CODE_USE_VERTEX=1), switches Gemini CLI to the public Google API with a real API key.

Running the fallback script takes seconds. Without it, a proxy outage would require manual environment variable editing under pressure, which is the kind of scenario that produces configuration drift and regret.

The Use Case: Every Agent, One Endpoint

After deploying LiteLLM, the configuration for every AI tool in the lab collapsed to a single endpoint. Claude Code, Gemini CLI, Cursor, Aider, and the LangGraph agents being built later all point at litellm.sdx.local (or its IP equivalent). Switching models is a parameter change. Adding a new model is a YAML entry and a config reload.

The proxy also centralises usage visibility. LiteLLM’s built-in logging shows which models are being called, with what token counts, and at what cost. The total monthly Vertex AI spend is now visible in one place rather than scattered across multiple GCP billing dashboards.

The proxy went live on May 7, 2026. In the week following, approximately zero configuration changes were required on any client. That’s the point of a good proxy.