Gemma4 at Home: Running 26 Billion Parameters on Consumer Hardware

The pitch for local LLM inference is simple: take the API call that costs money, replace it with electricity that costs less money, and get the bonus that your prompts never leave the house. The actual implementation involves driver versions, CUDA matrices, quantisation formats, and an amount of VRAM watching that your blood pressure didn’t sign up for.

Welcome to running Gemma4-26B at home.

Why Local Inference

At the query volume a busy AI lab generates, cloud API bills add up. The more interesting reason, though, isn’t cost - it’s independence. A local model works when the internet is down, has zero WAN latency, and can sit inside local automation jobs that you’d never want to pipe out to a cloud endpoint.

My main use case was the daily network security summary. Every morning the assistant pulled Nmap scan data, compared it against the previous day, and sent a voice note. Shipping a daily snapshot of my internal network layout to a public cloud API felt wrong on principle.

The Model: Gemma4-26B-A4B-AWQ

Gemma 4 uses Google’s Mixture-of-Experts (MoE) architecture: “26B” is the total parameter count, while A4B means only 4 billion parameters are active on any single forward pass. That’s why it runs fast and fits comfortably in VRAM where a dense 26B model would feel cramped.

The AWQ suffix is Activation-aware Weight Quantisation - a 4-bit post-training quantisation format that protects the weights that matter most to output quality instead of crushing every layer equally.

On an RTX 5090 with 32GB GDDR7 VRAM:

• Model footprint - ~14GB VRAM
• KV cache (FP8) - Remaining VRAM up to an 86% utilisation cap
• Maximum context - 98,304 tokens
• Speculative decoding - Enabled for faster token generation

Here’s the launch config:

vllm serve ~/models/Gemma4-26B-A4B-AWQ \
  --quantization compressed-tensors \
  --gpu-memory-utilization 0.86 \
  --kv-cache-dtype fp8 \
  --enable-chunked-prefill \
  --speculative-model auto \
  --max-model-len 98304 \
  --host 0.0.0.0 \
  --port 8000

--enable-chunked-prefill splits long prompts into manageable chunks so big contexts don’t trigger out-of-memory spikes, and --speculative-model auto lets vLLM pick a draft model automatically to speed up decoding.

vLLM on WSL2

Running vLLM in WSL2 rather than a full Hyper-V VM was a practical choice: WSL2 gets direct CUDA passthrough to the RTX 5090 without the overhead of GPU-PV partitioning.

The WSL2 setup:

• Ubuntu 24.04 in WSL2 with systemd enabled
• CUDA 12.x from NVIDIA’s WSL2 repo
• vLLM installed in an isolated venv at ~/ai_env/venv/
• Systemd service vllm-gemma-moe.service starting automatically on boot

A Windows Scheduled Task (WSL2-KeepAlive) fires at logon to boot WSL2 and start systemd so vLLM comes up whenever REX reboots - without it, WSL2 sits asleep until you open a terminal window.

Exposing Port 8000 to the Lab VLAN

WSL2 sits behind a virtual NAT adapter that other VMs on the LAN can’t reach directly. A Windows netsh port proxy bridges the gap:

netsh interface portproxy add v4tov4 `
  listenport=8000 listenaddress=0.0.0.0 `
  connectport=8000 connectaddress=$(wsl hostname -I).Trim()

That forwards port 8000 on REX’s LAN IP straight into WSL2, making vLLM reachable at http://172.16.50.5:8000 like any other Linux host on the lab subnet.

In LiteLLM (covered in the next post), I originally registered this endpoint under the alias claude-sonnet-4-6 so local clients could use standard Anthropic-style configs without caring which backend answered.

The Use Case: Always-On Inference for Automation

With local inference running on the 5090:

• Latency - First token in ~800ms on a cold request
• Throughput - ~45 tokens/sec output with an FP8 KV cache
• Cost - Electricity (~300W under full load)

At Queensland power rates (~$0.33/kWh), a full hour of non-stop inference costs about ten cents, and most requests finish in a couple of seconds.

Better still, it keeps working when the NBN drops out - which is usually the exact moment you want your local assistant to help diagnose the network.

What’s Next

Having a local model is great. Having a single proxy that can route between local inference and cloud frontier models on demand is even better. That’s LiteLLM.

Update - August 2026

The model, the GPU, and the WSL2 setup stayed in place through August, with three changes around the edges.

I killed off the claude-sonnet-4-6 alias on 18 July 2026, and good riddance. vLLM now serves one honest model name - gemma4-26b-a4b. The original idea was that clients shouldn’t care which backend answered. In practice it meant every log line, config file, or error message mentioning claude-sonnet-4-6 was a coin toss: was that local Gemma, or actual Sonnet on Vertex? Debugging a bad answer starts with knowing who gave it. The fake alias now returns a 404.

The local model is also reachable through the private LiteLLM proxy in GCP as gemma4-local, routed back home over the WireGuard tunnel. It’s the only entry on that proxy that goes dark if my house loses power.

Gemma 4 needed a parser patch before tool calling worked properly. The model wraps its internal chain-of-thought in <channel|> markers, and vLLM’s streaming parser was leaking those markers into the response body - harmless in a chat window, fatal when the output needs to parse as a JSON tool call. Running with --reasoning-parser gemma4 plus a small local patch to vLLM’s parser fixed it. I keep a copy of the patched file in the repo so I can re-apply it whenever vLLM updates.

Concurrency is the real ceiling, not quality. At a 98k context window, 32GB of VRAM comfortably holds two concurrent sequences. That’s plenty for one human and one assistant, and it’s the hard limit on what a single card can do at once.

I already have a shortlist for its replacement - Qwen3 30B-A3B is out in front, with a GLM flash MoE behind it. The test isn’t benchmark scores, which are useless for judging tool use - it’s whether the model can chain six MCP calls together without making up a parameter.

Update - October 2026

I retired Gemma 4 at the end of August.

• The replacement is Swift-Qwen3.8-27B-NVFP4 - I moved first to dense Qwen3.8-27B-NVFP4 and then in mid-September to the Swift fine-tune on vLLM 0.28.0, running mixed precision (modelopt_mixed - FP8 attention layers paired with native Blackwell NVFP4 MLP layers). It keeps the ~20 GB weight footprint inside the 32 GB envelope while cutting internal reasoning tokens by roughly 58%, bringing multi-step tool-call latency down to ~1.8 seconds on turn two.
• Qwen 3.8 inverted Gemma 4’s thinking-mode rule - Where Gemma 4 broke tool calling if thinking was left on, Qwen 3.8 breaks if thinking is explicitly disabled on tool-bearing requests. Passing enable_thinking: false causes the model to return empty responses on the turn after a tool result comes back. Letting the Swift variant think natively solved both the empty-response bug and the latency tax.
• Three systemd units share port 8000 for instant rollback - Conflicts= declarations make sure only one model binds the GPU at a time, and a companion outbound firewall service blocks the inference process from initiating any external internet connections.