Gemma4 at Home: Running 26 Billion Parameters on Consumer Hardware

llmvllmgemmawsl2gpuinference

The pitch for local LLM inference is straightforward: take the API call that costs money, replace it with electricity that costs less money, and get the added bonus of your queries not leaving the building. The implementation involves driver versions, CUDA compatibility matrices, quantisation formats, and an amount of GPU VRAM monitoring that your cardiovascular system did not consent to.

Welcome to running Gemma4-26B at home.

Why Local Inference

The honest answer: at the query volume a busy AI lab generates, cloud API costs accumulate. But there’s a more interesting reason than economics. Local inference gives you a model that runs without internet connectivity, has zero latency to the API gateway, and can be integrated directly into automation pipelines that would be impractical or expensive with cloud APIs.

The specific use case here: Clawdia, the lab’s AI assistant, generates daily network security summaries. These summaries pull from Nmap scan results, compare them to historical data, and produce a human-readable WhatsApp voice note. Running this against a cloud API would mean every morning’s network state snapshot leaving the building. That felt philosophically wrong.

The Model: Gemma4-26B-A4B-AWQ

Gemma 4 is Google’s Mixture-of-Experts architecture where the “26B” refers to total parameters but only a subset are active per forward pass. The A4B designation indicates 4 billion active parameters per inference step — which explains why it fits in VRAM that a dense 26B model wouldn’t.

The AWQ suffix is the quantisation format: Activation-aware Weight Quantisation. It’s a post-training quantisation method that applies different precision to weights based on their importance to the model’s outputs, resulting in a compressed model that retains more capability per byte than naive int8 quantisation.

On an RTX 5090 with 24GB GDDR7 VRAM:

  • Model footprint: ~14GB VRAM
  • KV cache (FP8): Remaining VRAM, configured at 86% utilisation
  • Maximum context: 98,304 tokens
  • Speculative decoding: Enabled (faster output token generation)

The configuration that works:

vllm serve ~/models/Gemma4-26B-A4B-AWQ \
  --quantization compressed-tensors \
  --gpu-memory-utilization 0.86 \
  --kv-cache-dtype fp8 \
  --enable-chunked-prefill \
  --speculative-model auto \
  --max-model-len 98304 \
  --host 0.0.0.0 \
  --port 8000

--enable-chunked-prefill handles long context inputs by processing them in chunks, preventing OOM errors on prompts with extensive context. --speculative-model auto lets vLLM select an appropriate draft model for speculative decoding automatically — this meaningfully improves output token throughput.

vLLM on WSL2

Running vLLM in WSL2 rather than a full Linux VM was a deliberate choice: WSL2 has direct access to the RTX 5090 via CUDA passthrough without the overhead of GPU virtualisation. A Hyper-V VM with GPU-PV (GPU partitioning) would have worked, but GPU-PV introduces latency and throughput overhead for inference workloads.

The WSL2 setup:

  • Ubuntu 24.04 in WSL2 with systemd enabled
  • CUDA 12.x from NVIDIA’s WSL2 package repository
  • vLLM installed in the ai_env virtual environment at ~/ai_env/venv/
  • Systemd service: vllm-gemma-moe.service — loads on WSL2 startup, auto-restarts on failure

The Windows Scheduled Task WSL2-KeepAlive runs at logon to boot WSL2 and initialise systemd, ensuring vLLM is available whenever REX starts. Without this, WSL2 only starts when a terminal window opens it, which is not useful for a service that other VMs need to reach.

The Service is Reachable at 172.16.50.5:8000

WSL2 runs on a virtual network adapter that, by default, uses NAT and isn’t directly reachable from other hosts on the LAN. The fix is a Windows port proxy:

netsh interface portproxy add v4tov4 `
  listenport=8000 listenaddress=0.0.0.0 `
  connectport=8000 connectaddress=$(wsl hostname -I).Trim()

This forwards port 8000 on the REX host IP to the WSL2 internal IP. Other hosts on the lab network can now reach the vLLM API at http://172.16.50.5:8000 as if it were a normal Linux service. The firewall allows traffic from the VM VLAN to the host on port 8000.

In LiteLLM (covered in the next post), this endpoint is registered as claude-sonnet-4-6 — an alias that lets clients request Claude by name and get Gemma4 locally. This dual identity is intentional: it allows the AI assistant to use local inference without the client needing to know which backend is serving the request.

The Use Case: Always-On Inference for Automation

The lab generates enough AI queries that running everything through cloud APIs would produce a non-trivial monthly bill. Daily security summaries, home automation queries via Home Assistant, and various automation scripts all hit the LLM endpoint.

With local inference:

  • Latency: First token in ~800ms on a fresh cold request
  • Throughput: ~45 tokens/second output at FP8 KV cache settings
  • Cost: Electricity. The RTX 5090 draws ~300W at full inference load.

At Queensland residential rates (~$0.33/kWh), a full hour of continuous inference costs about $0.10. Most inference tasks complete in seconds, not hours. The economics work out favourably.

The more interesting benefit: the model runs when the internet doesn’t. The NBN connection at this address has not been perfectly reliable. Local inference means the AI assistant keeps working during ISP incidents, which is exactly when you might want it to run the automated network summary.

What’s Next

A local model is useful. A local model that can be selected intelligently — use the cloud model when quality matters, use the local model when latency matters or internet is unavailable — is more useful. That requires a proxy layer that knows about both options. That’s LiteLLM.