Running Qwen 3.8 27B locally on an RTX 4090
I run Qwen 3.8 27B locally on a single RTX 4090 (24 GB). Here's the setup and the numbers I get.
Setup
- Loader: LM Studio (llama.cpp / Ollama underneath)
- Quant: Q4_K_M — the sweet spot for quality vs VRAM on 24 GB
- Context: kept moderate; large contexts eat the KV cache fast
What I observed
The 4090 handles a 27B Q4 comfortably. Model + KV cache lands around 17–19 GB VRAM, leaving headroom for the desktop.
| Setting | Observed |
|---|---|
| Quant | Q4_K_M |
| VRAM used | ~17–19 GB (typical for 27B Q4) |
| Tokens/sec (single user) | ~45–55 tok/s (generation) |
| Cold load time | ~6–10 s |
| Max comfortable context | ~16–24k tokens |
These are typical figures for a 27B Q4 on a 4090; your exact numbers shift a little with driver, loader and context size.
To measure quickly:
# LM Studio → read "tokens/sec" from the server log,
# or with Ollama:
ollama run qwen3.8:27b-q4_K_M "Write a paragraph about mass spectrometry."
Why local for this model
Keeping inference on your own GPU is the clean way to stay private — nothing leaves the machine. That's the same reasoning behind why local inference matters for sensitive work, and it's the same "keep it on your own hardware" pattern I used for the targeted RAG for app notes. The 4090 makes a 27B model practical on a single consumer card.