$ hardware --check
What you actually need to run a model locally
Most local-LLM advice is either a shopping list from someone selling hardware or a benchmark run on a machine nobody owns. Here is the constraint that actually decides it, and the honest answer about when not to bother.
Updated 2026-09-10
The short answer
The binding constraint for running a language model locally is VRAM, not raw compute. As a rule of thumb in 2026, a 7-8B model at 4-bit quantisation needs about 6GB of VRAM, a 13-14B model about 10GB, and a 30-34B model about 24GB. Below roughly 8GB you are limited to small models, and running from system RAM instead of VRAM typically costs you an order of magnitude in speed.
4-bit quantisation, 8K context, grouped-query attention assumed. The KV cache is the part most calculators leave out - and the part that decides whether a model fits.
- Weights
- KV cache (8K ctx)
- Runtime overhead
| 7-8B | Weights: 4 GB | KV cache: 0.9 GB | Overhead: 0.7 GB | fits 8GB |
|---|---|---|---|---|
| 13-14B | Weights: 7 GB | KV cache: 1.4 GB | Overhead: 1.3 GB | fits 12GB |
| 30-34B | Weights: 16 GB | KV cache: 2.1 GB | Overhead: 2.7 GB | tight on 24GB |
| 70B | Weights: 35 GB | KV cache: 3 GB | Overhead: 5.7 GB | needs 48GB+ |
VRAM is the constraint, everything else is secondary
- A model must fit in VRAM to run at usable speed - once it spills to system RAM, throughput collapses
- Quantisation is what makes local models practical: 4-bit roughly halves the memory of 8-bit for a modest quality cost
- Context length costs memory too, and it is the part people forget - a long context can add gigabytes on top of the weights
- Apple Silicon is the exception worth knowing: unified memory means a 32GB Mac can hold models a 12GB discrete GPU cannot
What each tier realistically gets you
- 8GB VRAM - 7-8B models at 4-bit, short context. Fine for summarising, classification and drafting
- 12-16GB - 13-14B comfortably, or 7-8B with long context. The practical sweet spot for most people
- 24GB - 30-34B models, or smaller models served to several users at once
- 48GB and up - 70B class. At this point compare the hardware cost honestly against an API bill
When you should not run it locally
- Bursty or occasional use - an idle GPU costs the same as a busy one, an API costs nothing when nobody is asking
- You need the strongest available reasoning - the best hosted models remain ahead of what fits on one card
- Nobody on the team wants to own updates, drivers and quantisation choices as an ongoing job
- The honest test: sustained daily use is what justifies hardware. Curiosity does not
Where local genuinely wins
- Data that must not leave the building - the argument that beats every cost calculation
- High, steady volume where per-token pricing has become the largest line on the bill
- Latency-sensitive work where a round trip to a provider is the bottleneck
- Offline or air-gapped environments, where there is no other option
Common questions
Can I run a useful model on a laptop with no discrete GPU?
Yes, but expect small models and modest speed. A recent Apple Silicon machine does this far better than an Intel laptop, because unified memory lets the GPU address far more than a typical integrated chip. On a CPU-only x86 laptop, treat it as a learning exercise rather than a working tool.
Is a used data-centre GPU a good idea?
Sometimes, and read the details first. They often need specific cooling, take a lot of power, and may not fit a normal case. The card being cheap does not make the build cheap.
How much quality do I lose to quantisation?
At 4-bit, less than most people expect for everyday tasks, and more than you want for careful reasoning. Test it on your actual task rather than trusting a benchmark - a summarisation job and a code-refactoring job degrade very differently.