LOCAL LLM GATE
No cloud. No walls. Run trillion-parameter LLMs on your own iron.
A hands-on field manual — from the first concepts to a working local endpoint.
What a “large language model” actually is
In one line: an LLM is a probability machine force-fed hundreds of billions of tokens. What it learned isn’t the text itself but the patterns and co-occurrence structure between words. It’s not a database — it’s a statistical brain that compressed the world.
Key parts:
tokenization → embedding → several
Transformer blocks (self-attention / feed-forward) → output probability distribution.
"hello world" ──▸ [ Tokenizer ] ─▸ [1024, 98]──┐
(split into tokens) ▼
each Token ──▸ [ project to vector + positional enc ]
▼
┌──────────── Transformer × N layers ──────────┐
│ · multi-head self-attention: who relates? │
│ └─ the real source of long-range deps │
│ — and of KV memory burn │
│ · residual + LayerNorm + RoPE │
│ · feed-forward FFN: the param-heavy │
│ "memory warehouse" │
└──────────────┬─────────┬──────────────────────┘
▼ ▼
Logits → Softmax ──▸ next-token distribution
│
argmax / top-p sampling ──▸ "world hello the ×"
（that is the text you actually see）
Two big VRAM costs: ① weights (the furnace) ② the runtime-growing KV cache
(temporary context memory that lengthens with the conversation).
Grasp these two and every “VRAM and quantization” argument downstream falls into place.
Why you should bring the model back home
Outsourcing every prompt to the cloud is, fundamentally, streaking your thinking in front of a third party.
Going local buys back sovereignty, privacy, control, and an offline freedom with no leash.
| Dimension | Cloud API | Local, self-hosted |
|---|---|---|
| Privacy | Data leaves your machine · terms may change | Fully controllable · never leaves the box |
| Availability | Needs network and an account | Works offline |
| Cost model | Per-token billing · queues at peak | One-off hardware · then mostly electricity |
| Customization / fine-tuning | Limited | Total freedom |
| Ceiling | Biggest, newest, most complete | Limited by your VRAM / RAM |
| Maintenance | None (someone else's problem) | You set up, upgrade and babysit it |
Hardware: VRAM decides how big a brain you can keep
The biggest law of physics for local deployment is VRAM. Don’t rush out and buy a card — first run one formula to nail down your requirement, then talk about anything else.
Will it fit? ── weights ≈ params(B) × bits-per-weight ÷ 8 [+ 2~4GB runtime / KV buffer]
Example: Qwen3-14B running Q4_K (≈ ~0.61 bytes per weight)
→ weight baseline ≈ 14 × 0.61 ≈ 8.5 GB + system/KV ≈ 1~3 GB
→ you need at least ~12GB VRAM to drag it around comfortably
(longer context = more KV = the requirement creeps up)
Key takeaway: what eats VRAM is “parameter count × precision”, not “how smart the model is”. The bar to comfortably run 8B inference is now as low as 6–10GB of VRAM (or pure CPU with plenty of RAM — just slower).
| Your VRAM (roughly) | Sweet-spot models (after Q4) | Notes |
|---|---|---|
| < 6GB (iGPU / old card) | 1B–4B: Qwen3-1.7B/4B, Llama3.2-3B, Phi-4-mini, Gemma-2B | Pure CPU also fine — for fun / edge |
| 8–12GB (2060/3060/RX6xxx) | 7B–8B: Qwen3-8B, DeepSeek-R1-7B/8B, qwen coder 7B | The entry sweet spot. Punches hard |
| 16–24GB (4060Ti-16/4070Ti/4080/4090) | 14B–34B: Qwen3-14B/32B, R1-14B/32B, Gemma3-27B | One card, sideways. Nice |
| 24–48GB (7900XTX ×2 / 4080 ×2) | Qwen3-30B class / R1-32B large ctx / MoE 235B or 70B Q5 | Workstation watershed; start agents/RAG |
| ≥ 2×24GB or big multi-GPU | 70B-class Q4 or MoE-Large (R1-70B, LLaMA3.3-70B, Qwen3-235B neighbours) | Multi-GPU tensor split; hobbyist small-server territory |
llama.cpp / Ollama both support pure CPU; enable AVX2 / Apple’s Metal / AMD’s Vulkan to speed up.Model arsenal: open-weight models worth running locally
Only ones that a personal PC / small workstation can actually move, with open lineage and a solid reputation. VRAM is estimated after quantization; for precision, read the official repository and licence.
VRAM figures are estimates after quantization. They shift with engine, context length and KV-cache settings — always trust your own ollama ps / nvidia-smi. Verify licences and parameter counts against the publisher's repository, Ollama or Hugging Face.
| Model | Publisher | Type | Licence | Available sizes | Rough VRAM after quant |
|---|---|---|---|---|---|
| DeepSeek-R1 | DeepSeek · China | Reasoning | MIT | 1.5b · 7b · 8b · 14b · 32b · 70b · 671b | 7B quant ≈ 4–6 GB · 32B ≈ 20 GB+ · one of the most-run local reasoners |
| DeepSeek-V3 family | DeepSeek · China | General · chat | MIT | 671B-A37B (smaller recipes also open) | Needs multi-GPU · Q4 still ~380 GB+ · small rigs should use R1 distills or an API |
| Qwen3 | Alibaba · open | General · chat · reasoning | Apache-2.0? | 0.6b · 1.7b · 4b · 8b · 14b · 30b · 32b (A3B MoE) · 235b | 8B/14B quant 8–12 GB, smooth · MoE 32B is even lighter |
| Qwen2.5-Coder | Alibaba · open | Code | Apache-2.0? | 0.5b · 1.5b · 3b · 7b · 14b · 32b | 7B/14B quant 6–12 GB · pairs beautifully with a local IDE |
| Llama 3.2 / 3.3 | Meta | General · edge | Llama 3 | 3.2: 1b · 3b | 3.3: 70b | 3B runs on pure CPU · 1B is phone-class · 70B Q4 ≈ 45 GB |
| Gemma 3 | Google DeepMind | General · vision · edge | Gemma | 1b · 4b · 12b · 27b | 4B/12B quant 4–10 GB |
| Phi-4 / Phi-4-mini | Microsoft | Chat · general | MIT | 3.8b · 14b | 3.8B ≈ 3 GB · drops straight into an edge box or phone |
| GLM-4-9B family | Zhipu Z.ai | General · chat | MIT | 9b (plus long-context, vision and air variants) | Q4 ≈ 5–6 GB · excellent for Chinese-language work |
| InternLM3-8B | Shanghai AI Lab | Chat · general | Apache-2.0? | 8b | 8–10 GB comfortably |
| Mistral / Nemo | Mistral AI | General · edge | Apache-2.0? | 7b · 12b (Nemo) | 7B ≈ 6 GB and up |
| gpt-oss-20b | OpenAI | Reasoning · code | Apache-2.0 (MIT)? | 20b · 120b (MoE) | 20B (GPT-OSS-20B-A4B) Q4 ≈ 11 GB |
| Qwen3-Coder / frontier preview | Alibaba / multi-source | Code · general | Apache-2.0? | 30b-A3B (agent variants too) | Q3/Q4 ≈ 12–16 GB |
Deployment: actually getting the model onto the machine
For newcomers always start with Ollama (one command) or
LM Studio (drag and drop GUI); move to llama.cpp / vLLM when you want
performance and control. Four paths below, easy to hard, all spelled out.
Main path · Ollama (Windows / macOS / Linux)
# 1) Install: https://ollama.com/download (Win/mac download & go) or Linux: $ curl -fsSL https://ollama.com/install.sh | sh # 2) Pull and run a model (auto-quantized download) | a solid 7B-class generalist $ ollama run qwen3:8b >>> hello, this is 100% local. # 3) Swap in something that "thinks": the DeepSeek-R1 distills $ ollama run deepseek-r1:8b # 4) List local models / exit / set concurrency and context $ ollama list # installed inventory $ /bye # exit the session $ env OLLAMA_CONTEXT_LENGTH=32768 ollama run qwen3:8b
# Is the service up? $ curl http://localhost:11434/api/tags {"models":[{... "name":"qwen3:8b" ...}]} # Fire one minimal request: $ curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "qwen3:8b", "messages": [{ "role": "user", "content": "tell me a cold-start joke only a geek would get" }] }' # So: point OPENAI_BASE_URL in your .env at this machine and almost any client flips to local. # e.g. (Python / OpenAI SDK): OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")
qwen3 8b or deepseek r1 →
pick a GGUF tag (Q4_K_M is a good default) → Download → hit Chat / Local Server top-right
and you get an OpenAI-compatible endpoint at `http://localhost:1234`. All mouse, great for demoing to a
non-technical teammate.ollama pull qwen3:8b-q4_K_M; want smaller and slower use -q3, more accurate -q8.
Many publishers on HuggingFace ship single-file Qwen_Q3/…_K_M.gguf; download it and one line
ollama create name -f ./Modelfile registers it as a local model.
③ llama.cpp — single GGUF file, runs on CPU too
# 0) Build it, or grab a prebuilt binary from GitHub Releases (ggml-...-bin) $ git clone --depth 1 https://github.com/ggml-org/llama.cpp && cd llama.cpp $ cmake -B build -DGGML_NATIVE=ON && cmake --build build --config Release -j # 1) Get a .gguf (from HF, for instance) wget https://…/models/…/qwen3_8b_q4_k_m.gguf # 2) Pure CPU / -ngl sets offloaded layers; -c context; -t threads; --jinja native template $ ./build/bin/llama-server -m ./qwen3_8b_q4_k_m.gguf \ --port 8080 -c 8192 -t 8 --jinja [..] server is listening on http://0.0.0.0:8080 # 3) Structured LLM apps usually hit /v1/chat/completions or /completion here $ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d \ '{"model":"qwen3","messages":[{"role":"user","content":"hi"}]}' # Tip: llama-server's edge is control over kernel params plus streaming / JSON-schema-guided output
④ vLLM — when you want production-grade throughput
# Needs an NVIDIA / decent CPU environment, Python 3.10+; on Windows use WSL2 or Docker $ pip install vllm # Serve HF weights such as Qwen or Llama: $ python -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen3-8B \ --gpu-memory-utilization 0.9 \ --max-model-len 16384 \ --port 8000 INFO: Started server process … http://0.0.0.0:8000 # It is now an OpenAI API — test it $ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \ -d '{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"write two lines of hello world"}]}' # Advanced combos: batch(packing) huge ctx / prefix caching / speculative decoding / tensor-parallel=2
| Engine | Learning curve | Strength | Who it's for | Platform notes |
|---|---|---|---|---|
| Ollama | ★ minimal | Minimal commands · auto-quantize · broad ecosystem | Beginners / the default for 80% of cases | Native Win/mac/Linux |
| LM Studio | ★ GUI | Visual model testing · one-click endpoint | Desktop folks / demos | Win/macOS |
| llama.cpp | ★★★ | Low-level GGUF control · best CPU compatibility | Deep customization / old machines | Win needs Git/CMake |
| vLLM | ★★★★ | High-throughput batching · OpenAI serving | Production / small multi-user service | Prefer CUDA / WSL2 |
| Docker (jetson etc.) | ★★ | Environment isolation · GPU passthrough | Zero-setup machine swaps / trying images | Win uses WSL2 backend |
Metal runs 8B fast and cool.
Ollama/llama.cpp both support it natively; don't believe the stale joke that "Macs can't do AI".
When picking models, look for `Metal / GGML` optimizations.ollama show or a run is extremely slow it is probably not using the GPU —
check that PROCESSOR in ollama ps says GPU. Clear these before moving on.Quantization deep end: compressing your iron heart
Same model, but the gap between Q8 and Q3 is not file size — it is the edge of “does every sentence still land”. Understand the mechanics and you can plant your flag firmly between size × speed × feel.
Q4_K_M or Q5_K_M runs, use those — don’t gamble on 2% of feel;
② when tight, cut context rather than bits — slimming the KV (e.g. KV-quant) is often a better trade
than hacking weights; ③ for code / long reasoning, prefer L+KV-quant or a capacity step up over Q3.War stories: pits I already fell into, so you don't have to
Symptom → cause → fix. Open the hole that looks like wherever you are stuck.
■It installed, but now it babbles fast / spews garbage+
Cause: quantization too low (Q2/Q1) or the context was artificially squeezed; the default temperature may also be too high.
Fix: switch to q4_K_M/q5_K_M; set num_ctx (OLLAMA_CONTEXT_LENGTH) to 8192+; drop temperature to 0.6–0.8; for code tasks keep top_p near 0.9. Still broken? Try that model's original F16 once to determine whether the damage is “quantization” or “the data was always weak”.
■GPU is clearly there, but it is slow / Ollama keeps chewing CPU+
Cause: it never actually landed on the card; driver or model layer was not recognized as GPU-capable.
Fix: check whether PROCESSOR in ollama ps says GPU; on Windows install the matching CUDA driver and use a supported backend; for llama.cpp check the -ngl layer count and offload in the logs. Don’t be fooled by “it started” — seeing is believing.
■Huge files: downloads stall halfway / VRAM never enough+
Cause: you are downloading the original FP16 rather than a quantization; or the context is pushed to the ceiling and KV blows up.
Fix: find a …-GGUF repository and pick Q4_K_M; start at -c 4096 and creep upward; if that is still not enough consider KV-cache quantization or --flash-attn. To do it in one shot, pick the model size that matches your VRAM tier.
■Client can't reach local / port already in use / CORS error+
Cause: the service isn't listening on 0.0.0.0; or traffic still goes through another proxy; or the browser blocked cross-origin.
Fix: confirm the endpoint answers with curl http://localhost:11434/v1/models; use the official web UI or enable the CORS switch; try another browser hitting 127.0.0.1 directly. Kill clash / system proxy and retry — that alone usually fixes it instantly.
■OOM crash / machine turns into a slideshow / swap storm+
Cause: weights + KV + runtime exceed physical VRAM, so the system falls back to RAM/disk thrashing.
Fix: drop one model tier or bit width; squeeze -c context; enable flash attention & KV quantization; close browser/AI processes competing for VRAM. If it still crashes, this model simply is not for this machine — downsizing is not shameful.
■Multilingual: small models only speak English / weak at other languages+
Cause: training corpora skew English-heavy; small parameter counts under-cover other languages.
Fix: for non-English work prefer the Qwen / GLM / InternLM / DeepSeek families and language-boosted variants; watch for chat/…zh in the model name; don't expect “full-strength academic-grade” output from a small model — when it is time to go 8B→14B, don't scrimp.
■Downloads crawl / HF is unstable from your region+
Cause: direct connections to huggingface.co are slow from some networks.
Fix: use a mirror via HF_ENDPOINT=https://hf-mirror.com; Ollama uses its own distribution channel; if needed run hf download with hf_transfer for multithreading; pull only the quantization file you need rather than a whole snapshot.
■Shared by several people and always saturated / vLLM OOMs then idles+
Cause: no concurrency or batching was configured, so every request monopolizes KV and VRAM.
Fix: add Continuous batching at the serving layer (Ollama / OpenWebUI have queues), cap max model len,
spread across tensor-parallel (multi-GPU) or data-parallel (multiple instances); don't hand everyone a 128k context.
1> Logs first — server stdout always has a clue;
2> Minimal reproduction — strip RAG/proxies/plugins and curl the bare model;
3> Version check — Ollama/llama.cpp are rolling releases, so run ollama update first. Don't dive straight into the source.Tuning & ecosystem: from “it runs” to “it's good”
Deployment is only step one. Getting throughput, context and tooling to shake hands with you is where the real engineering starts.
bench first, then act.kv cache reuse / prefix caching
makes long conversations feel more responsive.function-calling: hand it JS/JSON schemas and use
guided decode/grammar to guarantee valid output. Front-ends like
OpenWebUI / LobeChat / Continue (IDE) take over interaction and double the experience instantly.LoRA fine-tuning to learn your domain,
DPO/RMM to shape persona, and save an FP16 copy before quantizing as a baseline.
Version your Modelfile so you can roll back when you trip.| What you want | Recommended first step |
|---|---|
| A full local chat UI | OpenWebUI (containerized + multi-user + web search plugins) |
| AI code completion in your IDE | Continue / Tabby pointed at localhost's OpenAI endpoint |
| Wire AI into your own Python | OpenAI SDK with a changed base_url / or ollama-python |
| Runs on an old machine / no GPU | llama.cpp + a Q4 tier + batching |
| Small server / production throughput | vLLM + AWQ + prefix caching + tensor-parallel |
Community / data HuggingFace(huggingface.co) · Ollama library(ollama.com/library) · Open LLM leaderboard(HF Space) · each model's official repo and licence page
Tools / sentinels Local Remote (local comparison) · VRAM calculator · GGUF-KoboldCpp · CT2-Transformers
privilege escalation point. Never let it run as bare root;
run it in a container, least privilege, separate tokens, sandbox and review content; don't hand private keys to
model-generated code casually. Play wild, but bolt the back door.