CPU-Only LLMs: What Actually Works

📚 More on this topic: VRAM Requirements · GPU Buying Guide · Quantization Explained

No GPU? No problem — mostly.

CPU-only inference gets dismissed as unusable, and that’s wrong. It’s slower, yes. A 7B model that runs at 40+ tok/s on an RTX 4060 runs at 10-15 tok/s on a decent CPU. But 10-15 tok/s is still faster than you can read. For chat, quick coding questions, and summarization, that’s enough.

What nobody tells you is that CPU inference has real advantages: it’s stable (no CUDA driver nightmares), cheap (RAM costs a fraction of VRAM per gigabyte), and scales to enormous models. A ~$1,100 dual Xeon server with 128GB of RAM can run a 70B model that would need a $1,600 RTX 4090 — or two GPUs — to fit in VRAM.

This guide covers two tiers: what works on your existing laptop or desktop, and a researched budget server build for people who want to go bigger.

Who This Is For

• Laptop users with no discrete GPU (or a weak one) who want to try local AI
• Desktop users without a GPU, or with an older card that’s not worth using
• Budget builders who’d rather spend ~$1,100 on a full server build than on a single high-end GPU
• Quiet/stable operation — no GPU fan noise, no driver updates breaking things
• Privacy-first users who want local inference on whatever hardware they have

If you own a GPU with 8GB+ VRAM, you should use it. GPU inference is 3-6x faster for the same model. This guide is for people who can’t or don’t want to go that route.

The Honest Truth About CPU Inference

CPU inference is memory-bandwidth limited. When generating text, the CPU reads the entire model’s weights from RAM for every single token. The speed of that read — your memory bandwidth — determines your tokens per second, not your core count or clock speed.

This is why a 16-core CPU isn’t twice as fast as an 8-core for inference. Both are waiting on the same memory bus. It’s also why RAM speed and channel count matter far more than the CPU itself.

Here’s the baseline you can expect:

*Requires 48GB+ RAM. Slow but functional.

At 10+ tok/s, responses feel like a fast typist. At 3-5 tok/s, you’re watching words appear one by one — usable for batch work, not great for interactive chat. Below 2 tok/s, it’s a screensaver.

The Advantages Nobody Mentions

GPU inference gets all the attention, but CPU-only has underrated strengths:

• No driver hell. No CUDA version mismatches, no ROCm compatibility matrix, no “works on 535.xx but crashes on 545.xx.” llama.cpp on CPU just compiles and runs.
• RAM is cheaper per gigabyte. 128GB of DDR4 ECC RDIMM costs ~$800-1,000 used. 24GB of GDDR6X (an RTX 4090) costs $1,600+. You get 5x the memory capacity for half the price.
• Stability. CPU inference doesn’t crash when VRAM fills up. Models memory-map from disk, context windows scale with available RAM, and the OS manages memory gracefully.
• Silent operation. No GPU fan spinning up to 80%. CPU coolers under inference load are barely audible.
• Runs models GPUs can’t. A 70B model at Q4 needs ~40GB. That doesn’t fit on any consumer GPU. On a server with 128GB RAM, it runs without tricks.

What Affects CPU Inference Speed

1. Memory Bandwidth (The Big One)

Token generation speed scales almost linearly with memory bandwidth. This is the single most important factor.

The takeaway: a dual Xeon with 8 channels of slow DDR4 beats a modern desktop with 2 channels of fast DDR5 — because total bandwidth wins over per-channel speed.

2. Channel Count

More RAM channels = more bandwidth = more tokens per second. This is why server hardware matters.

Populate all channels. One DIMM per channel beats two DIMMs on fewer channels. On a dual Xeon board with 8 DIMM slots, use 8 sticks — even if the total capacity is the same.

3. Core Count (Less Than You Think)

More cores help with prompt processing (the initial “thinking” phase) but barely affect token generation. In benchmarks, switching from 6 threads to 12 threads on the same CPU gave almost no improvement in generation speed (5.37 vs 5.33 tok/s).

The rule: Set thread count to your physical core count (not logical/hyperthreaded). On an i7-13700, that means 8 P-cores — ignore the E-cores and hyperthreading.

4. AVX-512 Support

AVX-512 dramatically accelerates prompt processing (2.8-10x faster) but doesn’t help token generation because generation is memory-bound, not compute-bound.

*Some i9-12900K chips have AVX-512 accessible via BIOS, but Intel disabled it on later steppings.

If you’re buying a CPU for inference, Zen 4+ with AVX-512 is the best consumer option. But remember: it mostly speeds up the prompt phase, not generation.

Tier 1: Laptop & Desktop (Consumer CPUs)

This is the starting point. You already have the hardware — the question is what to run on it.

What Fits

The sweet spot: Llama 3.2 3B for speed, Qwen 2.5 7B for quality. If you have DDR5 and 16GB+ RAM, 7B models are comfortable. On DDR4 laptops with 8GB RAM, stick to 3B models.

What Won’t Work on Consumer Hardware

• 13B models: Technically runnable (~7GB RAM at Q4) but painfully slow at 4-7 tok/s on dual-channel DDR. Not a good experience.
• 30B+ models: Need 20GB+ RAM at Q4. Even if your laptop has 32GB, the ~50 GB/s bandwidth means ~3 tok/s. That’s a patience test.
• Anything at FP16: A 7B model at FP16 needs 14GB RAM and runs at half the speed of Q4. No reason to use FP16 on CPU.

Getting Started

# Install Ollama (works on Mac, Linux, Windows)
curl -fsSL https://ollama.com/install.sh | sh

# Run a 3B model (fast, low RAM)
ollama run llama3.2:3b

# Run a 7B model (better quality, needs 16GB+ RAM)
ollama run qwen2.5:7b

Ollama auto-detects CPU-only systems and runs inference on CPU. No configuration needed.

Tier 2: Budget Server Build (Dual Xeon)

This is where CPU-only gets interesting. Used server hardware is absurdly cheap, and a dual-socket Xeon build gives you 8-channel memory bandwidth and enough RAM to run models that don’t fit on any consumer GPU.

Why Server Hardware

The Xeon E5-2699 v4 launched at $4,115 per chip in 2016. Today, a pair costs $180-300 on eBay. The same collapse happened to motherboards and coolers. You can build a system with 44 cores, 128GB RAM, and 154 GB/s memory bandwidth for roughly the cost of a used RTX 3090.

The secret: memory bandwidth scales with channels, not speed. Eight channels of DDR4-2400 (~154 GB/s) beats two channels of DDR5-6000 (~96 GB/s). For LLM inference, this is the only number that matters.

The Build: Dual Xeon E5-2699 v4

Total: $1,510-2,080 (budget to mid-range)

RAM is the main cost bottleneck. Prices fluctuate — check current eBay listings for DDR4 ECC RDIMM before budgeting. Note: Xeon E5 v4 requires ECC RDIMM (Registered) memory — regular desktop DDR4 won’t work.

Budget-optimized version (Chinese X99 dual board, generic coolers, used PSU):

Budget total: $1,250-1,650

Expected Performance

The 7B-14B range is the sweet spot on this build. Fast enough for interactive use, with tons of RAM headroom for long context windows. The 70B tier works but at “patience required” speeds — useful for generating long documents, not for back-and-forth conversation.

Build Tips

• Populate all 8 DIMM slots. One stick per channel (4 per CPU) maximizes bandwidth. 8x16GB beats 4x32GB even though both are 128GB.
• Use DDR4-2400 if you can. DDR4-2133 works but leaves ~12% bandwidth on the table. The price difference is minimal.
• ECC RDIMM is mandatory. Xeon E5 v4 won’t boot with unbuffered or non-ECC RAM.
• Check ILM type on your motherboard. LGA 2011-3 has Square ILM (80x80mm) and Narrow ILM (56x94mm). The Noctua NH-U12DX i4 supports both. Most consumer tower coolers only support Square.
• Power draw is real. Two E5-2699 v4 at load pull 300-400W for the CPUs alone. Budget 500-600W total system draw. Your electricity bill will notice.
• Disable Sub-NUMA Clustering (SNC) in BIOS. Set 1 NUMA node per socket. Run inference with numactl --interleave=all for best performance.
• E5-2696 v4 is the same silicon as the E5-2699 v4 (same 22C/44T, same cache) but sold as an OEM part. Often $30-50 cheaper per chip. Most boards run it fine, but check BIOS compatibility.

Software: llama.cpp Is the Answer

For CPU-only inference, llama.cpp is the gold standard. It’s written in C/C++ with hand-optimized CPU kernels, supports every quantization format, and runs on everything from Raspberry Pis to dual-socket servers.

Why llama.cpp

• Native CPU optimization: AVX2, AVX-512, ARM NEON — it uses whatever your CPU supports
• Memory-mapped model loading: Models load from disk on demand, no huge RAM spike at startup
• Every quantization format: Q2 through Q8, K-quants, IQ-quants — all supported
• No Python dependency hell: Compiles from source in minutes, no conda environments

CPU-Optimized Setup

# Clone and build with CPU optimizations
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

# Run a model (set threads to physical cores)
./build/bin/llama-cli \
  -m models/qwen2.5-7b-q4_k_m.gguf \
  -t 8 \
  -c 4096 \
  --mlock \
  -ngl 0 \
  -p "Explain memory bandwidth in one paragraph."

Key flags:

Or Just Use Ollama

If you don’t want to compile anything, Ollama wraps llama.cpp with automatic model downloading and a clean interface. It detects CPU-only systems automatically.

ik_llama.cpp: The Faster Fork

The ik_llama.cpp fork has optimized CPU kernels that nearly doubled inference speed on Broadwell Xeon hardware in benchmarks (2.70 to 5.05 tok/s on 70B models). Worth trying on older Xeon builds specifically.

Quantization Matters Even More on CPU

On GPU, the difference between Q4 and Q8 is modest — maybe 20-30% speed difference. On CPU, it’s dramatic. Every extra bit per parameter means more data read from RAM per token, and RAM bandwidth is your bottleneck.

Notice that Q3 and Q2 aren’t faster than Q4 despite being smaller. The dequantization compute overhead eats the bandwidth savings. Q4_K_M is the sweet spot on CPU — below that, you lose quality without gaining speed.

The rule: Always use Q4_K_M or Q4_K_S for CPU inference. Only go to Q5 if you have bandwidth to spare (server builds). Never go below Q3 — it’s slower and worse.

RAM Requirements

The model must fit entirely in RAM. Unlike GPU inference where partial offloading is an option, CPU inference loads everything into system memory. If the model is larger than your available RAM, the OS will swap to disk, and inference drops to near-zero speed.

Leave at least 4GB free for the OS and other processes. On a 16GB laptop, your practical limit is a 7B model at Q4. On a 32GB desktop, you can stretch to 13B. For 30B+, you need 64GB minimum.

→ Check what fits your hardware with our Planning Tool.

CPU-Only vs. Saving for a GPU

Here’s the honest comparison:

Choose CPU-Only If:

• You already have the hardware and want to try local AI today — $0 investment
• You need to run 30B-70B models and can’t afford a $700+ GPU with 24GB VRAM
• You’re building a quiet, stable home server for background inference
• You value simplicity — no GPU drivers, no CUDA, no VRAM management
• You’re considering a dual Xeon server build as a fun project

Save for a GPU If:

• Interactive speed matters — you want 30+ tok/s for real-time chat
• You mainly need 7B-14B models (fit comfortably in 8-12GB VRAM)
• You also want image generation (Stable Diffusion needs a GPU)
• Budget allows a used RTX 3060 12GB (~$200) or RTX 3090 (~$700-850)

The $200 used RTX 3060 12GB is the inflection point. If you can afford it, GPU inference on 7B-13B models is 3-6x faster than any CPU. But if you need 70B models or have $0 to spend, CPU-only is a legitimate path.

The Bottom Line

CPU inference is slower than GPU. That’s the tradeoff. What you get in return is simplicity, stability, cheaper memory, and the ability to run models that don’t fit on any consumer GPU.

For laptop/desktop users:

1. Install Ollama and run llama3.2:3b. You’ll be chatting in two minutes.
2. If you have 16GB+ RAM and DDR5, step up to qwen2.5:7b at Q4.
3. Stick to Q4_K_M quantization — it’s the fastest format that doesn’t sacrifice quality.

For server builders:

1. Source a dual Xeon E5-2699 v4 (or 2696 v4) and a Supermicro X10DAi from eBay.
2. Fill all 8 DIMM slots with DDR4-2400 ECC RDIMMs — bandwidth is everything.
3. Run llama.cpp with numactl --interleave=all and threads set to physical core count.
4. Start with Qwen 2.5 14B at Q4 for the best speed/quality balance. Scale to 70B when you need it.

Your CPU is more capable than the benchmarks suggest. Use it.

Related Guides

• How Much VRAM Do You Need for Local LLMs?
• What Can You Run on 8GB VRAM?
• GPU Buying Guide for Local AI
• Build a Local AI PC for Under $500

Sources: OpenBenchmarking llama.cpp, DEV Community DDR5 LLM Benchmark, Hardware Corner Memory Bandwidth Guide, justine.lol CPU Matrix Multiplication, Dual Xeon Gold 5317 LLM Benchmarks, ik_llama.cpp Broadwell Benchmarks, XDA Used Xeon Home Lab, KernelCrash Old Xeon LLMs