Best GPUs for running local LLMs

Picking a GPU for local LLM inference comes down to VRAM (does the model fit?), memory bandwidth (how fast it generates), and software support. The list below is ranked by how many community reports each card has on llamaperf — a rough proxy for how heavily it gets used in practice — and surfaces the fastest tokens-per-second observed on each.

Ranked from 59 community reports on llamaperf.

Ranked by community reports

#	GPU	VRAM	Reports	Fastest t/s
1	RTX 5090nvidia	32GB	10	3238.0
2	RTX 3090nvidia	24GB	7	72.9
3	RTX 3060 12GBnvidia	12GB	5	70.0
4	M5 Max 128GBapple	128GB	5	7.5
5	RTX Pro 6000 Blackwellnvidia	96GB	3	3500.0
6	RTX 4090nvidia	24GB	3	149.6
7	H100 80GBnvidia	80GB	3	125.3
8	M5 Max 64GBapple	64GB	3	97.0
9	RX 7900 XTXamd	24GB	3	58.0
10	AMD Strix Halo 128GBamd	128GB	3	21.2
11	M2 Max 96GBapple	96GB	2	28.0
12	AMD MI50 32GBamd	32GB	2	9.7
13	RTX 4070 Ti Supernvidia	16GB	1	110.2
14	RTX 5080nvidia	16GB	1	56.0
15	RTX 4070nvidia	12GB	1	55.0
16	RX 9070amd	16GB	1	46.9
17	RTX 5060 Ti 16GBnvidia	16GB	1	22.0
18	M1 8GBapple	8GB	1	17.5
19	RTX A6000 48GBnvidia	48GB	1	16.9
20	AMD Threadripper 256GBamd	256GB	1	7.5
21	M3 Max 128GBapple	128GB	1	5.5
22	DGX Sparknvidia	128GB	1	—

No reports yet

These match the profile but nobody has submitted a report yet.

What to look for

VRAM is the gating constraint

Whether a model runs at all is decided by memory. A Q4_K_M quant of a 7B model needs ~5GB; a 13B needs ~8GB; a 30B needs ~20GB; a 70B needs ~40GB — plus headroom for context and KV cache. If the weights don't fit, generation either crawls (CPU offload) or fails outright.

Bandwidth determines tokens-per-second

Once weights fit, throughput is dominated by memory bandwidth, not raw FLOPs. An RTX 3090 (936 GB/s) and an RTX 4090 (1008 GB/s) are within ~10% of each other on inference-bound workloads despite the 4090's much larger compute budget. M-series Macs trade off here: massive memory pool, but Pro-tier bandwidth is closer to a midrange discrete card.

Software support gates which engines you can use

NVIDIA has CUDA kernels in every major engine (llama.cpp, vLLM, exllamav2, TensorRT-LLM). AMD support has improved sharply via ROCm but still trails on engine coverage. Apple Silicon is best-in-class for MLX and llama.cpp Metal but unsupported by vLLM. Match the engine you want to use to the hardware ecosystem.

Frequently asked

What is the best GPU for running local LLMs?

There is no single answer — it depends on which model size you want to run. For 7B–13B models, an RTX 3060 12GB or RTX 4060 Ti 16GB is enough. For 30B-class models, an RTX 3090 or 4090 (24GB) is the sweet spot. For 70B-class, you need 40GB+ of VRAM (RTX A6000, dual 3090s, or an M-series Mac with 64GB+ unified memory).

Is more VRAM or more compute better for local LLMs?

VRAM, by a wide margin. Inference throughput is memory-bandwidth bound, not compute bound. A card with enough VRAM to fit your model and decent bandwidth will outperform a faster GPU that has to offload weights to system memory.

Do I need an NVIDIA GPU for local LLMs?

No. AMD GPUs work via ROCm with most major engines, and Apple Silicon Macs run llama.cpp Metal and MLX natively. NVIDIA still has the broadest engine support and best out-of-the-box experience, but it's no longer the only option.

How is this list ranked?

By the number of community submissions on llamaperf for each GPU. More reports indicate a GPU is widely used in practice for local LLM inference. The fastest tokens-per-second observed on each is shown alongside as a quality signal.

How we rank

Hardware is sorted by the number of community submissions on llamaperf — a proxy for how widely each card is used in practice for local LLM inference. Within that, we surface the fastest tokens-per-second observed on each as a quality signal. Submissions come primarily from r/LocalLLaMA discussions and direct user uploads. Nothing here is sponsored or affiliate-driven.