81 performance reports, crowdsourced from the community.
Pick your GPU - see which models fit, at which quant, and how fast they run.
RTX 3090 · llama.cpp · 65,536 ctx
Benchmark of Qwen3.6-35B-A3B Q6_K on RTX 3090 with llama.cpp. Tuned config offloads 8 MoE expert layers to CPU, increasing batch sizes, improving PP from 564.5 to 1330.0 tok/s (2.36x) while TG unchanged (~97.7 tok/s). Context 64K, KV cache Q8. CPU: Threadripper PRO 3955WX, ~100GB DDR4. Two-repetition measurements, ~1.6% drift.
User expresses amazement at running a frontier model on a home PC with 24GB VRAM, but notes it's slow.
User is setting up a 16x DGX Spark cluster to run frontier models locally. Mentions DeepSeek V4 pro, Kimi K3, GLM 5.5, and Minimax M4 as future models. No benchmark numbers provided.
Post describes a new MLX quantization method (WinterMix) for Qwen3.5-122B-A10B, with two builds: WinterMix58 (82 GiB) and WinterMix48 (68 GiB). Benchmarks show perplexity improvements over existing MLX quants. The author is enthusiastic about the results and the method's advantages for agentic workflows on Apple Silicon.
Prefill performance mentioned but no numbers given. Decode speeds at various depths: 28 t/s at start, 23.5 t/s at 45k, 18 t/s at 192k. Maintained with 8k token output.
M2 8GB · Turbo-fieldfare
Custom Swift/Metal engine. Also tested on M5 MacBook Pro: 31-35 tok/s. OpenAI-compatible server with streaming and tool-call support.
User built custom w8a8 kernels for M5 MacBook Air. Baseline prefill 2193 tps, improved to 3029 tps for 130k tokens. Also mentions llama.cpp for Macs.
4× RTX 5060 Ti 16GB · 256,000 ctx
Benchmark on Vast AI instance with 4x RTX 5060 Ti 16GB. Q8 quant, FP16 KV cache, MTP enabled. 256K context. Cold prefill 608 t/s, decode 52.2 t/s. User considers this excellent for $2K hardware.
LLM prompt processing benchmark with Kimi K2.5 IQ3_M (80GB offload) at 500W. RTX 5090 achieved 471.40 t/s PP. Also tested GLM 5.1 IQ4_NL (70GB offload) at 574.98 t/s PP. Comparison with RTX 6000 PRO MaxQ shunt modded.
User emphasizes energy efficiency and versatility of Strix Halo compared to Nvidia cards. Mentions running Qwen 3.6 35B at 50 tps on Q8_XL quant.
Custom CUDA/C++ engine, 50-54 tok/s, 50% improvement over llama.cpp (33-34 tok/s).
2× RTX 3090 · llama.cpp
Multi-token prediction enabled. 96GB total VRAM (24+24? but user says 96GB system). Reliable for code generation and codebase ingestion.
2× RTX 3090 Ti · llama.cpp · 196,608 ctx
Tensor split-mode improved t/s from 70+ to 100+. Peak 130 t/s reported. Power draw 750W+.
M2 Max 96GB · llama.cpp
Champion model in vision benchmark. Best quality and stability with thinking disabled. 90/90 successful runs. Speed: 70 s/img. Tested on Apple M2 Max 96GB with llama.cpp b9690.
RTX 5060 Ti 16GB · llama.cpp · 131,072 ctx
User reports that offloading KV cache to RAM (with -nkvo) allows fitting the whole model on GPU with f16 KV cache, achieving 19 tps peak and 14 tps during long generation at 65k context. With 128k context and 63 layers on GPU, speed remained similar. KV cache quant to RAM didn't improve performance.
RX 7900 XTX · llama-swap
Benchmark of Gemma 4 QAT vs regular quants on AMD 7900 XTX. No token/s reported, but wall clock times show significant speedups (e.g., 12B QAT 45% faster, 83% throughput increase). Quality reported identical. Models tested: 12B, 26B, 31B, E4B.
RTX 5090 · vLLM
Benchmark of abliteration tools (Apostate, Huihui, Heretic) on Qwen 2.5 7B. Evaluated with lm-evaluation-harness via vLLM 0.19.0, bf16 on RTX 5090 32GB. Reports MMLU, GSM8K, HellaSwag, ARC Challenge, WinoGrande, TruthfulQA MC2, PiQA, LAMBADA ppl, HarmBench ASR, KL divergence. No tokens/sec reported.
Benchmark of Gemma 4 26B-A4B vs 12B on RTX 4090. 26B-A4B used 15GB VRAM, 138 tok/s; 12B used 9GB, 80 tok/s. 26B-A4B won every scene and ran ~1.7x faster. 12B ideal for 16GB laptop.
RTX 3090 · Ollama · 32,000 ctx
User replaced Claude with Qwen3.6-27B in multi-agent orchestrator for 2 weeks. Plan generation good, tool-call reliability poor (12% format error), long-context drift past ~14k tokens, cascade-failure handling weak. Viable as reasoning layer but not execution layer.
User currently runs Qwen3.6-35B-A3B-4bit on M3 Max 128GB for production sub-agent delegations. Also mentions GLM-5.1 for orchestration. Considering building a 5090 rig.
96 records · 26 GPUs · 9 model families · 5 engines