RTX 3060 12GB

throughput:

43.3 t/s gen · 456.1 t/s pp

quant:

Q4_K_S (gguf)

Dual RTX 3060 setup with tensor parallel. MTP enabled. Context 64k. Prefill 456 t/s, generation 43.26 t/s at 12k context. Without MTP, context 96k, generation 31 t/s. User praises value and stability of CUDA.

throughput:

70.0 t/s gen · 780.0 t/s pp

quant:

Q2-XS (gguf)

kv:

Q8

flash attention:

off

rating:

4/5

defnitly want to try an higher qwant. I vé took that one beacause gguf sise 11,9 go, ans barely offload this dense modèle to cpu

throughput:

46.8 t/s gen · 914.0 t/s pp

quant:

IQ4_XS (gguf)

kv:

Q8

flash attention:

on

Best plain llama-bench: pp512 ~914 t/s, tg128 ~46.8 t/s. Practical coding profile: 32k context, ~43.4 t/s generation. MTP gave ~47.7 t/s (2% improvement).

throughput:

60.0 t/s gen

quant:

Q4_K_M (gguf)

~60 tok/s on RTX 3060 12GB. E2B runs effortlessly. Source: estimated from compute-market tiers

throughput:

45.0 t/s gen

quant:

Q4_K_M (gguf)

~45 tok/s on RTX 3060 12GB. E4B fits easily. Source: compute-market.com