QuantThink Leaderboard
Real GPU results measuring how weight- and KV-cache quantization degrade small reasoning (long chain-of-thought) models on a 4GB consumer GPU — and, given a fixed VRAM budget, which config maximizes accuracy.
Memory-Budget Frontier: under the accuracy objective, the
optimal model family crosses over as the VRAM budget grows —
DeepSeek-R1-Distill-Qwen-1.5B wins below ~2.3GB, Qwen3-1.7B wins above it.
Under the Cost-to-Solve objective, the smaller model wins the entire
2–4GB grid instead, because the larger model's accuracy gain
doesn't pay for its higher token cost. Separately, Q4 KV-cache
quantization caused total generation collapse (not smooth degradation)
for the R1-distill model at Q4_K_M weights — confirmed by inspecting raw
output. First-pass, disclosed-small-N result (N=4–12 samples per
cell); see
docs/RUN_REAL.md
for the full write-up.
GSM8K, real GPU results
| Model | Quant | Acc | TL | CTS | VRAM |
|---|---|---|---|---|---|
| DeepSeek-R1-Distill-Qwen-1.5B | fp16 | 0.667 | 469.8 | 1541.5 | 3.50 GB |
| DeepSeek-R1-Distill-Qwen-1.5B | Q8_0 | 0.750 | 385.5 | 1246.4 | 2.16 GB |
| DeepSeek-R1-Distill-Qwen-1.5B | Q5_K_M | 0.667 | 244.3 | 985.5 | 1.67 GB |
| DeepSeek-R1-Distill-Qwen-1.5B | Q4_K_M | 0.583 | 420.8 | 1532.1 | 1.55 GB |
| DeepSeek-R1-Distill-Qwen-1.5B | GPTQ 4-bit (self-calibrated) | 0.750 | 317.4 | 1025.1 | 1.63 GB* |
| Qwen3-1.7B (thinking) | Q8_0 | 1.000 | 1191.8 | 2212.9 | 2.99 GB |
| Qwen3-1.7B (thinking) | Q4_K_M | 0.875 | 1910.1 | 3717.6 | 2.32 GB |
| Qwen3-0.6B (thinking) | fp16 | 0.500 | 1300.0 | 4934.3 | 2.40 GB |
| Qwen3-0.6B (thinking) | Q4_K_M | 0.375 | 1385.5 | 7455.3 | 1.65 GB |
* GPTQ VRAM measured via torch.cuda.max_memory_allocated, not
directly comparable to the nvidia-smi-based GGUF
measurements. Full table (including MATH-500, KV-cache axis,
thinking-cap grid) in
docs/RUN_REAL.md.
Acc = pass@1. TL = mean thinking-length (tokens). CTS = cost-to-solve
(expected tokens per correct answer).