QuenchBench leaderboard
Shared results of local models on QuenchBench: which model, in which quantization, on which graphics card, solves how much. Every row is what users chose to share after their own runs; results are combined per model, quantization, KV cache type, graphics card and task selection, and suspicious runs are held back.
- SolvedThe share of tasks whose hidden checks passed, the median over computers (one computer counts once, however many runs it shared).
- ComparableOnly default-harness runs of the same task selection sit in one row; ablation runs, hand-picked task lists and runs under 13 tasks stay off the board. A run on a cloud API is marked as such.
- Share yoursRun QuenchBench in Local Quench and click Share when the app asks after the run. Numbers only, never code or chats; see the data page.
Results from users
After each run the app asks whether to share it, and shows what would be sent: model, quantization, KV cache type, engine, sampling settings, hardware and per-task numbers, never code or chats. Results are combined per model, quantization, KV cache type and graphics card.
| Model | Quant | Graphics card | Solved | Tasks | Computers | tok/s |
|---|---|---|---|---|---|---|
| No shared results yet. Run QuenchBench in the app and share yours: it shows up here. | ||||||
Verified partners
People and companies with large hardware who signed a partner agreement and run QuenchBench under review. Only partners who chose to be named are listed; a Verified partner mark in the table above means some runs of that row come from one. The pass rate is the combined one for the model, quantization and card.
What quantization costs
The same model in different quantizations, on the same tasks and with the same KV cache type, against its largest quantization. Not perplexity: whether the agent still finishes the work. All graphics cards count together, since the card changes the speed, not the answers.
| Model | Quant | Bits | Solved | Change | Tasks | Computers |
|---|
Change: percentage points against the first quantization of the model. ± is the 95% margin of a single rate; a change smaller than the margins is noise. A quantization appears with 20 finished tasks or more.