QuenchBench
A coding benchmark for the models you run yourself. For each kind of work it answers one question: can this model, on this graphics card, do it well enough to rely on?
- Tasks537 in 13 kinds of work and 7 languages, from a one-line bug to a feature across a 30-file project.
- Checked byHidden tests, a real browser for web UI, or the expected answer for questions about code. Never by another model.
- RunsInside Local Quench, on your own computer, with your model and quantization. One section takes from minutes to a few hours.
- ResultsShared only if you choose, after each run, and combined per model, quantization and graphics card.
What a result says
Every section gets one of four verdicts, from the share of tasks solved and its 95% confidence interval, so a lucky streak on five tasks does not read as a strength.
- Reliable
- Even the low end of the interval is 70% or more.
- Use with review
- It solves a fair share, but check its work.
- Not suitable
- Even the high end of the interval is under 50%.
- Too few tasks
- The interval is wider than 45 points: run more tasks or repeats.
Sections and sizes
About 40 tasks per section, so a verdict rests on enough tasks. Start with ultra quick to see where a model stands, then run the sections you care about in full.
| Section | Tasks |
|---|---|
| Larger projects | 45 |
| New features | 42 |
| Writing tests | 42 |
| Terminal and git | 42 |
| Build errors | 42 |
| Data and scripts | 42 |
| Bug fixing | 41 |
| Algorithms | 41 |
| Refactoring | 40 |
| Reading code | 40 |
| Security fixes | 40 |
| Web UI | 40 |
| Vague requests | 40 |
Languages: Python, JavaScript, Go, C, Rust, Bash and HTML, CSS and JavaScript in a browser. Every task has a difficulty from 1 to 5: 12 at level 1, 150 at level 2, 168 at level 3, 140 at level 4, 67 at level 5. The headline score is weighted: every section counts the same whatever its number of tasks, and a task counts as much as its difficulty.
Built so it cannot be gamed
A benchmark is only worth citing if a good score means good work. These rules hold for every task, and the suite checks them automatically.
Every task is proven
The starting code fails the checks and the reference solution passes them. A task that does not meet both is not in the suite.
The checks stay hidden
The tests are copied in only after the agent has finished, so it cannot read or edit them.
No answers in the repository
Task folders contain no AGENTS.md, CLAUDE.md or similar instruction files, so a harness that reads them gets no hints.
Fair prompts
The prompt states everything the checks test: names, outputs, error messages, edge cases. Only the vague-requests section leaves the details out on purpose, and its checks test the conventional behaviour.
Same machine for everyone
Tasks run in a fresh container without network access, with the same toolchains on every computer.
Held-out tasks
30% of the tasks are held out. Harness changes, including the tuner in the app, are judged on them, never on the tasks they were tuned on.
Results from users
After each run the app asks whether to share it, and shows what would be sent: model, quantization, hardware and per-task numbers, never code or chats. Results are combined per model, quantization and graphics card.
| Model | Quant | Graphics card | Solved | Tasks | Computers | tok/s |
|---|---|---|---|---|---|---|
| The first shared results appear here when public builds are out. | ||||||
Verified partners
People and companies with large hardware who signed a partner agreement and run QuenchBench under review. Only partners who chose to be named are listed; a Verified partner mark in the table above means some runs of that row come from one. The pass rate is the combined one for the model, quantization and card.
Next to the public benchmarks
| Benchmark | Tasks | What the tasks are | Languages | Difficulty |
|---|---|---|---|---|
| QuenchBench | 537 | 13 kinds of work: bug fixes, features, refactoring, build errors, terminal, data, security, tests, algorithms, reading code, web UI, vague requests, larger projects | 7 | 1 to 5 per task |
| SWE-bench Verified | 500 | GitHub issues from 12 Python projects (Django alone has 231) | 1 | Under 15 min: 194, 15 min to 1 h: 261, 1 to 4 h: 42, over 4 h: 3 |
| SWE-bench Pro (public) | 642 | Issues from 11 large projects; trivial edits left out, a fix averages 107 lines over 4 files | Go, Python, JS, TS | No levels; a 51-task hard subset |
| SWE-bench Multilingual | 300 | Issues from about 40 projects | 9 | No levels |
| Terminal-Bench 2 | 89 | Terminal work in 16 categories, from software engineering to system administration and security | Mixed | Easy 4, medium 55, hard 30 |
| Aider Polyglot | 225 | The hardest Exercism exercises | 6 | Only the ones most models failed |
| LiveCodeBench | 1055 | Contest problems from LeetCode, AtCoder and Codeforces | Python | Easy, medium, hard |
From a QuenchBench score to a public one
Each public benchmark is compared with the QuenchBench sections that test the same kind of work: SWE-bench with bug fixes, features, refactoring and build errors; Terminal-Bench with terminal, build, data and security. Models that also have public scores (anchors) are run through QuenchBench, and the two scores are fitted with a line. With one anchor the line is a plain ratio, for example a factor of 1.3 when the anchor scores 50% here and 65% there.
| Benchmark | Compared sections | Factor |
|---|---|---|
| SWE-bench Verified | bug fixes, features, refactoring, build | measuring |
| SWE-bench Pro | + terminal | measuring |
| Terminal-Bench 2 | terminal, build, data, security | measuring |
| Aider Polyglot | algorithms, bug fixes, features | measuring |
The factors appear here once enough anchor runs are in. The app shows its current calibration in the leaderboard, with the error band.
Citing QuenchBench
Name the suite version, the sections, the model file and its quantization, and the harness that ran it. A score without the quantization says little about a local model.
QuenchBench (22 October 2026), 537 tasks, 13 sections. Model: <model>, <quantization>, <context>. Harness: <Local Quench version or other agent>. Hardware: <graphics card, VRAM, system RAM>. https://<site>/bench.html