QuenchBench leaderboard
The same model and the same QuenchBench tasks under different agent harnesses: how much the harness around a local model changes what it gets done. Newest first. Each comparison says how it was run and what went wrong in the runs; a preview is replaced once both sides are measured again under exactly the same conditions.
November 2026
Local Quench and pi on QuenchBench Short Preview
The same model, the same 26 QuenchBench Short tasks, three tries each, under two agent harnesses: Local Quench and pi with the community packages pi-tool-repair and pi-anti-doom-loop. Spark X2.6 4B, Q8_0, in LM Studio, four tasks at once, 30 minutes of model time per task, every task in its own container.
This is a preview, not a final result. Both runs had small mishaps that came from the test machine and its setup, not from how either harness is built. They are marked in the table below. Once both harnesses have been measured again under exactly the same conditions, this section is updated right away. Results can also differ with other versions of either harness and with other models.
- Local Quench: 3 tries were graded as failed because the grader could not create Go's build cache (a QuenchBench bug, fixed since), and two tries were cut by a reply-length limit that has been removed since.
- pi: it ran its commands on the host instead of in the task container, so in the Go and Rust tasks it first met a missing or different toolchain; 11 tries ended when its anti-loop package stopped the request, and they count as failures here. Both are fixed in the benchmark now: an outside agent's commands run in the task container, and a stopped try is graded as the agent left it.
- Local Quench
- pi
| Local Quench | pi | |
|---|---|---|
| Solved tries | 36 of 78 (46.2%) | 26 of 78 (33.3%) |
| Most likely range (95%) | 36–57% | 24–44% |
| Per try round (1st, 2nd, 3rd) | 12 / 12 / 12 | 7 / 9 / 10 |
| Median time per task | 15.1 min | 15.4 min |
| Model time per solved task | 35.5 min | 50.7 min |
| Stopped by the time limit | 13 | 25 |
| Output tokens per task (mean) | 15.4k | 16.1k |
| Input tokens per task (mean) | 427k | 339k |
| Steps per task (mean) | 24.2 | 21.5 |
| Largest context | 51,549 | 45,841 |
By task: both solved 13 tasks at least once, only Local Quench 4, only pi 2, neither 7. With 78 tries each, a gap of this size could still partly be chance (a paired test gives p ≈ 0.1); the ranges above overlap. Read it as a first measurement.
All 26 tasks, try by try
✓ solved, ✗ not solved, ✗ its changes passed the tests only after the time limit, ! a try with a mishap from the setup (hover for what happened). Rows in grey: neither harness solved the task.
| Task | Language | Local Quench | pi | Notes | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| algorithms grid-dijkstra | Go | ✗ | ✗ | ✗ | 0/3 | ✗ | ✓ | ! | 1/3 | pi: ran its commands outside the task container; the Go or Rust toolchain it found differed or was missing; pi: 1 try stopped by pi-anti-doom-loop |
| algorithms union-find | C | ✓ | ✓ | ✓ | 3/3 | ✗ | ✗ | ! | 0/3 | pi: 1 try stopped by pi-anti-doom-loop |
| bugfix bash-log-rotate | Bash | ✗ | ✗ | ✗ | 0/3 | ✗ | ✗ | ✗ | 0/3 | Neither solved it |
| bugfix binary-search | JavaScript | ✓ | ✓ | ✓ | 3/3 | ✓ | ✓ | ✗ | 2/3 | |
| build bash-set-e-build | Bash | ✓ | ✓ | ✓ | 3/3 | ✗ | ✓ | ✗ | 1/3 | |
| build py-import-cycle | Python | ✓ | ✓ | ✓ | 3/3 | ✓ | ✓ | ✓ | 3/3 | |
| data unit-convert | Rust | ✗ | ✗ | ✗ | 0/3 | ✗ | ! | ✗ | 0/3 | Neither solved it; pi: ran its commands outside the task container; the Go or Rust toolchain it found differed or was missing; pi: 1 try stopped by pi-anti-doom-loop |
| data w2-json-patch | JavaScript | ✗ | ✗ | ✗ | 0/3 | ✗ | ✗ | ✗ | 0/3 | Neither solved it |
| feature c-ringbuf | C | ✓ | ✗ | ✗ | 1/3 | ✗ | ✗ | ✓ | 1/3 | |
| feature go-cursor-pagination | Go | ! | ✓ | ✓ | 2/3 | ✗ | ✗ | ✗ | 0/3 | LQ: a try graded wrong (QuenchBench's Go build cache bug, fixed since); pi: ran its commands outside the task container; the Go or Rust toolchain it found differed or was missing |
| project w2-agent-capacity | Go | ✓ | ! | ✗ | 1/3 | ! | ! | ✓ | 1/3 | LQ: a try graded wrong (QuenchBench's Go build cache bug, fixed since); pi: ran its commands outside the task container; the Go or Rust toolchain it found differed or was missing; pi: 2 tries stopped by pi-anti-doom-loop |
| project w2-csv-quoting | Python | ✗ | ✓ | ✓ | 2/3 | ✓ | ✓ | ✓ | 3/3 | |
| reading batch-complexity | Rust | ✓ | ✓ | ✗ | 2/3 | ✓ | ✗ | ✓ | 2/3 | pi: ran its commands outside the task container; the Go or Rust toolchain it found differed or was missing |
| reading w2-pricing-pipeline | JavaScript | ✗ | ✗ | ✓ | 1/3 | ✓ | ✓ | ✗ | 2/3 | |
| refactor dedupe-email-check | Python | ✓ | ✓ | ✓ | 3/3 | ✓ | ✓ | ✓ | 3/3 | |
| refactor link-store-interface | Rust | ✗ | ✓ | ✗ | 1/3 | ! | ! | ✓ | 1/3 | pi: ran its commands outside the task container; the Go or Rust toolchain it found differed or was missing; pi: 2 tries stopped by pi-anti-doom-loop |
| security c-image-overflow | C | ✓ | ✗ | ✓ | 2/3 | ✗ | ✓ | ! | 1/3 | pi: 1 try stopped by pi-anti-doom-loop |
| security c-strlcpy | C | ✓ | ✓ | ✓ | 3/3 | ✗ | ✗ | ✓ | 1/3 | |
| terminal find-duplicates-nul | Bash | ✗ | ✗ | ✗ | 0/3 | ✗ | ✗ | ✗ | 0/3 | Neither solved it |
| terminal git-recover-deleted | Bash | ✓ | ✓ | ✓ | 3/3 | ✓ | ✓ | ✓ | 3/3 | |
| testing retry-backoff-tests | Go | ✓ | ✓ | ! | 2/3 | ✗ | ✗ | ✗ | 0/3 | LQ: a try graded wrong (QuenchBench's Go build cache bug, fixed since); pi: ran its commands outside the task container; the Go or Rust toolchain it found differed or was missing |
| testing w2-resolver-tests | JavaScript | ✗ | ✗ | ✓ | 1/3 | ✗ | ! | ✗ | 0/3 | pi: 1 try stopped by pi-anti-doom-loop |
| ui ui-cart-totals | Web | ✗ | ✗ | ✗ | 0/3 | ✗ | ✗ | ✗ | 0/3 | Neither solved it |
| ui ui-kanban-undo | Web | ✗ | ✗ | ✗ | 0/3 | ! | ✗ | ✗ | 0/3 | Neither solved it; pi: 1 try stopped by pi-anti-doom-loop |
| vague config-file-vague | Python | ✗ | ✗ | ✗ | 0/3 | ✗ | ✗ | ✓ | 1/3 | |
| vague w2-rate-limit-vague | Go | ✗ | ✗ | ✗ | 0/3 | ✗ | ! | ✗ | 0/3 | Neither solved it; pi: ran its commands outside the task container; the Go or Rust toolchain it found differed or was missing; pi: 1 try stopped by pi-anti-doom-loop |
Runs: Local Quench 20261102-104128-ed23 (app 0.1.2 development build, Guided harness), pi 20261103-110733-b3fc (pi 1.4 with pi-tool-repair 0.3.9 and pi-anti-doom-loop 0.0.14). Graded by the same QuenchBench checks in the same image. Task conversations are not published. Every try, with what happened when it failed: on GitHub.