44/89 → 69/89 — from attribution to memory gates
terminal-bench-2.1 (89 trials, separate task set from eval100 below — scores are not comparable across the two leaderboards). tb-v1 (parent code-v16) on deepseek-v4.1-flash, 2026-09-29: 44/89, 45 zeros fully attributed — the losses are config-and-protocol: the budget reminder was left disabled, the stream-idle window is 120s, and the gate skips rerun when command_to_verify is empty.
P0 is config-only (+12~18 points): turn the reminder on, raise recovery retries (2→3) and the idle window (120→300s), stretch agent timeouts (×2) with n_concurrent 2, and swap the dead Debian bullseye mirrors (+2 free). Then coding-task detection and an artifact-existence gate — the report's execution order targets 0.49 → 0.65, then 0.73+. Read tb-v1 → · tb-v2 0.596 → · tb-v2-mimo 0.719 → · tb-v3 0.719 → · selfctx 0.629 (failed) → · tb-v4 0.775
| Harness / model | TB-2.1 pass | What moved | Report |
|---|---|---|---|
tb-v1 · deepseek-v4.1-flash parent code-v16 · reminder OFF |
44/89 · 0.494 |
First run — 45 zeros: C15 / A11 / B10 / D9 | Read → |
tb-v2 · deepseek-v4 max_iter 80→120 · reminder ON · retries up |
53/88 · 0.596 |
+10pp (harness × timeout-×4 × dataset drift) · C 15→4, fixed 14 / regressed 5 | Read → |
tb-v2 · mimo-2.6-pro-ultraspeed same harness, model swap |
64/89 · 0.719 |
0 infra-class zeros · 25 zeros = 9×429 + 2 timeout + 2 apt-404 + 12 capability · true ~0.80 | Read → |
tb-v3 · deepseek-v4.1-flash-1017 provider_error 3→6, base_delay 0.5→2 |
64/89 · 0.719 |
Infra 22→15 · 60% of zeros are runtime (8 wall-clock + 7 TTFT-p99) · 2 qemu apt-404冤案 | Read → |
tb-v2-selfctx · deepseek-v4.1-flash CLM Context-as-File L2 · reverted |
56/89 · 0.629 |
−9pp vs tb-v3 control · mechanism healthy (362 syncs) but fidelity/task mismatch · 1 mteb singleton | Read → |
tb-v4 · deepseek-v4.1-flash context budget 60k · summary cap · hard-gate |
69/89 · 0.775 |
Series best · 4107 LLM calls · 27 true snapshots · input-balloon cap 150k · 7 harness/infra zeros | Read → |
eval100 Leaderboard
Data from real runsCross-model series — Ling-3.0-flash (v6–v9, v16 runs 1–3) and deepseek (v4-0731: v9d, v11d, v12d; v4.1-flash: v16), same task set eval100 (100 tasks, v6 99 judged + 3 errored). Click the pass rate / tool calls headers to sort.
| # | Harness | Model | eval100 pass ▼ | Tool calls | llm rounds (median) | Notes |
|---|---|---|---|---|---|---|
| 1 | code-v6 max_iterations 50 baseline |
Ling-3.0-flash | 43/99 · 0.43 |
4184 | — | max_iter 50 · 3 errored separate · difficulty Low 0.59 / Med 0.44 / High 0.11 · Read report → |
| 2 | code-v7 Baseline tool set |
Ling-3.0-flash | 41/100 · 0.41 |
4103 | 51 | [run_command, read_file, apply_patch, edit_file, write_file] · 22 zero-write / 37 edited-but-failed · Read report → |
| 3 | code-v7-2 Tool-describe-only change |
Ling-3.0-flash | 44/100 · 0.44 |
— | — | tool-describe-only · judged no effect |
| 4 | code-v8 + grep_search, glob_files |
Ling-3.0-flash | 41/100 · 0.41 |
3862 (−6%) | 44 (51→44) | Adds 2 search tools · search cost ≈ −10% · Read report → |
| 5 | code-v9 glob_files → find_files rename |
Ling-3.0-flash | 43/100 · 0.43 |
3973 | — | Rename only · baseline for the deepseek A/B · covered in the v9d report → |
| 6 | code-v9 · deepseek Same harness, model swap |
deepseek-v4-0731 | 40/100 · 0.40 |
3725 (−6%) | — | Search adoption lands: grep 96%, find_files 58%, run_command −29% · pass is noise · Read report → |
| 7 | code-v11 · deepseek + verification return_contract |
deepseek-v4-0731 | 48/100 · 0.48 |
4208 (+13%) | — | Prove-the-fix gate · +8pp vs same model, largest single jump · Read report → |
| 8 | code-v12 · deepseek + edit-necessity & rerun |
deepseek-v4-0731 | 50/100 · 0.50 |
6135 (+46%) | — | Edit-necessity gate + rerun · both design targets converted; rerun exit-2 noise drove the cost · Read report → |
| 9 | code-v16 · ling r1 + early budget nudge |
Ling-3.0-flash | 43/97 · 0.44 |
— | — | Pass-group duration −24%; pipe template failed (2% adoption) · Read report → |
| 10 | code-v16 · ling r2 patch validation + tool catalog |
Ling-3.0-flash | 43/100 · 0.43 |
— | — | Flat; audit-clean rejections exposed the hidden write_file bug · Read report → |
| 11 | code-v16 · ling r3 chunked-write fix verified |
Ling-3.0-flash | 45/99 · 0.455 |
— | — | Write-fail 10→0, tool errors −60% · best v16 Ling run · Read report → |
| 12 | code-v16 · deepseek-v4.1 the model swap |
deepseek-v4.1-flash | 66/100 · 0.66 |
— | — | Identical harness+code as Ling run 3: +20.5pp, 22 fixed / 2 broken · largest jump · Read report → |
| 13 | code-v16 · deepseek-v4.1 r2 same model, same genes — drift run |
deepseek-v4.1-flash | 69/100 · 0.69 |
— | — | McNemar χ²=0.44 — +3pp is noise; standing rules adopted · Read report → |
Pass-rate differences are judged by the McNemar test: v6 43/99 vs v7 41/100 is a −2pt flip (12/100 flips, noise); the Ling-3.0-flash 41–44 band is noise (41 vs 41 net 0; vs v7-2 +5/−8; v9 rename 43, noise); the same-harness model swap is 40 vs 43 (+7/−10, noise). The score only moves when verification lands: 48 vs 40, +14/−6, p≈0.07. v12's 48→50 (+11/−9, χ²=0.05) is noise. The v16 model swap is the real outlier: 66 vs 45.5 (+20.5pp, 22 fixed / 2 broken); the 66→69 drift is noise (χ²=0.44) — the standing rule: χ²>3.84 before celebrating. "—" means the stat was not provided for that run; such rows sort last. v6 carries 99/100 judged (3 errored separate); v16 Ling runs carry 97/100 evaluated (3 infra losses) for r1 and 99/100 for r3.
eval100 climbs to 50 → 66–69; terminal-bench climbs 44 → 69
Ling-3.0-flash: five harnesses land inside the same 41–44 band (v6 43/99 → v7 41 → v7-2 44 → v8 41 → v9 43, all judged noise); its v15/v16 runs reach 43–45.5 — the harness-fixed ceiling for this model. The verification gene lifts deepseek-v4-0731 to 48–50 (the +8pp jump). The deepseek-v4.1 swap on the identical code-v16 harness jumps to 66–69 — +20.5pp. Terminal-bench-2.1 (right group, /89 — not comparable with eval100 /100): tb-v1 44 → tb-v2 53 → tb-v3 / mimo 64 → tb-v4 69. Bars are raw pass counts; denominators differ, read rates in the tables.
Controlled Harness Experiments
We modify one or more related harness components and evaluate their impact through controlled experiments: does pass regress, does cost drop, are the new tools actually used?
harnesses/code-v6.yaml
harnesses/code-v7.yaml = code-v6 + write_file
harnesses/code-v8.yaml = code-v7 + search tools
harnesses/code-v9.yaml = code-v8 + rename
harnesses/code-v11.yaml = code-v10 + return-contract prompt
harnesses/code-v12.yaml = code-v11 + edit-necessity & rerun
harnesses/code-v13.yaml = code-v12 + submit_result
skills gene · budget nudge · infra fixes
harnesses/tb-v1.yaml = code-v16 + general-agent prompt
P0 config fixes → memory gates (tb-v2 / tb-v3 / tb-v4)
Experiment log
Each report is a static page under public/sunagent/, structured as: Setup → Headline → Adoption → Friction → Comparison → Conclusion / Next steps. This homepage updates with every report.
code-v6 · max_iterations 50 baseline
Result: 43/99 (mean 0.43, 3 errored separate), difficulty Low 0.59 / Med 0.44 / High 0.11, smoke-10 7/10; 4184 calls, edit:patch ≈ 8:1; 14 near-misses are the shortest path 0.43→0.55.
Read the full report →code-v7 · +write_file baseline
Result: 41/100 (mean 0.41), 4103 tool calls, median 51 llm rounds; 22 zero-write / 37 edited-but-failed; write_file 0/100 adoption — non-regression, zero benefit.
Read the full report →code-v7-2 · Tool-describe-only
Question: does changing tool descriptions alone help?
Result: 41→44 is noise; patch behavior 21→22 calls, 10→10 trials — no effect.
code-v8 · +grep_search/glob_files
Result: pass non-regression (41 vs 41, McNemar 0), total calls −6%, run_command −10%; grep_search half-won, glob_files undiscovered. Keep, don't expand.
Read the full report →code-v9 · glob_files→find_files rename
Result: rename-only gene on Ling: 43/100 (noise). The real test is the same harness on a stronger model — see the v9-deepseek A/B.
Covered as baseline in the v9d report →code-v9 · model-swap A/B
Question: same harness, stronger model — does it move?
Result: search adoption finally lands (grep 53→96%, find_files 1→58%, run_command −29%) but pass 43→40 is noise. Bottleneck: edit-commit.
code-v11 · return-contract verification
Result: 48/100 — +8pp vs same-model control (McNemar p≈0.07), 9/10 old regressions repaired, gate recovery 45%. Price: +13% tool calls, timeouts 9→12. Largest single jump.
Read the full report →code-v12 · edit-necessity + rerun
Result: 50/100 (+2pp, χ²=0.05 — noise) — edit gate delivered: psf-1142 / pytest-10051 converted with real edits; rerun precision broke (133/140 exit-2 parse noise), run_command +82%. Partially meets expectations.
Read the full report →code-v13 · submit_result + verbatim rerun
Result: 40/83 judged (48.2%, subset — job terminated early); rerun exit-2 class eliminated (typed submission); 6/28 matcher misfires and 25/40 passes bypassing the gate are open. Approach pending.
Read the partial report →code-v16 · early budget nudge + edit routing
Result: 43/97 (44.3% — flat, 5 fixed / 7 broken) but pass-group duration −24% and lucky passes 11→0; the pipe template failed at 2% adoption — text-only compliance does not move this model.
Read the full report →run 2 · patch validation + factual tool catalog
Result: 43/100 (flat). Audit-clean rejections (zero over-blocking) exposed a hidden infra bug — write_file embeds base64 as shell argv, breaking ~10 trials/round. Generic validation, precise hints.
Read the full report →run 3 · chunked-write fix verified
Result: 45/99 (+2.5pp, best v16 Ling) — write-fail 10→0, tool errors −60%, zero new agent behavior: the ceiling argument confirmed — infra fixes move the score.
Read the full report →the model swap · 66/100
Result: +20.5pp on the identical harness (22 fixed / 2 broken, sympy 30→100%) — the largest jump in the series. Format compliance is a model property: 83% apply_patch, zero alias errors, pipes for exploration but clean declarations.
Read the full report →run 2 · 69/100 — noise rules adopted
Result: +3pp drift is McNemar noise (χ²=0.44). Standing rules: report the McNemar table, χ²>3.84 before celebrating, never credit mechanism deltas inside noise. 30/31 failures are fix-quality.
Read the full report →new benchmark · 44/89
Result: first run on the new suite — not a zero: 45 zeros fully attributed (15 budget-exhausted, 11 stream-idle, 10 timeouts, 9 gate-blind). P0 is config-only, expected 0.49→0.65.
Read the full report →P0 fixes · 53/88 (0.596)
Result: 0.494→0.596 — reminder + 120 steps + agent×4 worked (C 15→4); fixed 14 (incl. install-windows rename), regressed 5 (early-idle variance); 35 zeros = A15 / D9 / B7 / C4.
Read the full report →same harness · mimo-2.6-pro · 64/89
Result: mean 0.7191 — 25 zeros = 9×429 rate-limit + 2 timeout + 2 verifier apt-404 + 12 capability; ~44% unrelated to model; true level ≈0.80 after infra exclusion.
Read the full report →deepseek-v4.1-flash · 64/89 + 2 time black holes
Result: mean 0.7191 — 60% of zeros are runtime (8 wall-clock CancelledError + 7 stream-idle TTFT-p99); 202 summaries = 4.98h, 99 retries = 3.3h; 2 qemu apt-404 are pure infra.
Read the full report →CLM Context-as-File L2 · 56/89
Result: −9pp vs tb-v3 (0.719→0.629). Mechanism healthy (62/89 trials, 362 syncs, rollback 0) but fidelity/task mismatch; only mteb-retrieve is a proven compression-casualty singleton.
Read the failure report →memory gates · 69/89 (0.775)
Result: 4107 LLM calls, 33 compacted / 27 true snapshots, frozen 0/89; input-balloon fix (150k cap) + raman 400 fix landed after the run; 20 zeros = 7 harness/infra + 1 disputed + 12 capability.
Read the full report →TB-2.1: verify the post-report fixes
Plan: rerun full suite with the 150k summarizer cap (breaks the invalid input-balloon loop), AgentTurn hard-gate integration test coverage, qemu images fixed to archive.debian.org + preinstalled curl/uvx. Acceptance: hold 0.775 with fewer invalid retries.
Pending