code-v13 (partial): tool-call submission — 40/83
The gate went quiet and precise; the budget became the bottleneck.
PARTIAL — job manually terminated after 83 judged trials. 87 trial dirs scraped: 83 with verifier reward (40×1.0, 43×0.0, mean 0.482), 4×CancelledError (killed by termination, no reward), 11×AgentTimeoutError. All comparisons below use the 83 judged trials only and are noisier than full-eval deltas — this run is not on the leaderboard or trend chart.
JUDGED PASS RATE
40 / 83
48.2% · 40×1.0 + 43×0.0 · flat vs v11/v12 within subset noise
RERUN PRECISION
0 exit-2
28 reruns: 23 ok / 5 exit-1 — typed params killed the prose-dirty class (v12: 133 exit-2)
GATE ACTIVITY
51 / 47
failed events / trials (v12: 179/66) · plain-text nudges 1 · fuse aborts 0
COST
median 30.3 min
agent_exec (v12 full-eval: 24.6) · 11 timeouts · 8 trials pinned at the 60-min cap
Verdict (partial): the rerun precision problem is fixed; two new problems take its place. Matcher misfires on shell-prefixed commands (6/28 reruns execute the wrong history command) and budget exhaustion (25/40 passes never touched the gate). Next-step approach: pending — not decided.
01SETUP · EXPERIMENT DESIGN
Submit through a tool, rerun the grounded call
The lineage step: code-v12 → code-v13. Verification switches to task_result mode — the model finishes by calling submit_result — and the rerun replays the matched history command instead of executing the declared string. Prompt rewritten; a finish-violation fuse caps abuse.
Harness chain
code-v12 ─(verification.mode: return_contract → task_result + tools: +submit_result + prompt rewritten + finish-violation fuse + rerun replays the matched call, cwd + 600s)→ code-v13
run_command
read_file
apply_patch
edit_file
write_file
grep_search
find_files
+ submit_result
verification.mode: task_result
finish_violation_limit: 3
rerun_declared_command: true
max_iterations: 50
The submission contract
Finish by calling submit_result with exactly three parameters:
submit_result( solution_description, # root cause and what you changed evidence, # concise output of the suite you already ran command_to_verify # the FULL test-suite command, exact-as-run, per-repo — declare it before running ) # verify incrementally during the task (targeted tests), not only at the end # plain-text replies cannot finish; a nudge redirects once, repeated violations abort the trial (fuse)
Counting rules
Same basis as v11/v12: tool.start authoritative (data.tool); verification.* trace events; reward keyed by task prefix; agent wall time from trial result.json agent_execution.
Trace check
harness_id = code-v13 confirmed in 87/87 agent-runtime.jsonl headers.
02HEADLINE · KEY RESULTS (SUBSET)
40/83 judged — flat within subset noise
Overlaps computed on the 83 judged tasks only. Both design targets still pass under v13 — no regression on the edit-necessity cases. The cost profile is the real change.
Subset comparison (83 judged tasks)
vs code-v11 overlap 42 → 40 · +7/−9 · McNemar b=7/c=9 — noise
vs code-v12 overlap 44 → 40 · +3/−7 · b=3/c=7 — noise
psf-1142 & pytest-10051 still pass
median agent time 30.3 min (v12: 24.6)
LLM 7.9 vs tools 22.9 min (~1:3)
Newly passed — 7 tasks (vs code-v11 overlap)
django-14725seaborn-3187psf-1142
xarray-6599pytest-10051sklearn-25102
sphinx-7985
Newly failed — 9 tasks (vs code-v11 overlap)
django-12209django-13568psf-1766
sklearn-12973sphinx-11510sympy-15599
sympy-18199sympy-18698sympy-19495
vs code-v12 overlap: newly passed 3 (django-14725, sklearn-14087, sphinx-8120), newly failed 7 (django-12858, psf-1766, xarray-3305, pylint-4661/6386, sklearn-12973, sphinx-10449).
03MECHANISM · QUIET GATE, BINDING BUDGET
The loop is gone — replaced by a budget wall
Typed submission plus the verbatim rerun eliminated the noise class in one step. What remains is time: full-suite runs and iteration burn.
verification.failed
51 / 47
events / trials — v12: 179 in 66. The churn loop is gone
Rerun exit codes
{0:23, 1:5}
zero exit-2 — cd-prefixed declarations exact-match; typed params killed the dirty-string class
Nudges / fuse aborts
1 / 0
across all 87 trials — the model submits via the tool immediately
Zero-edit invariant
19 / 87 · 0 pass
zero-edit trials, none passed — "a real fix necessarily writes" still holds
Fewer but longer
Tool volume 48/trial (v12: 61); run_command 28/trial (v12: 41); durations median 0.6s / p90 23s — but 140 runs ≥ 240s. The full-suite mandate made each verify cycle minutes-long: fix → full suite → fix → full suite.
The gate shaped, the budget decided
Edit-gap events 0 (edits happen upfront); zero-edit 0-passed unchanged. But 25/40 passes never touched the gate — the gate decided only ~15/40 outcomes. 11 timeouts + 8 trials at the 60-min cap are the same phenomenon at the extreme.
04PROBLEMS · WHAT THE PARTIAL RUN EXPOSED
Three problems — two structural, one behavioral
(a) Matcher misfires on shell-prefixed commands — 6/28 reruns
Declared and history commands both start with cd/source, so the program token is always cd/source and a 2-token overlap is trivially satisfied — the rerun then executes the WRONG history command:
| Trial | Declared | Actually reran | Exit | Outcome |
|---|---|---|---|---|
| astropy-14369 | pytest …test_format.py… | git log --oneline -3 && python -c print(version) | 1 | false reject |
| seaborn-3187 | pytest tests/_core/test_plot.py | python -c import matplotlib… | 1 | false reject |
| sphinx-8595 | pytest … | heredoc-python edit script | 1 | false reject |
| sklearn-12973 | pytest sklearn/linea… | grep -n … | 0 | false ACCEPT — trial still failed the verifier |
| sklearn-14053 | pytest sklearn/tree | pytest … (env-activate variant) | 0 | accept on wrong command |
| sphinx-7985 | pytest tests/test_bu… | version-check + pip download | 0 | false ACCEPT — trial passed the verifier anyway |
False rejects cost iterations; false accepts weaken the guarantee. The --tb=short adoption does not help here — this is matching logic, not output verbosity.
(b) 25/40 passes never touched the gate
25 passing trials have zero verification.passed events; their terminal event is iter 51: budget exhausted without a valid submit_result call. max_iter p50 = 51; 50 trials ≥ 45 iterations. The model works diligently to budget — fixes land — but long full-suite runs leave no rounds for a closing submit.
(c) Prompt micro-edits split
cd-prefix declarations: worked (high exact-match rate — half the credit for zero exit-2). Concise-output hint: adopted (--tb=short visible) but saved no time — the bottleneck is execution, not verbosity. Repeated full suites dominate (xarray-6721/6744 ~1300s ×2 each, pylint-8898 ×2); single monsters (django-15128 3239s, sklearn-14087 2420s) exceed any sane per-command cap.
05NEXT STEP · APPROACH PENDING
Not decided — three separate questions
- Matcher misfire — candidate fix: strip leading shell-setup segments (cd/source/export/conda … &&) before program extraction so the real program is compared. Held: the owner explicitly deferred the approach decision ("不打算这么修").
- Budget structure — with 25/40 gate-bypassed passes, gate accuracy matters less than reserving closing budget (e.g. forbid run_command in the last N iterations, via prompt or control gene). Also pending, and a lineage change — keep separate from any matcher fix.
- Long single runs (3000s+) — whether to cap is a third, separate question.
- Cherry-pick audit — pass-trial declaration shapes and evidence-count distribution to be tabulated from these traces once the matcher question is settled.
06CONCLUSION
Precision solved. Budget is the new frontier.
v13 的两处改动各得其所：submit_result 让门安静下来（拒绝事件 179→51、纯文本安抚 1 次、熔断 0 次），verbatim rerun 让 exit-2 噪音类彻底消失。但 25/40 的通过 trial 根本没走到门——预算耗尽成为主要矛盾，外加 6/28 的 matcher 误匹配。修复方向尚未拍板：matcher 剥壳、收官预算、长命令封顶，三个问题分开决策。
07REPRODUCTION
Artifacts and code locations
Artifacts
jobs/eval100-code-v13-deepseek-v4-0731/ (87 trial dirs scraped · 83 judged: 40×1.0 / 43×0.0, mean 0.482 · 4×CancelledError · 11×AgentTimeoutError)
Baselines: jobs/eval100-code-v11-deepseek-v4-0731/ · jobs/eval100-code-v12-deepseek-v4-0731/ (overlap analysis)
Record: none yet in evaluation/records/ for v13 (partial run)
Baselines: jobs/eval100-code-v11-deepseek-v4-0731/ · jobs/eval100-code-v12-deepseek-v4-0731/ (overlap analysis)
Record: none yet in evaluation/records/ for v13 (partial run)
# harness chain (one-gene step from code-v12) harnesses/code-v13.yaml = code-v12 + verification.mode: return_contract -> task_result + tools.enabled: +submit_result + prompt.system rewritten (full suite, exact-as-run, concise output) + finish_violation_limit: 3 # fuse + rerun replays the matched history call (cwd + 600s) # tools (8): [run_command, read_file, apply_patch, edit_file, write_file, # grep_search, find_files, submit_result] · max_iterations: 50 # counting rules (same as v11/v12) authoritative = tool.start gate events = verification.* reward = task prefix agent time = trial result.json → agent_execution
This is a PARTIAL analysis of a manually terminated job: all subset comparisons are noisier than full-eval deltas, and this run is not on the leaderboard or trend chart. Full analysis report: github.com/johnsonbuilds/sunagent · evaluation/analysis/eval100-code-v13-deepseek-v4-0731-partial.md. For run commands, defer to the repo README and the actual artifacts under jobs/; this page only visualizes the report — data and conclusions match it.