Benchmark home/eval100-code-v12-deepseek-v4-0731

EXPERIMENT · EVAL100 · CODE-V12 · RERUN GENE

code-v12: edit-necessity + declared-command rerun — 50/100

The gate learned to execute. The targets flipped; the rerun grabbed prose.

Harness code-v12 (= code-v11 + rerun + edit-necessity prompt) Model deepseek-v4-0731 Control code-v11-deepseek (48) Completion 100/100 jobs Report jobs/eval100-code-v12-deepseek-v4-0731/

PASS RATE

50 / 100

mean 0.50 · vs same-model control +2pp — McNemar b=11/c=9, χ²=0.05 · judged noise

DESIGN TARGETS

2 / 2 converted

psf-1142 & pytest-10051: zero edits, verifier failed → real edits, reward 1.0

RERUN PRECISION

140 / 171 failed

133 exit-2 shell parse errors — extraction grabbed trailing prose · only 5 real signals

COST

run_command +82%

2247 → 4083 · total calls 4208 → 6135 (+46%) · fail-group run mean 22.3 → 53.8

Verdict: partially meets expectations — the edit gate delivered, the rerun's extraction broke the deal. Free-text extraction grabs trailing prose and executes it verbatim; 96% of rerun rejections are false positives. Fix the extraction first, keep the gate.

01SETUP · EXPERIMENT DESIGN

One gene at a time — execute what was declared

The lineage stays clean — code-v11 → code-v12, one gene: after the return contract passes, the declared verify command is re-executed once. Grounding now also requires a real edit; tools are unchanged.

Harness chain

code-v11 ─(verification.rerun_declared_command: false → true + prompt.system advertises edit-necessity/rerun)→ code-v12

run_command read_file apply_patch edit_file write_file grep_search find_files verification.rerun_declared_command: true prompt.system: edit-necessity + rerun warning max_iterations: 50

The contract, now enforced by execution

Same three sections — plus two new teeth, advertised up front in prompt.system:

solution_description  # root cause and what you changed
evidence              # quote the actual shell output you observed — do not invent results
command_to_verify     # one shell command you already ran that exits 0 on success

# "You must make at least one source edit (edit_file/apply_patch/write_file)
#  before finishing, and your declared command is re-executed once to
#  confirm it exits 0."
# rerun: exit 0 accepts · non-zero rejects as command_to_verify with rerun output

Counting rules

Same basis as the v11 report: tool.start authoritative (data.tool); verification.failed / passed / rerun trace events; reward keyed by task prefix (rsplit("__",1)[0]). The gate never changes the score directly — it steers behavior; the verifier's verdict on repo state decides.

Trace check

harness_id = code-v12 confirmed in 100/100 agent-runtime.jsonl headers — no contamination.

02HEADLINE · KEY RESULTS

+2pp — and both design targets flipped, but χ²=0.05 says noise

50 vs 48 with +11/−9 flips: McNemar b=11/c=9, χ²=0.05 — far from the 3.84 needed for p<0.05 at n=100. The good news is the two flipped targets are exactly the "gated but wrong" cases v11 left open (§4); the bad news is the price (§3).

Pass comparison (same dataset eval100 · same model deepseek-v4-0731)

code-v11 · deepseek 48 code-v12 · deepseek 50 +11 / −9 · McNemar χ²=0.05 — noise design targets 2/2 converted timeouts 12 → 10 with churn run_command +82% · total +46%

Newly passed — 11 tasks (vs code-v11 deepseek)

django-12858seaborn-3187psf-1142 xarray-3305xarray-6599pylint-4661 pylint-6386pytest-10051sklearn-25102 sphinx-10449sphinx-7985

Newly failed — 9 tasks (vs code-v11 deepseek)

django-12209django-13568sklearn-14087 sphinx-11510sphinx-8120sympy-15599 sympy-18199sympy-18698sympy-19495

Timeout churn — 12 → 10, net −2, different set

5 newly timed out (django-13568, matplotlib-26208, pylint-4970, sympy-16597/19495 — of which 13568/19495 were v11 passes) vs 7 escaped (11848, 13449, 15128, 6028, 4075, 7080, 7277). 13568/19495 "burned the whole budget without ever finishing" — zero verification events in v12, plausibly rerun-driven iteration burn. The other regressions mostly died on the summary turn (shape/edit-gap) or show no gate events at all.

03MECHANISM · RERUN PRECISION

Rerun works — and 96% of its rejections are false positives

The rerun fired 171 times after the contract passed. Its root cause is now located: free-text command_to_verify extraction grabs trailing prose — an 800-char window including markdown backticks and explanatory sentences ("this command was just re-run…") — and executes the blob verbatim.

verification.rerun events

171

only 31 successes · 140 failures

exit-2 parse errors

133

unexpected EOF while looking for matching '`' — prose-wrapped commands

Real signals

clean commands, exit 1 — the "unworthy verify command" class rerun was built to catch

Rejected → later passed

1 → 36

v11 → v12 · gate churn multiplied, productive repair a minority

The vicious cycle

rerun rejection → the model restates with more explanation ("exit 2 is a quoting artifact, the command itself succeeds") → dirtier extraction → another rejection. Clearly visible over 3–4 rounds in the astropy-13977 and psf-1142 traces.

Fail-reason mix & gate activity

v11 {shape:36, unrun-cmd:5} → v12 {rerun:140, shape:78, unrun-cmd:11, edit-gap:7}. Failed gate events 17 → 179; trials with rejections 11 → 66 — almost entirely rerun-noise churn.

The bill

run_command 2247 → 4083 (+82%); total tool calls 4208 → 6135 (+46%); trials with max_iter≥45 40 → 62; fail-group run mean 22.3 → 53.8 — and the pass group rose too (22.6 → 27.9). Even passing trials got more expensive.

04TARGET CASES · EDIT-NECESSITY DELIVERED

A real fix necessarily writes

v11's two "gate passed, verifier failed" walk-throughs were the design targets. Both flipped — by being forced into real edits.

psf-1142 — from walk-through to pass

v11: 5 runs / 0 edits / iter 9, gate passed, verifier failed → v12: 33 runs / 3 edits / iter 50, reward 1.0. Survived 3 exit-2 rerun rejections plus 1 unrun-cmd along the way, then a genuine rewrite of prepare_content_length() plus a runnable verify script.

pytest-10051 — same pattern, 4th attempt

v11: 1 run / 0 edits / iter 8 → v12: 19 runs / 4 edits / iter 28, reward 1.0. Same 3×exit-2 pattern, then a cleanly declared command with a rerun exit-0 accept on the 4th attempt.

The global picture

7 explicit edit-gap rejections (django-13398, matplotlib-23476/25775, seaborn-3069, sphinx-11510/8120, sympy-14248) all ended at reward 0 — forcing edits doesn't convert by itself; passing still requires a correct fix. Zero-edit trials 29 → 23, and zero-edit never passed the verifier in either job — confirming the premise.

05COMPARISON · SIDE BY SIDE

The score is noise — the behavior change is not

48 → 50 moves nothing statistically. What moved: zero-edit trials 29 → 23, both targets forced into real fixes — and the cost curve bends upward.

Slate bars = Ling-3.0-flash; gradient bars = deepseek-v4-0731. 48 → 50 is judged noise (McNemar χ²=0.05); the v12 deliverable is the edit gate — both design targets converted — not the score. The plateau band marks the Ling 41–44 noise band.

06CONCLUSION

Fix the extraction. Keep the gate.

edit gate 证明了"真修复必然写入"：两个目标案例都被逼出真实改动并转为通过。rerun 的自由文本抽取把尾随说明文字原样执行，exit-2 噪音淹没了真信号，还触发"越解释越脏"的恶性循环。先修抽取，再谈保留——门本身便宜，且已被验证。

07NEXT STEPS · ONE GENE AT A TIME

v13: fix command extraction only

Fix extraction — two options: (a) extract only from fenced code blocks / JSON values / the first line, stripping backticks and trailing prose; (b) cleaner — the rerun never executes the declared string but the already-grounded history command, using the declared string for matching only. Either zeroes the exit-2 noise class and leaves exit-1 as the true signal.
Keep the edit gate: cheap and proven — both named targets converted, 7 rejections with zero false positives.
Re-run eval100 after the fix: acceptance metrics — exit-1 share of rerun failures, number of rerun-triggered trials, nudge-recovery rate back near v11 levels, fail-group run mean back down.
Housekeeping: the report noted evaluation/records/ had no v12 record yet and suggested one — it has since been added (harness_id code-v12, 2026-09-15).

08REPRODUCTION

Artifacts and code locations

Artifacts

jobs/eval100-code-v12-deepseek-v4-0731/ (100/100 completed, mean 0.50)
Baseline: jobs/eval100-code-v11-deepseek-v4-0731/ (mean 0.48 — same model, direct control)
Record: evaluation/records/eval100-code-v12-deepseek-v4-0731__code-v12__dataset-eval100.json

# harness chain (one-gene step from code-v11)
harnesses/code-v12.yaml = code-v11 + verification.rerun_declared_command: false -> true
                         + prompt.system advertises edit-necessity / rerun warning
# verification: return_contract · require [solution_description, evidence, command_to_verify]
# tools unchanged (7) · max_iterations: 50

# counting rules (same as v11)
authoritative = tool.start                        # call counts
gate events   = verification.failed / passed / rerun   # trace events
reward        = keyed by task prefix              # rsplit("__",1)[0]

Counting conventions: the record snapshot logs 49 passed + 10 errored (timeouts); the report counts rewards keyed by task prefix (50) — a timeout trial whose repo state passes the verifier still counts, the same convention as v11's django-12209. Full analysis report: github.com/johnsonbuilds/sunagent · evaluation/analysis/eval100-code-v12-deepseek-v4-0731.md. For run commands, defer to the repo README and the actual artifacts under jobs/; this page only visualizes the report — data and conclusions match it.