Benchmark home/eval100-code-v11-deepseek-v4-0731

EXPERIMENT · EVAL100 · CODE-V11 · VERIFICATION GENE

code-v11: return-contract verification — 48/100

The harness finally moves the score: make the agent prove the fix, not the story.

Harness code-v11 (= code-v9 + verification + prompt) Model deepseek-v4-0731 Controls code-v9-deepseek (40) · code-v9 Ling (43) Completion 100/100 jobs Report jobs/eval100-code-v11-deepseek-v4-0731/

PASS RATE

48 / 100

mean 0.48 · 48×1.0 + 52×0.0 · vs same-model control +8pp (p≈0.07)

LARGEST SINGLE JUMP

40 → 48

+14 / −6 · McNemar b=14/c=6, χ²=3.2 · 9/10 old regressions repaired

GATE EVENTS

17 in 11 trials

5 recovered to pass (45%) · 89/100 passed the gate first attempt

TOTAL TOOL CALLS

3725 → 4208

+13% — the price · timeouts 9 → 12

Verdict: +8pp — the largest single jump, from a prompt-only verification gene. The gate works via upfront planning, not post-hoc rejection. Next loophole: the gate checks "ran + cited", not "changed".

01SETUP · EXPERIMENT DESIGN

One gene at a time — evaluated as a pair

The lineage is clean — code-v9 → code-v10 → code-v11, one gene per step. This run evaluates the pair of verification genes together against the code-v9-deepseek control; tools are unchanged.

Harness chain

code-v9 ─(verification.mode: off → return_contract)→ code-v10 ─(prompt.system: empty → return-contract instruction)→ code-v11

run_command read_file apply_patch edit_file write_file grep_search find_files verification.enabled: true (return_contract) prompt.system: return-contract instruction

The return contract (required in the final answer)

Three sections, advertised up front in prompt.system:

solution_description  # root cause and what you changed
evidence              # quote the actual shell output you observed — test names, counts, key lines; do not invent results
command_to_verify     # one shell command you already ran that exits 0 on success

# an answer that cites output you never produced, or a command you never ran, is rejected
# rerun_declared_command: false — no gold-test execution in v11, by design

Counting rules

Counts use tool.start as the authority (data.tool); gate events via verification.failed/passed trace events; reward keyed by task prefix (rsplit("__",1)[0]).

Trace check

harness_id = code-v11 confirmed in 100/100 agent-runtime.jsonl headers — no contamination.

02HEADLINE · KEY RESULTS

+8pp vs the same model — largest single jump so far

48 vs 40 with +14/−6 flips: McNemar b=14/c=6, χ²=3.2, p≈0.07 — suggestive, not yet 0.05-significant at n=100. 9 of the 10 v9-deepseek regressions are repaired (all except pylint-6386); the price is 6 fresh regressions, 2 of them timeouts. Vs the old-model baseline (code-v9, Ling): +9/−4, net +5.

Pass comparison (same dataset eval100 · same model deepseek-v4-0731)

code-v9 · deepseek 40 code-v11 · deepseek 48 +14 / −6 · McNemar χ²=3.2, p≈0.07 9/10 old regressions repaired 6 fresh regressions · 2 timeouts vs code-v9 Ling (43): +9 / −4

Newly passed — 14 tasks (vs code-v9 deepseek)

django-12209django-13568django-14007 matplotlib-25332xarray-4094sklearn-12973 sklearn-13124sphinx-11510sphinx-9591 sympy-12419sympy-15599sympy-18199 sympy-18698sympy-19495

Newly failed — 6 tasks (vs code-v9 deepseek)

django-11848django-15128psf-1142 xarray-3305pytest-10051sphinx-10449

One ID, both lists · django-12209

The agent timed out at iter 30 — but its edits persisted and the verifier passed. Reward is keyed by task prefix, so the task counts as pass while the trial also sits in the timeout list (9 → 12 timeouts overall: 14725, 13449, 12858, 4075, 15128, 8898, 6028, 12209, 13212, 11848, 7277, 7080).

03MECHANISM · WHY IT WORKS

The upfront ad worked; the nudge rarely fired

Advertising the 3-section return contract in the system prompt from round 1 made the model plan evidence collection during the run. The feared "always-collide-then-cosmetically-patch" loop did not materialize.

First-attempt gate pass

89 / 100

trials whose first finish attempt already satisfied the contract

verification.failed

17 events / 11 trials

the gate rejected the final answer this often

Gate recovery

5 / 11 = 45%

astropy-14309, matplotlib-25332, sklearn-11578, sphinx-8120, sympy-19495 — rejected, then re-ran and passed

Nudge-recovery example

matplotlib-25332

failed once on grounding, re-ran the command, passed on the next finish

Repair anatomy — "search → edit → run" completion, via the prompt not the nudge

sympy-18698 (0 edit → edit + patch, gate passed first try at iter 51) · django-12209 (run 17 → 30 + 1 edit) · sphinx-9591 (run 1 → 14 + 1 edit). Tool volume: run_command 1994 → 2247 (+13%; pass-group mean 18.7 → 22.6, fail-group 20.8 → 22.3), grep_search 577 → 695, read_file 897 → 989; find_files flat (110 → 102), edit_file flat (107 → 109), apply_patch 30 → 40.

04FRICTION · FAILURE MODES

Two new problems the gate created

The jump is real, but the mechanism has loopholes — and the budget is starting to bind.

(a) The gate checks "ran + cited", not "changed"

2 regressions walked through open: psf-1142 (29 → 9 iters, 5 runs / 0 edits, gate accepted, tests fail) · pytest-10051 (28 → 8 iters, 1 run / 0 edits, accepted at iter 8). Both quote real output and name a command they ran — shape + grounding pass with zero source edits. Zero-edit & fail barely moved (31 → 29). A real fix necessarily writes.

(b) Volume costs budget

+13% tool calls → +3 timeouts (new: xarray-4075, psf-6028, django-13212; plus django-11848, the v9d robustness win, lost to timeout). 50 iterations now bind — either reserve verify budget or accept the exchange rate.

05COMPARISON · SIDE BY SIDE

The plateau breaks — with the harness, not the model alone

Ling-3.0-flash sits at 41–44 across four harnesses; deepseek on the same harness (code-v9) lands at 40 — still inside the band. The verification gene is what lifts the score out of it: 48.

Slate bars = Ling-3.0-flash; gradient bars = deepseek-v4-0731. Same-model comparison: 40 → 48 (+8pp, McNemar p≈0.07). The 48 bar clears the 41–44 Ling plateau; the same model without verification (40) does not.

06CONCLUSION

Make the agent prove the fix.

return-contract 验证用一次纯 prompt 基因把 40 拉到 48：提前告知契约，模型从第 1 轮就开始规划取证，而不是事后补答案。保留这道门——然后堵上"跑过 + 引用过"但不"改过"的漏洞，同时盯住预算。

07NEXT STEPS · ONE GENE AT A TIME

v12: grounding += edit-necessity

Require a real change: reject edit-less finishes — if history contains no edit_file / apply_patch / write_file, reject as missing solution_description. Expected: convert the psf-1142 / pytest-10051 class from "gated but wrong" into "forced to attempt a patch".
Watch the budget: extra forced edits cost rounds — track the timeout count and the 5/11 nudge-recovery rate.
Deferred: rerun_declared_command = true (re-execute the declared command once) stays off until the edit gate proves out — no gold-test execution in v11, by design.

08REPRODUCTION

Artifacts and code locations

Artifacts

jobs/eval100-code-v11-deepseek-v4-0731/ (100/100 completed, mean 0.48)
Baselines: jobs/eval100-code-v9-deepseek-v4-0731/ (40×1.0, mean 0.40 — same model, direct control) · jobs/eval100-code-v9/ (43×1.0, mean 0.43 — old model)
Record: evaluation/records/eval100-code-v11-deepseek-v4-0731__code-v11__dataset-eval100.json

# harness chain (two one-gene steps, evaluated together)
harnesses/code-v10.yaml = code-v9 + verification.mode: off -> return_contract
harnesses/code-v11.yaml = code-v10 + prompt.system: empty -> return-contract instruction
# verification.require: [solution_description, evidence, command_to_verify]
# rerun_declared_command: false · max_iterations: 50 · tools unchanged

# counting rules
authoritative = tool.start                        # call counts
gate events   = verification.failed / passed      # trace events
reward        = keyed by task prefix              # rsplit("__",1)[0]

Full analysis report: github.com/johnsonbuilds/sunagent · evaluation/analysis/eval100-code-v11-deepseek-v4-0731.md. For run commands, defer to the repo README and the actual artifacts under jobs/; this page only visualizes the report — data and conclusions match it.