code-v9 on deepseek-v4-0731: search adoption finally lands
Same harness, different model. — the model-only A/B the series needed; the variable is the model, nothing else.
PASS RATE
40 / 100
mean 0.40 · 40×1.0 + 60×0.0 · net vs code-v9 (Ling) −3 (noise)
TOTAL TOOL CALLS
3973 → 3725
−6% — composition flips, volume drops
grep_search COVERAGE
53% → 96%
174/53 → 577/96 trials (+3.3×)
find_files COVERAGE
1% → 58%
glob_files rename finally adopted: 2/1 → 110/58
Verdict: the rename is validated — search tools finally adopted — but pass is flat: localization is no longer the bottleneck, edit-commit + verification is.
01SETUP · EXPERIMENT DESIGN
One harness, one swapped model
The harness is byte-identical to code-v9 (one rename gene vs code-v8). The model is the only variable — this run doubles as the discoverability validation for the rename and as the control for what a stronger model does on the same harness.
Harness gene (unchanged from code-v9)
harnesses/code-v9.yaml = code-v8 + tools.enabled: glob_files → find_files
run_command
read_file
apply_patch
edit_file
write_file
grep_search
find_files ← glob_files (renamed)
Counting rules
Counts use tool.start as the authority (data.tool); llm.start/end only for iteration counts. result.json has n_input/output_tokens: null — no token/$ comparison possible.
Model attribution caveat
config.json/lock.json do not record the model name — the switch to deepseek-v4-0731 is inferred from the job name plus the behavior shift below, not from an artifact field.
02HEADLINE · KEY RESULTS
No score gain — the harness finally got used
40 vs 43 is −3pp at n=100 (se≈4.9%): within noise. Same picture vs code-v8: +8/−9. Timeouts are flat (9×AgentTimeoutError vs 10×AgentTimeout + 1×VerifierTimeoutError). What did move is tool behavior — see §03.
Pass comparison (same dataset eval100 · same harness line)
code-v9 (Ling-3.0-flash) 43
code-v9 (deepseek-v4-0731) 40
+7 / −10, se≈4.9% → noise
vs code-v8 (41): +8 / −9, noise
reference band: v7 41 · v7-2 44 · v8 41
Newly passed — 7 tasks (vs code-v9 Ling)
astropy-13977django-11848django-15128
django-15161xarray-6744pytest-5840
sphinx-10449
Newly failed — 10 tasks (vs code-v9 Ling)
django-12209django-14007matplotlib-25332
xarray-4094pylint-6386sklearn-13124
sphinx-9591sympy-12419sympy-18698
sympy-19495
Only clear robustness win · django-11848
Ling baseline: 1 tool / 1 iter (instant abort). Deepseek: 21 tools / 20 iters (run9 + grep5 + find2 + read4 + edit1) — and passed. This task later regresses in code-v11 via timeout (see the v11 report).
03ADOPTION · TOOL UPTAKE
Tool use improved — in exactly the harness-intended direction
Total tool.start drops 6%, but the composition flips: native search replaces shell exploration. Deepseek is the first model that actually complies with the code-v8/v9 intent.
| Tool | code-v9 · Ling (calls / trials) | code-v9 · deepseek (calls / trials) | Read |
|---|---|---|---|
| run_command | 2812 / ~100 | 1994 / ~100 | −29% |
| grep_search | 174 / 53 | 577 / 96 | +3.3× · 53%→96% |
| find_files (renamed glob_files) | 2 / 1 | 110 / 58 | 1%→58% |
| read_file | 821 | 897 | +9% |
| edit_file / apply_patch / write_file | 146 / 16 / 0 | 107 / 30 / 9 | edits −, patch/write + |
| tool.error | 19 | 8 | −11 |
Correctness of calls
19 → 8 errors
Hallucinated bare grep (2–3 trials in v8/v9) disappears: 0 in deepseek
Pace · tools/trial
39.7 → 37.2
median 46 → 37
Pace · iters/trial
38.3 → 34.0
less wall-hitting, not just fewer calls
50-iter ceiling hits
47 → 37
fewer trials run out of budget
04FRICTION · FAILURE MODES
Search-rich, edit-poor
Passing still requires a write in both jobs (zero-edit & pass = 0 in both). The extra search did not convert into patches — zero-edit & fail rises 25 → 31.
Early abandon, zero-edit
django-12209 50/51 → 24/17 iters · matplotlib-25332 51/51 → 18/13 · xarray-4094 50 → 27 · sphinx-9591 53 → 27 with only 1× run_command (never tested).
Searched but never committed
pylint-6386 52 → 57 tools (grep12 + read34, 0 edit vs baseline 2 edits) · sympy-12419 / 18698 / 19495 all ~50 iters, 0 edit · sklearn-13124 searched more (25 → 45 tools) yet flipped pass→fail.
Style, not outcome
run_command drops in both cohorts (pass-group mean 27.0 → 18.7, med 24 → 16; fail-group 28.6 → 20.8, med 31 → 19). grep_search rises symmetrically (pass 1.6 → 5.6, fail 1.9 → 5.9). Static search substituted dynamic repro/pytest — not just ls/grep exploration.
New-pass anatomy is the mirror
All 7 new passes contain 1–3 edits with a full grep + find + read + edit chain: sphinx-10449 (find9 + grep17 + read21 + edit1), astropy-13977 (62 tools, edit3 + patch1). django-15161 passed via brute force (77 tools, run72) — the only non-search win.
05COMPARISON · SIDE BY SIDE
The plateau survives a model swap
Ling-3.0-flash sits at 41–44 across four harnesses. Put a different model on the same harness and the score stays inside the band (40, noise) — while its tool behavior transforms. Score follows the harness, not the model.
Slate bars = Ling-3.0-flash; gradient bar = deepseek-v4-0731 (this report). All differences within the band are judged noise (se≈4.9% at n=100). The score only moves when the harness changes what the agent must prove — see the code-v11 report.
06CONCLUSION
The tools finally won. The score didn't — yet.
v9 的 glob_files → find_files 改名在这个模型上验证了可发现性（1%→58%），搜索采纳终于落地；但 localization 不再是瓶颈，edit-commit + edit→verify 循环才是。不要再追搜索工具了。
07NEXT STEPS · ONE GENE AT A TIME
Enforce the edit→verify micro-loop
- Do not chase another search tool: the rename is validated; search adoption is solved for this model.
- Enforce the micro-loop (already proposed in v7-T2): verification.enabled=true or a harness rule — "every edit_file/apply_patch must be followed by a targeted run_command (repro/pytest) within N rounds; reject zero-edit submissions".
- Validation metrics for the next run: zero-edit-fail count 31 → ≤25; run_command-within-3-rounds-after-edit ratio; first-edit round median — pass stays primary.
- Must-watch: the 10-regression list above (django-12209, matplotlib-25332, xarray-4094, pylint-6386, sklearn-13124, sphinx-9591, sympy-12419/18698/19495, django-14007).
- This validation ran: as eval100-code-v11-deepseek-v4-0731 — return-contract verification lifts 40 → 48. Read the v11 report →
08REPRODUCTION
Artifacts and code locations
Artifacts
jobs/eval100-code-v9-deepseek-v4-0731/ (100/100 completed, mean 0.40)
Baseline: jobs/eval100-code-v9/ (100/100 completed, mean 0.43 · 43×1.0, 56×0.0 + 1 missing trial, n_trials=99)
Record: evaluation/records/eval100-code-v9-deepseek-v4-0731__code-v9__dataset-eval100.json
Baseline: jobs/eval100-code-v9/ (100/100 completed, mean 0.43 · 43×1.0, 56×0.0 + 1 missing trial, n_trials=99)
Record: evaluation/records/eval100-code-v9-deepseek-v4-0731__code-v9__dataset-eval100.json
# harness gene (identical to code-v9 — the variable here is the model) harnesses/code-v9.yaml = code-v8 + tools.enabled: glob_files -> find_files # tools: [run_command, read_file, apply_patch, edit_file, write_file, grep_search, find_files] # max_iterations: 50 · verification.enabled: false · memory: llm_summary # counting rules authoritative = tool.start # data.tool is the call-count authority iterations = llm.start / llm.end # iteration counts only tokens = null # result.json has no token accounting — no $ comparison
Full analysis report: github.com/johnsonbuilds/sunagent · evaluation/analysis/eval100-code-v9-deepseek-v4-0731.md. For run commands, defer to the repo README and the actual artifacts under jobs/; this page only visualizes the report — data and conclusions match it.