code-v7: + write_file, non-regression, zero benefit
Same model, different harness. — code-v6 plus exactly one tool; the zero-score autopsy that motivated code-v8.
PASS RATE
41 / 100
mean 0.41 · 41×1.0 + 59×0.0 · net vs v6 −2pt (noise)
ZERO-SCORE SPLIT
22 + 37
22 never wrote to disk · 37 edited but still failed
TOTAL TOOL CALLS
4184 → 4103
−2% · run_command flat, edit 193→147 (coverage 88→77 tasks)
write_file ADOPTION
0 / 100
zero adoption — no regression but no benefit either
Verdict: +write_file non-regression holds, but zero benefit — the 59 zero-score autopsy points at search cost, not file creation.
01SETUP · EXPERIMENT DESIGN
One-gene change: one new tool
Relative to code-v6, code-v7 adds exactly write_file to the tool list — so the −2pt net and the 12 flips can be read as noise around a single-gene edit.
Harness gene
harnesses/code-v7.yaml = code-v6 + tools.enabled += write_file
run_command
read_file
apply_patch
edit_file
+ write_file
Counting rules
tool.start is authoritative (llm.end.messages compressed via llm_summary retains only ~70% of calls, used for intent sampling only). Baseline jobs/eval100-code-v6-1/ (43×1.0 / 56×0.0 / 3 errors, mean ~0.43).
Related code
Tool prefer definitions: src/agent_runtime/tools/tools.py:77-336
02HEADLINE · KEY RESULTS
41 vs 43: 12 flips, within noise
Both-pass 36, both-fail 49 (hardcore); 7 regressions vs 5 improvements with a consistent signature — v7 edited less and shelled more, exploration got stuck, not write_file's fault.
Pass comparison (same task set eval100)
code-v6 43 (43×1.0 / 56×0.0 / 3 errors)
code-v7 41 (41×1.0 / 59×0.0)
net −2pt · 12/100 flips · within noise
regressions 7 · improvements 5
Improvements — 5 tasks (vs v6)
xarray-4094xarray-6938pytest-10051
sklearn-25973sphinx-9591
Regressions — 7 tasks (vs v6)
django-12209django-14007django-15161
matplotlib-25332requests-1724
sklearn-14087sympy-12419
Each row normalized to its own v6/v7 maximum. apply_patch 23 → 21 (10/100 trials); write_file 0/100; read_output 0/100. Regression pattern: v7 fewer edits, more shell (sklearn-14087 3→0 edits, requests-1724 4→1, django-15161 12→2, sympy-12419 reads 15→4 / runs 34→46).
03ADOPTION · TOOL UPTAKE
Compliant in name, substitute in practice
4103 calls: run_command 3101 (100/100) / read_file 834 (97/100) / edit_file 147 (77/100) / apply_patch 21 (10/100) / write_file 0 / read_output 0. All 41 passing tasks contained edit calls (77 total, ~1.9/task) — writing to disk is a necessary condition for passing.
run_command · 75.6%
3101
Visible subset ~50% grep/find/ls/git log exploration, ~20% repro, ~18% pytest. Harness enabled no search tool, so exploration went through shell.
read_file · compliant
834
3 exceptions bypassed prefer: matplotlib-25775 (50 calls all run, sed -n reads), requests-6028, sympy-18698.
edit_file · voted default
147
Compliant single-point small edits; every pass task contains edit calls.
apply_patch · right scene, wrong use
21
15 calls in zero-score group with 8 validation failures — fixable format issues.
write_file (v7 new) · 0/100 zero adoption
No regression but no benefit either — schema self-description didn't change model habits.
read_output · 0/100
Not triggered or replaced by re-runs (.outputs/ overflow path unused).
04FRICTION · FAILURE MODES
59 zero-score tasks: 22 never wrote, 37 wrote and failed
By repo: matplotlib 8/8, mwaskom 2/2, pylint 8/9, django 7/10, sympy 7/10, sphinx 6/10, pytest 6/11, astropy 5/10, pydata 4/11, sklearn 4/11, psf 2/7, pallets 0/1.
A · Never wrote to disk — 22/59
astropy-13398/13977/14369, django-14725/15128, matplotlib-25775/26208, seaborn-3187, requests-6028, pylint-4970/7080/8898, pytest-6197/7236, sklearn-14087/25102, sphinx-11510/7985/9229, sympy-12419/14248/18698.
16 hit the budget ceiling (llm 51 rounds, ntools 42~55): "read-only loop" — e.g. sklearn-14087: 36×run + 14×read, root cause identified but submitted text only.
6 stopped early (ntools<35): requests-6028, sklearn-25102, sphinx-11510/9229, sympy-18698, pylint-4970.
16 hit the budget ceiling (llm 51 rounds, ntools 42~55): "read-only loop" — e.g. sklearn-14087: 36×run + 14×read, root cause identified but submitted text only.
6 stopped early (ntools<35): requests-6028, sklearn-25102, sphinx-11510/9229, sympy-18698, pylint-4970.
B · Edited but still failed — 37/59
edit_file 70 successes + 1 failure; apply_patch 15 calls with 8 failures; 22/37 also hit the 50-step ceiling (wrong edit → verify fail → no steps left).
apply_patch misuse: matplotlib-24149 passed unified-diff, sphinx-8595 3× missing >>>>>>> REPLACE, xarray-6992 all 4 failed.
Env issues: astropy-13236 (erfa/numpy install), xarray-3993/6992 30s timeout, matplotlib-24149/8595 OSError: Argument list too long: 'docker'.
Zero-score group 38/59 hit ≥50 calls (pass group only 15/41); mean llm rounds 43.8 vs 35.9.
apply_patch misuse: matplotlib-24149 passed unified-diff, sphinx-8595 3× missing >>>>>>> REPLACE, xarray-6992 all 4 failed.
Env issues: astropy-13236 (erfa/numpy install), xarray-3993/6992 30s timeout, matplotlib-24149/8595 OSError: Argument list too long: 'docker'.
Zero-score group 38/59 hit ≥50 calls (pass group only 15/41); mean llm rounds 43.8 vs 35.9.
05COMPARISON · SIDE BY SIDE
v6 → v7: fewer edits, same shell
Total calls 4184 → 4103 (−2%), run flat, edit 193 → 147 with coverage 88 → 77 tasks. Conclusion: +write_file non-regression holds, but zero benefit.
| Metric | code-v6 | code-v7 | Δ |
|---|---|---|---|
| Pass | 43 (56 fail + 3 err) | 41 (59 fail) | −2pt · noise |
| Total tool calls | 4184 | 4103 | −2% |
| run_command | 3109 | 3101 (100/100) | flat |
| read_file | 859 | 834 (97/100) | −25 |
| edit_file | 193 | 147 (77/100) | −46 · coverage 88→77 |
| apply_patch | 23 | 21 (10/100) | −2 |
| write_file | n/a | 0/100 | zero adoption |
06CONCLUSION
Non-regression holds; the cost is search, not file creation.
v7 的 +write_file 无回退亦无收益（41 vs 43 属噪声）；~一半 run_command 在做 grep/find/ls 探索——下一步应加搜索工具而非继续改写文件路径。
07NEXT STEPS · ONE GENE AT A TIME
T1 / T2 / T3 by ROI
- T1 · tools.enabled += grep_search, glob_files (planned code-v8): ~half of run_command is exploration; 3 bypass-reads and 7 regressions (incl. sympy-12419 read collapse) all trace to inefficient search → budget burned → zero writes. Expected: exploration steps halved; most salvageable zero-write tasks convert to edits. Validate: run_command ratio, first-edit round median, zero-write count.
- T2 · enforce edit→verify micro-loop (planned code-v9: verification.enabled=true or prompt "apply small patch then run targeted tests"): 38/59 zero-score hit the ceiling while the pass group averages 8 fewer rounds; sklearn-14087-class "analysis correct, no write" is directly convertible. Validate: first-edit round, pytest-within-3-rounds ratio, both-fail 49 narrowing.
- T3 · fix apply_patch schema comprehension + give write_file a real role (planned code-v10: minimal SEARCH/REPLACE few-shot, write_file scoped to create/full-rewrite): 8 apply failures all format issues; write_file 0/100 means self-description didn't move habits. Validate: apply error count, write_file adoption, multi-file (django/sphinx) pass rate.
08REPRODUCTION
Artifacts and code locations
Artifacts
jobs/eval100-code-v7/ (100/100 completed, 41×1.0 + 59×0.0, mean 0.41)
Baseline: jobs/eval100-code-v6-1/ (43×1.0 / 56×0.0 / 3 errors, mean ~0.43; eval100-code-v6 with only 19 completed excluded)
Baseline: jobs/eval100-code-v6-1/ (43×1.0 / 56×0.0 / 3 errors, mean ~0.43; eval100-code-v6 with only 19 completed excluded)
# harness gene (the only difference in this experiment) harnesses/code-v7.yaml = code-v6 + tools.enabled += write_file # tool list: [run_command, read_file, apply_patch, edit_file] → + write_file # counting rules authoritative = tool.start # llm.end.messages (llm_summary) keeps ~70%, intent sampling only # related code src/agent_runtime/tools/tools.py:77-336 # tool prefer definitions
For run commands, defer to the repo README and the actual artifacts under jobs/; this page only shows the gene definition and counting rules from the report.