code-v6: 43/99 baseline, difficulty validated
Same model, different harness. — code-v6 = code-v4 + max_iterations 50; the eval100 baseline everything after compares against.
PASS RATE
43 / 99
mean 0.43 · 3 errored counted separately (2×AgentTimeoutError + 1×VerifierTimeoutError)
DIFFICULTY TIERS
0.59 / 0.44 / 0.11
Low 22/37 · Medium 19/43 · High 2/19 — monotonic, proxy empirically validated
SMOKE-10
7 / 10
identical to historical best; failure signatures reproducible
TOOL VOLUME
4184 calls
only 20 errors (0.5%) · trial median 7.5min / mean 10.6min / max 62.8min
Verdict: solid 0.43 baseline with validated difficulty labels — 14 near-misses are the shortest path from 0.43 → 0.55.
01SETUP · EXPERIMENT DESIGN
Baseline run: code-v4 plus a 50-iteration cap
One harness change versus code-v4 — max_iterations 50 — so the headline number is a clean baseline for the v7/v8 gene experiments that follow.
Harness gene
harnesses/code-v6.yaml = code-v4 + max_iterations 50
run_command
read_file
apply_patch
edit_file
4 parallel · 100 tasks · wall 4.6h
Dataset & difficulty proxy
evaluation/swe_bench/eval-100.json (smoke-10 + dev-50 + 40 new). Difficulty from manifest human-estimated fix duration: Low <15min / Medium 15min–1h / High 1–4h+.
Counting rules
43/99 scored; 3 errored counted separately as unrelated to the model: 2×AgentTimeoutError (pylint-4661/4970, agent 3600s timeout), 1×VerifierTimeoutError (sphinx-7590, suite > 1800s).
02HEADLINE · KEY RESULTS
0.43 with monotonic difficulty tiers
The pass rate falls cleanly with the human-estimated difficulty proxy — Low 0.59, Medium 0.44, High 0.11 — so the tiers can serve as official difficulty labels going forward.
Grey = tier size, gradient = passes. The 2 high-tier passes were both django (13449 smoke + 14007). Overall 43/99 = 0.43; 3 errored counted separately.
smoke-10 subset: 7/10, identical to historical best
passed: astropy · django · pylint · psf · pytest · sklearn · sympy
failed: sphinx (f2p 0/2) · matplotlib (fixed target, broke regression p2p 71/81) · xarray (f2p 0/1)
failure signatures consistent with previous 5 smoke rounds, reproducible
Verifier health: clean
No insane totals (81/25470 class-bug zero recurrence). Exit distribution 0:40 / 1:55 / 2:3 / 4:1, all semantically normal.
03ADOPTION · TOOL UPTAKE
The model votes with its feet: edit_file 8:1 over apply_patch
4184 tool calls with only 20 errors (0.5%) — and a clear revealed preference for single-point edits over batch patches.
run_command
3109
shell repro / test workhorse
read_file
859
paginated reads
edit_file · voted default
193
edit:patch ≈ 8:1
apply_patch
23
batch multi-file, rarely chosen
Error budget: 20 / 4184 (0.5%)
unexpected_arg 8
unterminated 3
bad_header 3
match ambiguity 2
edit mismatch 4
Circuit breaker: loop.guard × 3
matplotlib-26113 · xarray-4094 · sphinx-8595 — all from consecutive apply_patch failures, consecutive=3 exact trigger.
04FRICTION · FAILURE MODES
56 fails: the next-phase goldmine
The last two categories — 14 near-misses — are the shortest path from 0.43 to 0.55.
| Category | Count | Meaning |
|---|---|---|
| Missed target, no regression | 24 | Pure capability gap (includes sphinx class with 0 edits) |
| Missed target + regression | 18 | — |
| Target fixed, regression failed | 10 | e.g. xarray-3993 (f2p 2/2, p2p 2374/2398, only 24 broken) |
| Multi-test target partially achieved | 4 | e.g. astropy-13977 (12/20), django-13212 (3/5) |
05COMPARISON · CAP SETTING
max_iterations=50: zero false positives
The cap question answered empirically: among 43 passes the latest successful edit landed at iteration 38 (median 14); 9 passes ran to iteration 51 on verification tail only.
Passes vs the cap
latest edit @ 38 · median 14
9 passes ran to iteration 51 (50 iterations + summary round) — verification tail, not late fixes.
Fails vs the cap
46/100 hit the cap · 36 fails
Among fails, 12 had zero successful edits throughout — more iterations wouldn't help; the cap saves pure waste.
Image GC: perfect score
100 images pulled, only 1 remaining; disk 55GB → 80GB available; no false positives.
06CONCLUSION
0.43 baseline stands; chase the 14 near-misses.
code-v6 设下 43/99 = 0.43 基线，难度分层单调可直接转正；target-fixed-but-regression（10）与 multi-test partial（4）是 0.43 → 0.55 的最短路径。
07NEXT STEPS · BY ROI
Where the next points come from
- 14 near-misses → edit→test→fix micro-loop (see Q&A section of the report for details).
- High-tier 2/19 → switch model or drop; stop tuning on smoke (overfitting risk).
- sphinx total wipeout → standalone project (won't even write patches, unrelated to editor).
08REPRODUCTION
Artifacts and code locations
Artifacts
jobs/eval100-code-v6-1/ (100 tasks, 4 parallel; 99 scored + 3 errored)
Dataset: evaluation/swe_bench/eval-100.json (smoke-10 + dev-50 + 40 new)
Dataset: evaluation/swe_bench/eval-100.json (smoke-10 + dev-50 + 40 new)
# harness gene (baseline for all later A/B) harnesses/code-v6.yaml = code-v4 + max_iterations 50 # headline numbers score = 43/99 = 0.43 # 3 errored counted separately tiers = Low 22/37=0.59 / Medium 19/43=0.44 / High 2/19=0.11 smoke = 7/10 # identical to historical best tools = run 3109 / read 859 / edit 193 / apply 23 (total 4184, 20 errors) guard = loop.guard x3 (consecutive=3, apply_patch)
For run commands, defer to the repo README and the actual artifacts under jobs/; this page only shows the gene definition and headline numbers from the report.