Benchmark home/eval100-code-v8

EXPERIMENT · EVAL100 · CODE-V8 · ONE-GENE A/B

code-v8: + grep_search / glob_files

Same model, different harness. — code-v7 plus exactly two search tools; everything else untouched.

Model Ling-3.0-flash Task set eval100 (100 tasks) Baselines code-v7 / code-v7-2 Completion 100/100 jobs Report jobs/eval100-code-v8/

PASS RATE

41 / 100

mean 0.41 · 41×1.0 + 59×0.0 · net vs v7 0 (non-regression)

TOTAL TOOL CALLS

4103 → 3862

−6% — efficiency is the only real win

run_command CALLS

3101 → 2793

−10% (−308 calls) — search replaces shell exploration

LLM ROUNDS (MEAN / MEDIAN)

40.6 → 36.6

−10% · median 51 → 44

Verdict: pass non-regression holds, search cost down ~10% — keep both tools, but don't expand (Keep, don't expand).

01SETUP · EXPERIMENT DESIGN

One-gene change: one harness, two new tools

The harness difference is a single gene, so causal attribution stays clean: relative to code-v7, code-v8 only adds grep_search and glob_files to the tool list.

Harness gene

harnesses/code-v8.yaml = code-v7 + tools.enabled += grep_search, glob_files

run_command read_file apply_patch edit_file write_file + grep_search + glob_files

Counting rules

Counts use tool.start as the authority; llm.start messages with role=tool/user are used only to dedup error kinds by _id.

Related code

src/agent_runtime/tools/search.py:82-182
specs src/agent_runtime/tools/tools.py:221-263,344-345

02HEADLINE · KEY RESULTS

Pass non-regression, efficiency is the win

v8 and v7 each pass 41 tasks — net zero. The flips are perfectly symmetric and the McNemar statistic is 0: pure noise. Efficiency is the only real win.

Pass comparison (same model Ling-3.0-flash · eval100)

code-v7 41 code-v7-2 44 (tool-describe-only, judged noise) code-v8 41 McNemar = 0 → pure noise vs v7-2 (44): +5 / −8, also noise

Newly passed — 6 tasks (vs v7)

matplotlib-25332requests-1724sklearn-25102 sympy-12419sympy-15599sympy-18698

Newly failed — 6 tasks (vs v7)

django-12858django-13568requests-1921 xarray-6599xarray-6938sklearn-13124

Each row is normalized to its own v7/v8 maximum — read relative change only; the numbers are the report's raw values. edit_file trials 77→74; apply_patch error rate 15/6 vs 12/4 (~71%→75%, n too small — judged flat).

Additional observations

Trials using run_command: 100 → 93 Trials using read_file: 97 → 92 7 trials dropped the shell entirely edit_file back at the v7 level (147→146)

03ADOPTION · TOOL UPTAKE

grep_search half-won, glob_files failed

Whether the new tools are actually used — and used correctly — matters more than whether they were added.

grep_search · adopted reliably

122 calls / 44 trials

120 successes + 2 errors (1.6% error rate)
pass among users 19/44 = 43%
non-users 22/56 = 39% (+4pp, n.s. — at least not toxic)

glob_files · undiscovered

3 calls / 3 trials

Effectively unused: the ls / find habit in the shell persists

write_file · first-ever adoption

1 call / 1 trial

requests-2931, which passed; single sample, no signal — zero use on SWE-bench remains expected (no full-rewrite tasks)

read_output

Still zero use

04FRICTION · FAILURE MODES

Model priors vs tool schema

Three concrete failure modes, all telling the same story: the model arrives with pretrained calling habits, and our schema didn't catch them.

Hallucinated short name grep · 3 trials

requests-1766 / xarray-6599 / pytest-5809 hit unknown tool: grep. Adding grep_search activated the short-name association without capturing the name.

Pretrained kwargs pollution · 2 trials

grep_search() got an unexpected keyword argument 'output_mode' (astropy-14995, repeated), 'type' (django-14007). The model expects a ripgrep-style schema.

Our schema

pattern / path / include / ignore_case / max_results

Counting-rule warning

Repeated tool errors keep accumulating in llm.start messages and inflate the history — dedup by unique _id is required (same lesson as the v7-2 malformed-JSON single-trial case).

05COMPARISON · SIDE BY SIDE

41–44 is the current plateau

v7-2 (tool-describe-only) was already judged no-effect; v8's 41 further confirms that this band is where Ling-3.0-flash stands on eval100.

v7-2 changed tool descriptions only: patch behavior 21→22 calls, 10→10 trials — no effect; v8 did not convert v7's zero-write (22/59) or edited-but-failed (37/59) groups into passes at scale; the flips are symmetric with no repo-level pattern.

06CONCLUSION

Keep, don't expand.

v8's +grep_search/glob_files non-regression holds, with ~10% search-cost saving. Keep both tools — but don't keep piling tools into the harness.

07NEXT STEPS · ONE GENE AT A TIME

Fix the schema, don't add tools

Alias or explicitly reject: handle output_mode / type in the grep_search description — either accept them as aliases or reject them with a clear error.
Capture the short name: add a grep alias, or state in the description "don't call it grep".
Rescue glob_files: give it a basename example so the model can discover it.
Re-test with a stronger model: swap a stronger LLM onto the winning harness; the metric is pass/$, not just pass (per plan).

08REPRODUCTION

Artifacts and code locations

Artifacts

jobs/eval100-code-v8/ (100/100 completed)
Baselines: jobs/eval100-code-v7/ (41×1.0, mean 0.41) · jobs/eval100-code-v7-2/ (44×1.0, mean 0.44)

# harness gene (the only difference in this experiment)
harnesses/code-v8.yaml = code-v7 + tools.enabled += grep_search, glob_files
# tool list: [run_command, read_file, apply_patch, edit_file, write_file] → + grep_search, glob_files

# counting rules
authoritative = tool.start          # the authority for call counts
dedup        = llm.start._id        # role=tool/user messages are only for error-kind dedup

# related code
src/agent_runtime/tools/search.py:82-182
specs src/agent_runtime/tools/tools.py:221-263,344-345

For run commands, defer to the repo README and the actual artifacts under jobs/; this page only shows the gene definition and counting rules from the report.