SunAgent Harness Benchmark

LEADERBOARD

eval100 Leaderboard

Data from real runs

Current series — same model Ling-3.0-flash, same task set eval100 (100 tasks, 100/100 jobs completed). Click the pass rate / tool calls headers to sort.

#	Harness	Model	eval100 pass ▼	Δ vs code-v7	Tool calls	llm rounds (median)	Notes
1	code-v7 Baseline tool set	Ling-3.0-flash	41/100 · 0.41	—	4103	51	[run_command, read_file, apply_patch, edit_file, write_file]
2	code-v7-2 Tool-describe-only change	Ling-3.0-flash	44/100 · 0.44	+3 (judged noise)	—	—	tool-describe-only · judged no effect
3	code-v8 + grep_search, glob_files	Ling-3.0-flash	41/100 · 0.41	0 (non-regression)	3862 (−6%)	44 (51→44)	Adds 2 search tools · search cost ≈ −10% · Read report →

Pass-rate differences are judged noise by the McNemar test (41 vs 41 net change 0; vs v7-2 +5/−8). "—" means the stat was not provided for that run; such rows sort last.

TREND

Pass rate sits in a 41–44 plateau

All three harnesses land inside the same band and the difference is judged noise. Efficiency is where v8 actually wins — total tool calls 4103→3862 (−6%), run_command 3101→2793 (−10%), median llm rounds 51→44.

Task set eval100 (100 tasks) · Model Ling-3.0-flash · Data: jobs/eval100-code-v7, jobs/eval100-code-v7-2, jobs/eval100-code-v8. Efficiency details in the eval100-code-v8 report.

METHODOLOGY

Controlled Harness Experiments

We modify one or more related harness components and evaluate their impact through controlled experiments: does pass regress, does cost drop, are the new tools actually used?

harnesses/code-v7.yaml

run_command read_file apply_patch edit_file write_file

Baseline · 41/100 · mean 0.41

+ 2 TOOLS

harnesses/code-v8.yaml = code-v7 + search tools

run_command read_file apply_patch edit_file write_file grep_search glob_files

41/100 · non-regressiontool calls −6%run_command −10%

Controls & counting rules

Same model: Ling-3.0-flash Same task set: eval100 (100 tasks) 100/100 jobs completed Counts based on tool.start Errors deduped by llm.start _id

Measures

pass mean tool calls llm rounds (mean/median) error rate tool adoption

EXPERIMENTS

Experiment log

Each report is a static page under public/sunagent/, structured as: Setup → Headline → Adoption → Friction → Comparison → Conclusion / Next steps. This homepage updates with every report.

EVAL100 · CODE-V7Completed

code-v7 · Baseline

Result: 41/100 (mean 0.41), 4103 tool calls, median 51 llm rounds.

Report in preparation

EVAL100 · CODE-V7-2Judged noise

code-v7-2 · Tool-describe-only

Question: does changing tool descriptions alone help?
Result: 41→44 is noise; patch behavior 21→22 calls, 10→10 trials — no effect.

Report in preparation

EVAL100 · CODE-V8Completed

code-v8 · +grep_search/glob_files

Result: pass non-regression (41 vs 41, McNemar 0), total calls −6%, run_command −10%; grep_search half-won, glob_files undiscovered. Keep, don't expand.

Read the full report →

NEXT · SCHEMAPlanned

Fix the schema, don't add tools

Plan: alias or explicitly reject output_mode and type; add a grep alias; give glob_files a basename example so it gets discovered.

Pending

NEXT · STRONGER LLMPlanned

Re-test the winning harness with a stronger model

Plan: swap a stronger LLM onto the winning harness; the metric is pass/$, not just pass.

Pending