terminal-bench-2-1, tb-v2: 0.494 → 0.596
Budget reminder + 120 steps + agent-timeout ×4. Category C nearly eliminated — the biggest win lands exactly where the fix aimed.
tb-v2: the P0 config fixes, confirmed in-trace
First jsonl line reports tb-v2, control={max_iterations:120, budget_reminder:{enabled:true}}, verification.require=[solution_description,evidence]. Harness confirmed live — not just declared in yaml.
0.494 → 0.596: fixed 14, regressions 5
result.json: n_total_trials=89, evaluated=88, n_errors=9, mean=0.5955056179775281. filter-js-from-html passed despite AgentTimeoutError — work done before the timeout, mirroring v1 extract-elf.
FIXED · v1 zero → v2 pass (14 effective)
REGRESSIONS · v1 pass → v2 zero (5, variance/flakiness)
Three died on early LLM idle (≤22 steps) — likely to flip back on rerun. Not a harness regression.
Where the 35 zeros went
A · LLM stream idle, no chunk for 120s (15 — grew from 11, now largest)
cancel-async-tasks(2), circuit-fibsqrt(2), distribution-search(1), dna-assembly(19), count-dataset-tokens(44), llm-inference-batching-scheduler(5), model-extraction-relu-logits(22), raman-fitting(20), regex-chess(3), reshard-c4-data(4), torch-pipeline-parallelism(49), torch-tensor-parallelism(11), write-compressor(2), path-tracing-reverse(95), path-tracing(121). max_retries 2→5 delayed death but did not prevent it — next step is the stream-timeout env / model choice, not more retries.
D · verification.passed but official 0 (9 — unchanged, as expected)
caffe-cifar-10(62, rerun+passed; 5/6, accuracy gap 0.09 > 0.05), dna-insert(53; primer 55nt > 45), headless-terminal(42; 5/7), large-scale-text-editing(61; 4/5), protein-assembly(89), qemu-alpine-ssh(93), qemu-startup(54), rstan-to-pystan(57), sanitize-git-repo(79; same SHA d6987af dangling failure). qemu pair still bullseye-security 404 + curl/uvx not found — Debian bullseye EOL, agent-independent. tb-v2 require unchanged so no movement was expected.
B · agent timeout even at ×4 (7)
adaptive-rejection-sampler(7), cobol-modernization(53), extract-moves-from-video(65), gcode-to-text(73), gpt2-codegolf(42), schemelike-metacircular-eval(35), train-fasttext(35). Compile/train-style tasks — split the work, checkpoint artifacts, or raise the agent multiplier again.
C · budget exhausted at 120 steps (4 — down from 15, the biggest win)
mailman(121), make-doom-for-mips(121), make-mips-interpreter(121), video-processing(121) — all verification.failed: budget exhausted without a valid submit_result call. Remaining long-chain (MIPS/QEMU/mail) tasks need an artifact-existence gate next.
What tb-v2 leaves on the table
- Group A is now the largest (15/35): set AGENT_RUNTIME_STREAM_IDLE_TIMEOUT=120→180/300 (chat.py:19) and run this group on a stronger reasoning model; stop adding retries.
- Group D (9) + 2 near-miss regressions: implement coding auto-detection in loop.py:_validate_submission + verification.py (has_source_edit / executed_commands), plus an artifact-existence gate for sol.sql / result.txt / model.bin / :wq. caffe / dna-insert / headless / large-scale are each one check away.
- qemu-* (2): switch the build to archive.debian.org and preinstall curl — agent-independent free points.
- Rerun the 3 early-idle regressions (torch-tensor-parallelism, model-extraction-relu-logits, reshard-c4-data) — all stopped at ≤22 steps and will likely flip back to 1.