tb-v3 × deepseek-v4.1-flash: 64/89 · 0.7191
Same 0.7191 as mimo — but 60% of the zeros are runtime, not model. Two time black holes eat the budget before the model ever gets to think.
Same dataset, three different failure mixes
| Job | Harness / Model | Zeros | Infra-class | True capability |
|---|---|---|---|---|
| terminal-bench-2-1-tb-v2 | tb-v2 / deepseek-v4 | 35 (39.8%) | 22 (15 stream + 7 timeout) | 13 |
| terminal-bench-2-1-tb-v2-mimo-2.6-pro-ultraspeed | tb-v2 / mimo-2.6-pro | 25 (28.1%) | 0 | 25 |
| terminal-bench-2-1-tb-v3-deepseekv4.1-flash-1017 | tb-v3 / deepseek-v4.1-flash | 25 (28.1%) | 15 | 10 |
8 killed by the clock — signature: CancelledError at exactly base×3.0
result.json agent_runtime.error = {"type": "CancelledError"} and wall time equals task.toml [agent] timeout_sec × 3.0. 4 more trials hit the same wall but finished first (filter-js-from-html 5400s, torch-pipeline/torch-tensor 2700s, tune-mjcf 2700s) — 12 wall hits total, 8 fatal.
| Case | Base → ×3 actual | Summary burn | Long tools | Iters |
|---|---|---|---|---|
| cobol-modernization | 900s → 2700s | 1.0 min | — | 56 |
| count-dataset-tokens | 900s → 2700s | 0.1 min | 2×901s | 37 |
| gcode-to-text | 900s → 2700s | 22.6 min | 2×120s | 63 |
| make-doom-for-mips | 900s → 2700s | 8.9 min | 4×(250~380)s | 39 |
| large-scale-text-editing | 1200s → 3600s | 0 min | 51.2 min | 45 |
| make-mips-interpreter | 1800s → 5400s | 46.7 min | — | 86 |
| schemelike-metacircular-eval | 2400s → 7200s | 104.8 min (87% of budget) | — | 28 |
| train-fasttext | 3600s → 10800s | 0 min | 176.8 min | 34 |
Black hole 1 · memory.llm_summary — 202 calls / 4.98h
memory.py:378-396 _generate_summary re-sends the whole history via non-streaming llm.chat() — no idle timeout, no size cap. Single summaries cost 250~700s: schemelike 15 summaries ate 104.8/120 min (agent reasoning left ~15 min), make-mips-interpreter 46.7/90, gcode-to-text 22.6/45, video-processing 55.2/85. Compression often fails open: request_chars 70k~78k > 60k budget (video-processing original_chars → 394k), re-triggering summaries in O(n²) growth.
Black hole 2 · unbounded tool timeout — no budget awareness
run_command schema (tools.py:92) + execute_code (code.py:223) accept any timeout number; the model passes 600/900/1200s and the executor obeys. train-fasttext: 17 long commands (600~1201s), 176.8/180 min in tools, re-running the same timed-out train command. Tool budget and agent wall budget have no linkage.
Tracing trap: trace.span() hides the kill
trace.py:99-109 span() only catches Exception — asyncio.CancelledError inherits BaseException since Py3.8, so status = "error" if failed else "success" in finally always lands on success. All 8 killed trials end their trace with agent.end {status: "success"} while result.json says CancelledError. Reading trace alone yields the wrong conclusion ("agent finished normally").
7 dead on first packet — TTFT p99 sits exactly on the 120s cliff
adaptive-rejection-sampler · circuit-fibsqrt · feal-linear-cryptanalysis · gpt2-codegolf · path-tracing-reverse · regex-chess · write-compressor. Uniform error: LLM stream produced no chunk for 120 seconds (chat.py:143-148).
Why retries can't save it
AGENT_RUNTIME_STREAM_IDLE_TIMEOUT defaults to 120s; tb-v3 stream_idle_timeout.max_retries=5 means replaying the same request 5× = 10 idle minutes, then loop.py:273-277 raises _TurnFailure and the whole turn ends — discarding 13~40 completed rounds. Failed iterations received chunk count = 0: not mid-stream interruption but TTFT timeout. Job-wide TTFT over 4247 successful calls: p50 2.4s · p90 18.7s · p99 119.4s · max 708.9s — the 120s threshold sits exactly on the cliff edge. 99 retries = 3.3h pure waiting; 24/89 trials triggered at least once. All 5 retries replay the identical request — for "first reasoning packet inherently >120s" requests, retry deterministically reproduces the failure.
10 finished-but-failed — 2 wrongful, 8 real
Env faults — wrongful (2)
qemu-alpine-ssh · qemu-startup — verifier's own test.sh never ran (same bullseye 404 → curl → uv → uvx chain as tb-v2/mimo). Only 2/89 hit it. Rerun or fix the image.
Real gaps — near-miss flavor (8)
caffe-cifar-10 (acc gap 0.0823 > 0.05) · dna-assembly (BsaI ggtctc missing ≥1nt clamp, assert i ≥ 1) · install-windows-3.11 (VNC key <10% image diff) · pytorch-model-cli (15 test images nearly all misclassified) · rstan-to-pystan (beta[0] posterior -0.1455 outside [-0.07, 0]) · sam-cell-seg (IoU 0.4930 < 0.5, off 0.007) · video-processing (jump frame 98 vs [219,223]) · extract-moves-from-video (121/120 iters exhausted, never wrote /app/solution.txt).
All 25 zeros, one table
| Case | Class | State | Retry | Summaries | Summary time | Trace span |
|---|---|---|---|---|---|---|
| adaptive-rejection-sampler | 2 stream | failed | 5 | 0 | 0.0 min | 32.6 min |
| circuit-fibsqrt | 2 stream | failed | 5 | 0 | 0.0 min | 14.0 min |
| feal-linear-cryptanalysis | 2 stream | failed | 5 | 0 | 0.0 min | 12.3 min |
| gpt2-codegolf | 2 stream | failed | 6 | 0 | 0.0 min | 17.3 min |
| path-tracing-reverse | 2 stream | failed | 8 | 4 | 1.5 min | 30.5 min |
| regex-chess | 2 stream | failed | 5 | 0 | 0.0 min | 12.3 min |
| write-compressor | 2 stream | failed | 8 | 0 | 0.0 min | 19.1 min |
| cobol-modernization | 1 wall-clock | failed | 8 | 3 | 1.0 min | 45.0 min |
| count-dataset-tokens | 1 wall-clock | failed | 0 | 1 | 0.1 min | 45.0 min |
| gcode-to-text | 1 wall-clock | failed | 0 | 7 | 22.6 min | 45.0 min |
| large-scale-text-editing | 1 wall-clock | failed | 0 | 0 | 0.0 min | 60.0 min |
| make-doom-for-mips | 1 wall-clock | failed | 1 | 7 | 8.9 min | 45.0 min |
| make-mips-interpreter | 1 wall-clock | failed | 10 | 11 | 46.7 min | 90.0 min |
| schemelike-metacircular-eval | 1 wall-clock | failed | 2 | 15 | 104.8 min | 120.0 min |
| train-fasttext | 1 wall-clock | failed | 0 | 0 | 0.0 min | 180.0 min |
| caffe-cifar-10 | 3 verify | completed | 0 | 1 | 0.3 min | 73.4 min |
| dna-assembly | 3 verify | completed | 6 | 3 | 1.3 min | 49.5 min |
| extract-moves-from-video | 3 verify | completed | 0 | 14 | 6.9 min | 68.9 min |
| install-windows-3.11 | 3 verify | completed | 0 | 6 | 1.7 min | 27.8 min |
| pytorch-model-cli | 3 verify | completed | 0 | 1 | 0.2 min | 6.7 min |
| qemu-alpine-ssh | 3 verify (env) | completed | 0 | 4 | 1.4 min | 29.0 min |
| qemu-startup | 3 verify (env) | completed | 0 | 1 | 2.6 min | 22.6 min |
| rstan-to-pystan | 3 verify | completed | 0 | 3 | 0.7 min | 81.3 min |
| sam-cell-seg | 3 verify | completed | 0 | 4 | 0.8 min | 160.7 min |
| video-processing | 3 verify | completed | 0 | 18 | 55.2 min | 85.2 min |
Six fixes, in priority order
- Fix trace.span exception capture (trace.py:99): catch BaseException so CancelledError kills surface as error instead of success.
- Cap the summary layer: faster model via AGENT_RUNTIME_SUMMARY_MODEL, or streaming + own timeout, or deterministic compact_observations; cap summary input size (currently 186k~394k chars) and keep request_chars under context_budget.
- Stream idle 120s → 300s+ (chat.py:19); on retry exhaustion degrade (treat as empty reply, keep iterating) instead of ending the turn; backoff + request rewrite (e.g. shorter reasoning) instead of identical replay.
- Cap tool timeout and link it to remaining budget: min(user value, remaining budget × 20%); tell the model in the system prompt not to set hundred-second timeouts.
- Fix the qemu verifier image: pin apt to archive.debian.org or preinstall curl/expect/uv.
- Rerun the 8 wall-clock-killed + 2 env-fault trials, then re-rate the true capability level.