tb-v2 × mimo-2.6-pro: 64/89 · 0.7191
Same harness, stronger model — and nearly half the zeros turn out to be infrastructure, not capability.
89 trials, 4 classes of zero
| Class | Count | Nature |
|---|---|---|
| 1 · LLM API 429 rate-limit crash | 9 | Infrastructure |
| 2 · AgentTimeoutError | 2 | Budget / task-duration |
| 3 · Verifier env failure (apt install) | 2 | Infrastructure (wrongful zeros — rerun advised) |
| 4 · Agent finished, tests failed | 12 | True capability gap + borderline near-misses |
Nine 429 deaths in one half-hour window
agent-runtime.jsonl ends with agent.end status=error, LLM error: Error code: 429 - Too Many Requests — the agent process dies mid-task and the unfinished work scores 0.
The 9 zero-trials
All clustered in the 2026-10-19 01:31–02:02 UTC half-hour — a textbook API rate-limit window. Corroboration: kv-store-grpc__Ufmjtbd (00:34) and regex-log__Liy53Ue (01:44) died the same 429 death but had finished the solution first, so the verifier still gave 1.0 — the blast radius is really ~11 trials, and whether a trial scored was pure luck of death timing.
2 timeouts, 2 wrongful zeros
AgentTimeoutError (2)
torch-pipeline-parallelism__iPsnako (2700s) · train-fasttext__TAQ8Tvh (10800s), both at timeout_multiplier=3.0. (query-optimize__mBH2YMY also hit 2700s but finished first — scored 1.0.)
Verifier env failure — wrongful zeros (2)
qemu-alpine-ssh__LvUiGBy · qemu-startup__9fQAmAP — no ctrf.json; test-stdout shows the verifier's own script never ran: bullseye-security 404 on libnghttp2/curl debs → curl missing → uv missing → uvx missing. Only 2 of 89 trials hit this. Pure infra wrongs — rerun advised.
E: Failed to fetch .../libnghttp2-14_1.43.0-1+deb11u3_amd64.deb 404 Not Found E: Failed to fetch .../curl_7.74.0-1.3+deb11u16_amd64.deb 404 Not Found /tests/test.sh: line 8: curl: command not found /root/.local/bin/env: No such file /tests/test.sh: line 19: uvx: command not found
Twelve all-or-nothing failures — mostly near-misses
Reward is all-or-nothing: one failing sub-test is a zero.
| Case | Pass rate | Failure |
|---|---|---|
| schemelike-metacircular-eval | 59/63 | closures.scm + 05-simple-io.scm (shadow_test), test_interp assert 4==0 |
| sam-cell-seg | 8/9 | test_mask_alignment: IoU 0.4960 vs 0.5 — off by 0.004 |
| dna-insert | 0/1 | test_primers: Tm gap 5.268 vs ≤5 |
| mcmc-sampling-stan | 5/6 | test_hierarchical_model_implemented: Stan structure check timeout |
| raman-fitting | 2/3 | test_2D_Peak: x0/gamma/A right, offset wrong |
| pytorch-model-cli | 5/6 | test_cli_tool_output: predictions mismatch |
| gcode-to-text | 1/2 | flag typo: flag{gc0de_iz_ch4LLenging} vs flag{gc0d3_iz_ch4LLenGiNg} |
| mteb-retrieve | 1/2 | result.txt has HumanEval instead of MTEB (wrong dataset) |
| extract-elf | 1/2 | output match 0% vs ≥75% (format completely off) |
| make-mips-interpreter | 1/3 | no DOOM boot output; frame.bmp similarity 0.756 < 0.95 |
| make-doom-for-mips | 0/3 | VM exec timeout; frame.bmp never generated |
| extract-moves-from-video | 1/2 | content similarity 27.8% < 90% (drifted/clever solution) |
~0.80 ex-infra; near-misses dominate the rest
Three fixes
- Rerun (or exclude) the 11 429-hit trials + 2 apt-fault trials before quoting a capability number.
- Add 429 backoff-retry to the agent SDK — exponential backoff + jitter, or extend provider_error.max_retries to cover the 429 path.
- Pin the verifier image apt sources (or preinstall curl/uvx) so the test toolchain can't 404.