Terminal-Bench 2.0: per-arm detail¶

These are the per-arm result breakdowns for the Terminal-Bench 2.0 run: baseline, context-guru, headroom, and rtk.

Full results — baseline (Terminal-Bench 2.0, 89 tasks)¶

Baseline arm: no compaction — the claude-code agent on aws/claude-sonnet-5, run LIVE through the harness against Terminal-Bench 2.0's 89 tasks. Routing goes through the context-guru off transparent passthrough proxy (identical plumbing to the compaction arms; zero content change), so this is the like-for-like reference the framework arms are measured against. Cache-aware billed input cost (fresh $2/M · cache-read $0.20/M · cache-write $2.50/M) + output $10/M, recomputed from each trial's own token tiers — the same model as the SWE-bench study. See REPRODUCE.md.

Totals¶

* mean steps over the 82 completed tasks (timed-out runs are truncated). Solve rate over completed-only tasks: 56/82 = 68.3%.

Time-budget policy. Wall-clock budget = the task-authored timeout × a multiplier. Most tasks ran at 1.5×; the long-horizon tasks that first timed out were retried at low concurrency and, if still short, given an extended 4× budget (up to ~4 h) to measure capability rather than a latency-truncated result. 3 tasks solved only under the 4× budget (counted as solved here); the 7 below exhausted even 4×.

Analysis — where the agent is strong / weak¶

• Terminal-Bench 2.0 is a much harder, longer-horizon benchmark than SWE-bench Verified. The baseline solves 62.9% of tasks vs 86% on SWE-bench, at ~$1.13/task (vs $0.64) and ~1.7M prompt tokens/task — TB tasks are open-ended terminal goals (build/compile/train/exploit), not a localized patch, so the agent runs longer and reads far more.
• Difficulty is the dominant axis. Easy/medium solve at 71–75%; hard drops to 47% and costs 3.4× more per task ($2.04 vs $0.61). Every one of the 7 unrecoverable timeouts is a hard/long task.
• Category tells the same story from the other side. The agent is reliable on bounded, verifiable goals — debugging (100%), system-administration (78%), security (75%), data-processing/model-training (75%) — and weak on sprawling build/implement tasks: software-engineering (38%, the largest bucket at 26 tasks) and video-processing (0%).
• It is still a ~98%-cached agent (98.2% cache-hit), so — exactly as on SWE-bench — cache-read is the single biggest cost term (43% of the bill). That is the lever the compaction arms must pull; a layer that shrinks the cached context should move TB cost the same way it moved SWE cost.
• Latency, not just capability, caps the ceiling. The gateway's ~26 s/request (5–10× a normal endpoint) means long-horizon tasks can run out of wall-clock before finishing. 3 tasks that first timed out solved once given a 4× budget — so 62.9% is a floor, and a compaction arm that reduces round-trips could recover more of the remaining 7. The timeout count is therefore itself a comparison metric, not just noise.

Token & cost accounting (cache-aware, all 89 tasks)¶

Cache-read is 43% of the bill at a 98.2% cache-hit rate — as on SWE-bench, a heavily-cached agent, so the lever a compaction layer must pull is cache-read tokens.

Timeouts (7 long-horizon tasks)¶

These tasks still hit the wall-clock budget under the extended 4× timeout (up to ~4 h each) and scored reward 0 — counted as failures in the solve rate above. A large part of the cause is gateway latency, not only agent capability: Terminal-Bench's timeouts assume a fast endpoint (~2–5 s/request), but this IBM LiteLLM gateway runs ~26 s/request (5–10× slower), so long-horizon tasks that need many round-trips run out of clock (concurrency is not the cause — latency was flat ~23–30 s/req from n=1 to n=24). They are all hard/long software-engineering and compute tasks (path-tracing, a MIPS Doom port, a metacircular evaluator, COBOL modernization, GPT-2 code-golf, CIFAR training). A compaction arm that cuts round-trips could bring some under budget, so the timeout count is itself a comparison metric.

By difficulty (all 89 tasks; timeouts = failures)¶

By category (all 89 tasks)¶

Per-task (all 89)¶

Full results — context-guru (codesmart) (Terminal-Bench 2.0, 89 tasks)¶

Full per-task results for the context-guru (codesmart) arm on Terminal-Bench 2.0 (claude-code on aws/claude-sonnet-5, live). Same cache-aware cost model and 4× budget as the other arms. For the four-way analysis (cost decomposition, per-component, verdict) see the Terminal-Bench comparison; the reference arm is the baseline. See REPRODUCE.md.

Totals¶

* mean steps over the 78 completed tasks (timed-out runs are truncated). Solve rate over completed-only tasks: 57/78 = 73.1%.

Token & cost accounting (cache-aware, all 89 tasks)¶

Cache-read is 41% of the bill at a 96.9% cache-hit rate — as on SWE-bench, a heavily-cached agent, so the lever a compaction layer must pull is cache-read tokens.

Timeouts (11 long-horizon tasks)¶

These tasks still hit the wall-clock budget under the extended 4× timeout (up to ~4 h each) and scored reward 0 — counted as failures in the solve rate above. A large part of the cause is gateway latency, not only agent capability: Terminal-Bench's timeouts assume a fast endpoint (~2–5 s/request), but this IBM LiteLLM gateway runs ~26 s/request (5–10× slower), so long-horizon tasks that need many round-trips run out of clock (concurrency is not the cause — latency was flat ~23–30 s/req from n=1 to n=24). They are all hard/long software-engineering and compute tasks (path-tracing, a MIPS Doom port, a metacircular evaluator, COBOL modernization, GPT-2 code-golf, CIFAR training). A compaction arm that cuts round-trips could bring some under budget, so the timeout count is itself a comparison metric.

By difficulty (all 89 tasks; timeouts = failures)¶

By category (all 89 tasks)¶

Per-task (all 89)¶

Full results — headroom (hd-cache) (Terminal-Bench 2.0, 89 tasks)¶

Full per-task results for the headroom (hd-cache) arm on Terminal-Bench 2.0 (claude-code on aws/claude-sonnet-5, live). Same cache-aware cost model and 4× budget as the other arms. For the four-way analysis (cost decomposition, per-component, verdict) see the Terminal-Bench comparison; the reference arm is the baseline. See REPRODUCE.md.

Totals¶

* mean steps over the 78 completed tasks (timed-out runs are truncated). Solve rate over completed-only tasks: 64/78 = 82.1%.

Token & cost accounting (cache-aware, all 89 tasks)¶

Cache-read is 35% of the bill at a 94.1% cache-hit rate — as on SWE-bench, a heavily-cached agent, so the lever a compaction layer must pull is cache-read tokens.

Timeouts (11 long-horizon tasks)¶

These tasks still hit the wall-clock budget under the extended 4× timeout (up to ~4 h each) and scored reward 0 — counted as failures in the solve rate above. A large part of the cause is gateway latency, not only agent capability: Terminal-Bench's timeouts assume a fast endpoint (~2–5 s/request), but this IBM LiteLLM gateway runs ~26 s/request (5–10× slower), so long-horizon tasks that need many round-trips run out of clock (concurrency is not the cause — latency was flat ~23–30 s/req from n=1 to n=24). They are all hard/long software-engineering and compute tasks (path-tracing, a MIPS Doom port, a metacircular evaluator, COBOL modernization, GPT-2 code-golf, CIFAR training). A compaction arm that cuts round-trips could bring some under budget, so the timeout count is itself a comparison metric.

By difficulty (all 89 tasks; timeouts = failures)¶

By category (all 89 tasks)¶

Per-task (all 89)¶

Full results — rtk (Terminal-Bench 2.0, 89 tasks)¶

Full per-task results for the rtk arm on Terminal-Bench 2.0 (claude-code on aws/claude-sonnet-5, live). Same cache-aware cost model and 4× budget as the other arms. For the four-way analysis (cost decomposition, per-component, verdict) see the Terminal-Bench comparison; the reference arm is the baseline. See REPRODUCE.md.

Totals¶

* mean steps over the 82 completed tasks (timed-out runs are truncated). Solve rate over completed-only tasks: 55/82 = 67.1%.

Token & cost accounting (cache-aware, all 89 tasks)¶

Cache-read is 43% of the bill at a 97.3% cache-hit rate — as on SWE-bench, a heavily-cached agent, so the lever a compaction layer must pull is cache-read tokens.

Timeouts (7 long-horizon tasks)¶

These tasks still hit the wall-clock budget under the extended 4× timeout (up to ~4 h each) and scored reward 0 — counted as failures in the solve rate above. A large part of the cause is gateway latency, not only agent capability: Terminal-Bench's timeouts assume a fast endpoint (~2–5 s/request), but this IBM LiteLLM gateway runs ~26 s/request (5–10× slower), so long-horizon tasks that need many round-trips run out of clock (concurrency is not the cause — latency was flat ~23–30 s/req from n=1 to n=24). They are all hard/long software-engineering and compute tasks (path-tracing, a MIPS Doom port, a metacircular evaluator, COBOL modernization, GPT-2 code-golf, CIFAR training). A compaction arm that cuts round-trips could bring some under budget, so the timeout count is itself a comparison metric.

By difficulty (all 89 tasks; timeouts = failures)¶

By category (all 89 tasks)¶

Per-task (all 89)¶