Full results — context-guru (codesmart) (Terminal-Bench 2.0, 89 tasks)¶

Full per-task results for the context-guru (codesmart) arm on Terminal-Bench 2.0 (claude-code on aws/claude-sonnet-5, live). Same cache-aware cost model and 4× budget as the other arms. For the four-way analysis (cost decomposition, per-component, verdict) see the Terminal-Bench comparison; the reference arm is the baseline. See REPRODUCE.md.

Totals¶

* mean steps over the 78 completed tasks (timed-out runs are truncated). Solve rate over completed-only tasks: 57/78 = 73.1%.

Token & cost accounting (cache-aware, all 89 tasks)¶

Cache-read is 41% of the bill at a 96.9% cache-hit rate — as on SWE-bench, a heavily-cached agent, so the lever a compaction layer must pull is cache-read tokens.

Timeouts (11 long-horizon tasks)¶

These tasks still hit the wall-clock budget under the extended 4× timeout (up to ~4 h each) and scored reward 0 — counted as failures in the solve rate above. A large part of the cause is gateway latency, not only agent capability: Terminal-Bench's timeouts assume a fast endpoint (~2–5 s/request), but this IBM LiteLLM gateway runs ~26 s/request (5–10× slower), so long-horizon tasks that need many round-trips run out of clock (concurrency is not the cause — latency was flat ~23–30 s/req from n=1 to n=24). They are all hard/long software-engineering and compute tasks (path-tracing, a MIPS Doom port, a metacircular evaluator, COBOL modernization, GPT-2 code-golf, CIFAR training). A compaction arm that cuts round-trips could bring some under budget, so the timeout count is itself a comparison metric.

By difficulty (all 89 tasks; timeouts = failures)¶

By category (all 89 tasks)¶

Per-task (all 89)¶