SWE-bench Verified: per-arm detail¶

These are the per-arm result breakdowns for the SWE-bench Verified run: baseline, context-guru, headroom, rtk, and observe mode.

Full results — baseline (off) (SWE-bench Verified, 50 tasks)¶

Live through the harness, claude-code agent on aws/claude-sonnet-5. Cache-aware billed input cost (fresh $2/M · cache-read $0.20/M · cache-write $2.50/M) + output $10/M, recomputed from each trial's token tiers. See REPRODUCE.md.

Totals¶

Per-task¶

Full results — context-guru codesmart (final) (SWE-bench Verified, 50 tasks)¶

Live through the harness, claude-code agent on aws/claude-sonnet-5. Cache-aware billed input cost (fresh $2/M · cache-read $0.20/M · cache-write $2.50/M) + output $10/M, recomputed from each trial's token tiers. See REPRODUCE.md.

Totals¶

Context-guru proxy savings: 1.09% content; own LLM cost $0.3071; added latency/req 116.9 ms; expand bounces 4.

Per-component tokens removed (cumulative): extract_llm 129,966, extract 34,293, dedup 1,120, cmdfilter 6

The pipeline that produced these numbers is not today's codesmart: it had no toon, used cacheinject rather than cachesplit, and failed_run removed nothing because of a gating bug (since fixed), which is why it is absent from the per-component line above. See Reproduce the results for what a re-run today would execute.

Per-task¶

Full results — headroom (hd-cache) (SWE-bench Verified, 50 tasks)¶

Live through the harness, claude-code agent on aws/claude-sonnet-5. Cache-aware billed input cost (fresh $2/M · cache-read $0.20/M · cache-write $2.50/M) + output $10/M, recomputed from each trial's token tiers. See REPRODUCE.md.

Totals¶

Per-task¶

Full results — rtk (Rust Token Killer) (SWE-bench Verified, 50 tasks)¶

Live through the harness, claude-code agent on aws/claude-sonnet-5. Cache-aware billed input cost (fresh $2/M · cache-read $0.20/M · cache-write $2.50/M) + output $10/M, recomputed from each trial's token tiers. See REPRODUCE.md.

Totals¶

vs the no-compaction baseline ($31.98 / 43 solved / 36.1 steps): rtk is −9.0% billed cost, reward-neutral (43 = 43), −8% steps, at zero request-path latency and $0 tool cost. It is the 2nd-cheapest of the four arms — cheaper than headroom (−5.3%), behind context-guru (−13.2%). Full four-way table: comparison.md.

What rtk did (its own ledger)¶

rtk is a Claude Code PreToolUse hook that rewrites Bash commands in-container (pytest→rtk pytest, cat→rtk read, git status→rtk git status, …) and compresses the command output at the shell, before it enters the transcript. Because the compressed form is what gets cached from the first turn, rtk never mutates already-cached content — it sidesteps cache invalidation by construction (lowest cache-write of the four arms, 1.83M).

Across the 50 clean trials it rewrote 637 bash commands and removed 338k bash-output tokens (65.8% of bash output, its own bytes/4 estimate). Note the denominator: that 65.8% is of bash output only — a small slice of a ~98%-cached agent's total context — which is why the end-to-end saving is −9% billed cost, not −66%. (This is rtk's own documented caveat: it cuts bash output, not the bill, one-to-one.)

Savings concentrate in file reads via cat (62%) and grep (24%, the most-invoked command). Claude Code's built-in Read/Grep/Glob tools bypass the Bash hook, so rtk only sees output the agent routed through Bash — this is the structural ceiling on its reach.

Real before → after (rtk is deterministic; these reproduce its in-container behavior)¶

cat a source file → rtk read -l aggressive (332 B → 164 B): keeps imports + every signature, elides bodies.

import os
import sys
def alpha(a, b):
    // ... implementation
def beta(x):
    // ... implementation
class Widget:
    def render(self):
    // ... implementation

a failing pytest run → rtk test (1,055 B → 195 B, ~81%): failures + summary kept, 213 passing lines dropped.

[FAIL] FAILURES:
  FAILED tests/test_utils.py::test_parse_edge_case - AssertionError: assert None...
SUMMARY:
  ======================== 1 failed, 213 passed in 4.21s =========================

Per-task¶

Results — observe mode¶

Live through the harness, claude-code agent on aws/claude-sonnet-5, codesmart pipeline, cache-aware billed cost (fresh $2/M · cache-read $0.20/M · cache-write $2.50/M · output $10/M) recomputed from each trial's token tiers. See REPRODUCE.md.

Scale caveat, stated up front. 2 SWE-bench tasks and 2 Terminal-Bench tasks per mode at n=1, plus one real Claude Code session per mode. Enough to answer the two questions observe mode has to answer — does it add latency, and do its projections mean anything — and nowhere near enough for a cost or solve-rate claim. The 50-task arms in the other results pages are the ones to cite for savings.

Does observe add latency to the enforced path?¶

No.

Four orders of magnitude on SWE-bench, and it is structural rather than tuned: the request path never runs the pipeline, so the only cost is copying the body and an enqueue.

Observe is not free in other respects — it moved 75.0 s of compaction off-path on SWE-bench and spent $0.0779 of cheap-model tokens doing the measuring. It costs money and CPU, just not request latency, and observe_llm_notice labels that spend for what it is.

Do observe's projections match what sync actually achieved?¶

This is the question that validates the mode. Three independent lines of evidence, in descending order of strength:

1. Controlled same-traffic comparison — exact agreement¶

The same five turns driven through the real handler under each mode:

sync:    before=43445  saved=10020  (23.06%)
observe: baseline=43445 potential=10020 (23.06%)

Identical. A test pins this, and it fails at ratio 0.33 if observe's own store is removed.

2. Terminal-Bench — correct agreement near zero (negative control)¶

On traffic where sync achieves almost nothing, observe correctly projects almost nothing rather than inventing a headline. This is the more convincing shape of the evidence: a mode that only ever agreed on high-savings traffic would be much weaker proof that its projections mean anything. It also correctly reported the overhead sync would have added as 9.1 ms/req — small here because the pipeline made no model calls on these tasks.

3. SWE-bench arms — consistent, but too noisy to confirm independently¶

6.40% projected against 0.82% enforced. The gap is not explained away:

• the arms are different agent trajectories (22.5 vs 15.5 mean steps) — observe saw 46 requests and 492,652 baseline tokens, sync saw 35 and 244,319. These are not the same conversations.
• observe's projection never pays a bounce. Nothing is offloaded, so no expand round trip can claw savings back and wasted_tokens is structurally 0. Under sync some savings do come back. Observe's projection is an upper bound on content savings, documented as one.
• 2 tasks at n=1 cannot separate a real bias from trajectory noise.

A 50-task paired run is the honest next step for this line specifically.

Two bugs this question found¶

Answering it honestly was the most valuable thing the benchmark did, because the first comparison was wrong twice, in opposite directions:

1. 11x overstatement. The observe job ran without the session tracker, so its cached-prefix boundary was unknown, the tail gate never fired, and 50 extract_llm candidates passed where sync allowed 5 — 9.53% projected against 0.82% enforced. A projection that ignores cache-awareness projects what a cache-blind proxy would do and overstates by exactly what cache-awareness costs.
2. 3x understatement. Fixing that exposed the opposite error: observe ran against a discarded buffer and so lost the frozen decisions offloaders replay on every later turn — where most of the sustained saving lives.

Both are fixed, both have a test that fails without its fix, and the exact agreement in §1 is the result.

Namespace separation, verified in production¶

From the SWE-bench observe arm's live /stats:

• enforced: requests: 0, saved_tokens: 0, sync_enforced: 0, components: {} — all zero, all empty;
• hypothetical: observe_hypothetical_requests: 46, actual_baseline_tokens: 492652, projected_optimized_tokens: 461112, potential_saved_tokens: 31540, potential_components: {…} — fully populated.

No aggregate over the enforced savings rollups can reach a hypothetical, because they are different accumulators with disjoint serialized names.

Live Claude Code sessions¶

Same prompt and workspace through each mode against the live gateway:

Observe's actual_baseline_tokens = 6,025 is exactly sync's tokens_before = 6,025 on the same prompt — the hypothetical namespace accounts for identical traffic identically, measured independently. And requests: 0 with every enforced savings aggregate at zero is the machine-readable form of "context-guru did not modify anything".

What is not established here¶

• Any cost or solve-rate claim per mode. 2 tasks at n=1; the billed-cost figures track trajectory length far more than they track mode.
• Whether observe's projection matches sync on large-savings traffic at scale. §1 shows exact agreement on controlled traffic and §2 correct agreement near zero; the SWE-bench arms are too small and too differently-shaped to confirm the middle of that range.
• The off-path queue under pressure: dropped was 0 on every arm, so that path is exercised only by tests, never yet by production load.