extract_llm¶

The honest verdict¶

On a prompt-caching backend (the default for Anthropic/Bedrock traffic), extract_llm is usually not worth running, and the measurements say so plainly:

The improvement plan's original figure was ~8× underwater. That implicitly priced the saved tokens as fresh input; they were not. All 197,548 sat in the cached prefix, so they bill at the cache-read rate, and the honest ratio is 82× — an order of magnitude worse, plus the 1,592,467 ms of blocking time, which is not priced at all. A later Terminal-Bench arm that excluded extract_llm entirely re-derived this independently. The verdict below was right; it was understated.

The reason is arithmetic, not implementation quality. A request to a caching backend is ~99.95% cached, so a token removed from a cached region saves the cache-read rate ($0.30/MTok), not the fresh-input rate ($3/MTok) — a 10× haircut. An extraction call costing ~$0.012 must therefore remove a lot of tokens to break even:

The warm figures are why the component is off by default on caching backends and why Running ONLY the cold sweep is the right setting on a prompt-caching agent: the agent truncates every tool result near 30,000 characters, so the largest output that can exist is ~7,399 tokens — 5x below the warm break-even. The gate would only ever be declining. Independently confirmed on production: the same saved tokens valued at the warm cache-read rate were worth $0.017 against $0.6039 of spend, ROI 0.03x.

Where it does pay is the cold turn. On a TTL-expired turn the whole transcript is re-billed at 1.25x fresh, so a removed token is worth 12.5x more, and break-even lands inside the reachable size range — but only for content that actually compresses. See content classes.

Content classes¶

The gate used to price every candidate with one compression ratio learned across all kinds of tool output at once. Measured over 9,763 captured messages, that pooling is why it could not tell a paying call from a losing one — the classes span 23x:

JSON blobs and ANSI CLI output are 31% of the reachable token mass in the two classes that compress worst, which is the direct cause of the flat size-versus-yield relation the corpus shows (r = −0.10 between candidate size and reduction ratio). Raising a size floor selects for exactly the material that cannot pay. So the class's own measured ratio is handed to the economic gate in place of the pooled one and the existing expected-saving comparison does the rest — no new threshold and no list of banned classes. Unrecognised content keeps using the learned ratio. Refusals are counted as low_yield_content_class:<class>.

These use the measured compression ratio, and that measurement is the uncomfortable part: on real captures an accepted extraction removed only 31–254 tokens per call on outputs of 400–2,000 tokens — an actual ratio around 0.10–0.12, not the ~0.45 one might assume. The model declines to cut aggressively, and correctly so: its contract is recall-first.

Most tool outputs are nowhere near the warm break-even — in one measured Terminal-Bench capture the largest tool output was 2,053 tokens, and across 19,805 production requests the largest that exists at all is 7,399, because the agent truncates every tool result near 30,000 characters. That is why the same component wins on a non-caching backend and loses on a caching one, and why the fix is not "compress harder" but "decide per call". The economic gate makes that decision automatically, so the component is safe to leave enabled — it simply declines to spend where it cannot win.

Measured with the gate on (replay of real captures, aws/claude-haiku-4-5)¶

forced = gate off (economic_gate: false); gated = the shipped defaults. Same capture, same floor, same model. "Saved" is extract_llm's OWN savings, not the pipeline's: crediting the whole pipeline's savings to this component reports a win that does not exist. Attribution is the difference between "positive" and "negative" here, so it is worth stating twice.

Terminal-Bench capture (20 requests):

SWE-bench capture (19 requests):

Reading these:

• The gate is a strict improvement in every arm. It never loses more than the ungated behaviour and usually far less: −$0.0172 → −$0.0054 (68% less waste) on Terminal-Bench non-caching while saving more tokens, and 26 calls → 1 on SWE-bench non-caching, taking a −$0.0652 loss to break-even.
• On a caching backend the component now makes zero calls and loses nothing, because it is disabled there by default (see below). That is the honest resolution: the gate could reduce the loss but never eliminate it, so the default stops paying for it at all.
• Even on a non-caching backend the component does not clearly earn its place on these workloads — the best result is break-even, not profit. It removes only 31–254 tokens per call at ~10 s of added latency. It earns its place when outputs are genuinely large (>~1,800 tokens on a non-caching backend); these captures mostly are not.

Four defects fixed, August 2026¶

A measurement pass over real production traffic and fresh Claude Code CLI sessions found four independent reasons this component was losing money. All four are fixed; the numbers below are measured, not projected. Full context in Measured value, Aug 2026.

1. The model was writing Python, and every program was thrown away¶

On real traffic claude-haiku-4-5 returns a plausible filter program on essentially every call, and the sandbox discarded 12 of 13. Starlark is a Python subset: it has no generator expressions, and any(k in ln for k in ids) — the single most natural way to write the filter — is a syntax error.

The prompt contract now states what Starlark is not, naming the constructs models actually reach for: generator expressions, f-strings, while, try/except, set literals, sorted(key=…) closing over a mutable local. codeContract is part of PromptVersion, so results cached against the old prompt are not reused.

Measured on the same real 33,932-character file read, three runs each: before, 0/3 produced any output; after, 3/3, at 56%, 83% and 55% reduction.

2. Every failure reported the same reason¶

RunExtractionDetail collapsed a sandbox syntax rejection, a transport error and "the model never replied" into one string, "no usable program or reply". Three causes with opposite fixes — fix the prompt, retry the call, stop calling — reported identically. That is why defect 1 survived: the component looked like it was being ignored when it was being answered and then thrown away.

The reason now escapes to the call record and into the dashboard, as one slug per cause: syntax error: <the parser's message>, truncated reply: …, empty reply, program produced an empty OUTPUT, runtime error: …, model call failed: <status>, result not smaller, and acceptance check: <which check>. The first thing this bought was a status 401 in a test harness, visible immediately instead of after an afternoon; the second was the measurement below, which found the real dominant failure hiding inside the acceptance bucket.

3. The economic gate under-priced its own calls by 21–31׶

callCost modelled the prompt as preamble(1463) + min(candidate, 5000) + 200 — at most 6,663 tokens. Under context: full, which every cold sweep used to force, the rendered transcript is the prompt. Measured on production: five calls on one request each sent ~138,000 prompt tokens. For a 3,433-token candidate the gate weighed an expected saving of $0.0077 against an estimated cost of $0.0046 and allowed the call; it cost $0.1422 and removed nothing.

The true figure was already computed two lines from the gate, as promptOverhead, and discarded. The gate now receives it, and analyticBaseline scales with it so the observed-cost blend stays comparable.

4. Our own calls were not being prompt-cached¶

Five calls on one request with cache_read = 0 and cache_write = 0 on all five: the same ~138k transcript sent fresh, five times over. The conversation context is computed once per request and is identical for every candidate in it — but it sat in the user message, which is not cacheable.

It is now a trailing system block, inside the prefix CompleteBlocks marks — but only when the request will make more than one call (Cfg.CacheContext, set from the final candidate count). That condition is the design, not an optimisation: a cache write costs 1.25× fresh, so paying it for a single call is a 25% loss with nothing to read it back.

It also lifts the prefix over the model's minimum cacheable size. The invariant preamble alone is 1,893 o200k tokens, below claude-haiku-4-5's 4,096-token floor — so on haiku nothing was being cached at all, breakpoint or not.

4b. …and the floor itself compared unlike units¶

claude-haiku-4-5's 4,096-token minimum is in the provider's tokens. The gate compared it against internal/tokens' o200k_base count. That is a unit error, and it silently declined a cache the provider was willing to grant: measured 2026-08-19, a 3,673 o200k prefix billed 4,217 input tokens and cached perfectly, while 3,673 < 4,096 withheld the breakpoint.

The comparison now converts first (cheapmodel.minCacheableO200k, 4,096 / 1.20 = 3,413 o200k, using the markup measured on haiku itself). Raw usage from the gateway, same prefix, three consecutive calls — this is the whole of the fix's effect:

model=claude-haiku-4-5  prefix=3673 o200k  new floor=3413 o200k  (old gate compared it against 4096)
call 1  {"input_tokens":8,"cache_creation_input_tokens":4217,"cache_read_input_tokens":0,
         "cache_creation":{"ephemeral_5m_input_tokens":4217,"ephemeral_1h_input_tokens":0}}
call 2  {"input_tokens":8,"cache_creation_input_tokens":0,"cache_read_input_tokens":4217}
call 3  {"input_tokens":8,"cache_creation_input_tokens":0,"cache_read_input_tokens":4217}

Being wrong in the loose direction is nearly free — a breakpoint below the provider's real minimum is ignored, not charged (measured: input_tokens identical with and without it), and claimCacheWrite's release records that and retries. Being wrong tight forgoes the cache forever, which is what was happening.

Verified against the live gateway: first call cache_creation_input_tokens: 30007, every repeat cache_read_input_tokens: 30007 with cache_creation: 0. On a five-call request that is $0.114 → $0.038, a 67% reduction.

The one thing not fixed here¶

94% of the losing spend ($16.38 of $17.48 on the affected account) was made against this gate's own written advisory. fire_on: size demotes the economic gate to advisory and leaves the per-turn and per-session caps as the only brake. The gate was right and was overridden by configuration. The advisory is recorded per call so the counterfactual is visible, but no code change can rescue an operator from switching the brake off — if you set fire_on: size, read the economic_gate_advisory gate count and the net column.

Accuracy of the code leg, measured (August 2026)¶

Basis: 40 real tool outputs captured from Claude Code traffic (/home/vpcuser/cg-research/bench + SWE-bench proxy captures, 563–15,473 tokens each), replayed through the code leg alone — strategy: code, no deterministic fallback, rewrite: true, aggressiveness: medium, compactor claude-haiku-4-5, real gateway. Reproduce with CG_DIAG=1 CG_DIAG_CORPUS=<dir> go test ./internal/extract/ -run TestDiagCodeLegCorpus -v.

Rejections, by cause — the slugs are new, and the breakdown is the whole point:

What actually moved the number:

1. The keep-set was full of English words. HarvestIdentifiers matches any word of four or more letters, and it harvests from the agent's recent turns — which are prose. So against, including, large, exactly, credits entered both the KEEP list the model is told to preserve verbatim and the recall check that refuses any reduction dropping one. 13 of 40 calls were refused for "dropping" a common English word — a good compaction, paid for in full and thrown away because it removed a noise line containing the word "large". A reference now has to carry a mark prose does not: a separator, a digit, or an inner capital (parse_config, src/api/users.py, IndexError, sympy-1.11). This was invisible until the acceptance bucket was split, and it is the single largest lever here.
2. The Starlark contract said things that are not true. Verified against the real interpreter: %-formatting, dict.setdefault, sorted(key=…), lambda and dict comprehensions all work, and the block forbade all of them — fixed overhead on every call, pushing the model into contortions for nothing. The two forms that actually killed calls in production were not named at all: a type annotation (kept: list = [] — the recorded extract.star:4:7: got ':', want newline) and a set comprehension. Both are now named, the false bans are gone, and TestTheContractOnlyBansWhatTheSandboxActuallyRejects checks every claim in both directions against the interpreter.
3. One repair round-trip on a syntax error. The parser's message is actionable on its own, so the rejected program and the error go back for exactly one correction — without the tool output, since fixing syntax needs the program and not the data. Measured: 2 attempts, 2 recovered, ≈$0.004 marginal per attempt. It pays at the cold-turn valuation of a removed token (the cache-creation rate — $2.50/MTok on this gateway, read from the request's own rate card rather than a constant; see Economics) and is about break-even at the fresh rate; it does not pay if the removed tokens would only ever have been billed as cache reads. Kept on, bounded to one retry.
4. Truncation is no longer reported as a syntax error. A reply cut off at the output cap leaves an incomplete program that fails with whatever error the cut produced. Detected from the source (unbalanced brackets or an unterminated string — a complete program always balances), so "raise the reply budget" stops looking like "fix the prompt".

Prompt size. Example D in the aggressiveness blocks restated the source-file skeleton rule that codeContract already gives in prose (327 tok in medium, a near-verbatim second copy at 299 tok in high). Both are cut to the loop alone / one sentence: the preamble goes 1,893 → 1,798 tokens at medium and 1,973 → 1,778 at high (the shared contract grew 52 tokens for the corrected Starlark facts, so low goes 1,547 → 1,599). This buys money, not latency — measured, prompt build is 0.089 ms against a 1,812 ms p50 gateway floor on an 8-token call.

The result derives from the input, and that is now checked¶

In the shipped default (rewrite: true) nothing verified the relationship between the result and the input: the containment proof is deletion-only, MinKeepRatio is 0, and a pervasive keep-id is exempt. A paraphrase, a renumbered file read or an invented value passed, and the only real protection was that the original stays recoverable via context_guru_expand — at the cost of an agent turn.

The choice made here is not to default rewrite: false (that would refuse the column strips and elision markers the prompt asks for, and the measured deletion-only leg is a weaker compactor) but to require the result to derive from the input: the fraction of the result's bytes matchable in order against the body, with the elision markers the model was told to add excluded, must be ≥ 0.9. Measured over the corpus, every accepted reduction scored 1.00 (0.93 and 0.97 in one earlier run) — a filter deletes, it does not retype — so the floor costs 0 acceptances and 0.2 ms, while a reordered paraphrase or a fabricated value lands far below it. It is a calibration knob, not a law; the number is in minDerivationRatio.

Still unmitigated: a reduction that drops a record the agent needed next but keeps every harvested id (harm #1 — recoverable via expand, at a turn), and a cross-session cache hit derived toward a different goal (harm #10 — recoverability is gated, correctness is not re-verified).

How it works¶

For a large tool output, extract_llm asks a cheap model to write a short Starlark filter specific to that content. The program sees the full output (bounded to ~32k chars) and the recent conversation, so it can delete the exact irrelevant lines/records — and, in rewrite mode, reword or collapse spans — while keeping ids, paths, and error lines verbatim. The program runs in a sandbox (no imports/IO, step + 2s limits) and the result must pass a sanity check (non-empty, strictly smaller, required ids present); on any miss the output is left verbatim. It has RE2 regex helpers (re_sub / re_findall / re_split / re_match). JSON bodies are decoded and filtered structurally.

• Deletion-only guarantee (opt-in): set rewrite: false and the result is accepted only if it is an in-order character subsequence of the input — the model can trim anything but provably cannot fabricate, reorder, or reword. Default rewrite: true is the more powerful mode (reword / summarize allowed; ids/paths/errors still required verbatim by the sanity check).
• Model source: model.source is incoming (default — reuse the proxied request's own model + key) or config (a dedicated cheap model set via CHEAP_MODEL* env / the gateway's CheapModel). With no model available it degrades to a no-op (the deterministic extract still runs if present).
• Frozen and replayed: a reduced output is checkpointed by content hash — the same output re-sent on a later turn reuses the prior compaction (no new model call, byte-identical result → the request prefix stays KV-cache stable). This replay is where ~93% of the component's realized value comes from.

• Cache-aware skip_file_reads: tri-state. Unset = AUTO — skip line-numbered source-file dumps when the request is prompt-cached (they already bill at the cheap cache-read rate), reduce them otherwise. The economic gate now generalizes this same reasoning to every candidate.

  AUTO was not implemented until recently

  Unset used to mean "always reduce", which defeated the entire measured rationale for the flag. Live confirmation of the shape it produces: a ~7k-token Go file read went to the model, which spent 40 s on a reply that hit the output cap and saved nothing. AUTO now behaves as documented, and a cold-cache sweep is the deliberate exception — there nothing is cached and file reads are the largest mass being re-billed at the write rate.

Compaction target and how much conversation the model sees¶

Two knobs the prompt itself carries.

aggressiveness is low | medium (default) | high. Measured live on identical bodies, two samples per level:

Note the access-log row: low is explicitly told that returning the input unchanged is an acceptable answer, so on an almost-all-noise shape it may decline — and a declined call still costs its money and its seconds. low is for "I would rather pay for tokens than risk a re-read", not for "spend less".

It is taught rather than thresholded: the second system block states a target and carries three or four worked examples demonstrating it, across the shapes real traffic contains — JSON, bash and test logs, prose, and a source-file read. It changes what the model is asked for and never what is accepted: the verbatim-preservation rule, the strictly-smaller rule and (in rewrite: false) the subsequence proof are identical at every level.

context is goal | recent (default) | full, with context_messages (2) as the N for recent. The model is asked to reduce one output "toward what the agent needs next", so what it is told about the conversation is the whole basis of that judgement. goal carries the task and the latest turn; recent adds every user turn plus the last N non-tool messages, which is what puts mid-session corrections ("it must default to 30s, not 10s") in front of the model that is deciding whether the line saying so may go; full carries the entire transcript and is what a cold-cache sweep always uses.

The two system blocks are ordered general-contract-first because a provider caches a prefix: the half that is byte-identical for every account has to come first or it is shared with nobody. Both the level's text and the level itself are part of the result cache key, so switching level misses rather than replaying the previous level's answer.

Economics¶

The component only calls the LLM when expected saving > expected cost.

expected saving = tokens expected to remove
                x (1 + expected future replays)
                x per-token value       <-- cache-read rate when cache-aware, else fresh rate

expected cost   = analytic size-aware cost of one extraction call
                  (preamble + shown content + overhead, at real rates),
                  reconciled with the observed mean once calls exist

Each input is measured rather than assumed:

Every decision records a reason, visible at /stats → extract.reasons and extract.top_reason, because the first question about an expensive component is always "why did this run?". Set economic_gate: false to restore the older spend-on-size behaviour — needed only to reproduce old benchmark numbers.

Triggering¶

There is no per-workload threshold to tune. When min_tokens / trigger is unset, the component derives its own gating from context pressure and growth:

A merely growing context does not fire on every step; that was the behavior that produced 271 calls. When the context window is unknown (0) the derived logic is skipped and the configured absolute trigger applies — the same fail-open convention Trigger already uses.

min_tokens still governs when set explicitly, so existing configs keep their behavior.

The /compact endpoint resolves the context window exactly as the chat path does, so fraction-based trigger thresholds and the pressure logic behave identically in offline replay and live traffic.

Cold-cache sweep¶

The one regime where this component's economics are not in doubt.

When a session resumes after the provider's prompt-cache TTL, the cached prefix is gone and the whole transcript is re-billed as cache creation at 1.25x the fresh rate. Two things are true only on that turn: a removed token is worth 12.5x what it is worth on a warm turn (cache-write rate vs cache-read rate), and rewriting deep history is free, because there is no live cached prefix left to invalidate.

Measured on the hosted service, 1.4 days of real traffic:

TTL-expired turns were 4% of requests and 31% of spend — $1.64 each against $0.144 for a warm turn — and the shipped pipeline saved 0.015% of it (baseline_cost_usd $360.14 vs an actual $360.09).

extract_llm:
  # extract_llm_sweep, in its own block, leaves ordinary turns alone
  extract_llm_sweep:
    enabled: true
    min_tokens: 1000

On such a turn the sweep lifts the tail gate (nothing is cached, so depth is free), prices saved tokens at the cache-write rate, and escalates to the agent's own model if the transcript will not fit the extraction model's window. Every removal leaves a <<cg:HASH>> marker with a one-line summary, and the result is frozen so later warm turns replay it byte-for-byte and the new prefix stays cache-stable.

The sweep no longer forces context: full — measured before/after¶

Corrected 2026-08-20. An earlier version of this section presented the whole -$0.039 -> +$0.020 turnaround as the effect of dropping the forced context: full. Independent re-measurement on the fixed code shows full is ALSO net-positive (+$0.0119, accepting 3/3), so the keep-list fix in HarvestIdentifiers carries most of the change on its own and dropping the force adds roughly +$0.005. The before/after below spans two independent changes; do not attribute it to one.

It used to, on the argument that judging what an old message may lose requires knowing what happened since. The argument does not survive measurement, and removing that one line is the largest single change to this component's economics.

full is the whole request: at 127 turns the rendered context was 138,596 tokens against a 138,341-token request, sent once per candidate. Break-even removal at k=4 falls from 113,286 tokens (1.13x the transcript — structurally impossible) to 6,833. The counter-argument was real but is about the keep-list, not the context — a full-transcript context took acceptance from 3/4 to 0/6 because every unique token in the noise became a required identifier — and HarvestIdentifiers now reads ctxRecent explicitly, so the two are separate concerns.

Replayed on bench/cold.jsonl (9 real Claude Code requests, 3 of them verified ttl_expiry with cache_read = 0), CG_IDLE=430, compactor claude-haiku-4-5, agent model aws/claude-sonnet-5 at $2.00/MTok. Identical config on both sides; only the code differs. n=3 per arm, because this arm's LLM output is noisy enough that the shipped default's net sign flips between identical runs (bench/BASELINE.md):

The same candidate (15,473 tokens) went from a 36,686-token prompt rejected by the acceptance check on all three runs, to an 18,657-token prompt accepted on all three, removing 14,438 tokens. Halving the prompt is the smaller half of the win; the acceptance rate going 0/3 → 3/3 is the larger one, and it is the keep-list effect above, now visible because the context shrank.

context: full still exists and still means full; it is simply no longer imposed. Also measured: on real Claude Code traffic recent renders 2,734–5,681 tokens, not the 100–436 of a synthetic transcript — the last seven messages are mostly large tool results, clipped at recentContextCap. So the real prompt reduction is ~2x, not ~16x, and the R* table above is an upper bound on this corpus.

The context cache write is now earned before it is read¶

CacheContext moves the per-request conversation context into a trailing cacheable system block, and was set whenever a request had more than one candidate — but claimCacheWrite deliberately withholds the breakpoint from concurrent siblings (an entry that is only ever written costs 1.25x fresh and buys nothing), so with llmConcurrency = 4 the first call took the write slot and calls 2–4 sent no mark and paid plain fresh input for the identical context. Measured on production: five haiku calls on one request each sent ~138,000 prompt tokens with cache_read = 0 and cache_write = 0.

The sweep therefore issues its first call alone, then the rest concurrently — one writer, then readers. At T = 180k, k = 4 that moves break-even removal from 198,620 tokens to 79,088.

The cost is wall clock and the trade is deliberate. Per-call latency here is gateway queue time, not prompt size (measured: an 8-token call has a 1,812 ms p50 floor and is not faster than an 8k-token one), so serializing one call adds roughly one whole queue round — ~2–4 s at p50, 12–16 s in the tail. It is paid only on a turn whose entire transcript is re-billing at 1.25x fresh; the warm per-output path stays fully concurrent, where the extra second would buy a fraction of a cent. TestSweepEarnsTheContextCacheWriteBeforeReadingIt pins both halves.

Caching¶

Three distinct caches, easily confused:

1. Global result cache. An extraction is a context-free derived result, so it is keyed on sha256(content + prompt version + model + config fingerprint) with no session prefix — identical content in a different session reuses the reduction. A prompt-version bump, model switch or config change misses rather than serving a stale extraction. Bounded by the store's existing TTL + LRU.

2. Provider prompt cache on the extraction preamble. The 1,893-o200k invariant preamble is sent as a stable system block with a cache_control breakpoint (a leading system message on the OpenAI backend, which has no explicit breakpoints).

   Measured: this buys nothing on claude-haiku-4-5

   A breakpoint below the model's minimum cacheable prefix is silently ignored — no error, cache_creation_input_tokens: 0. That minimum is 4096 tokens on claude-haiku-4-5 and 1024 on claude-sonnet-5, against a 1,893-o200k preamble — and the floor is in the provider's tokens, so the comparison converts first (see 4b). Verified against the gateway:

   Prefix	Model	Result
   ~1.5k	claude-haiku-4-5	write=0 read=0 — inert
   ~4.5k	claude-haiku-4-5	write=5401 then read=5401 — caches
   ~1.5k	claude-sonnet-5	write=2653 then read=2653 — caches

   So with the default cheap model the split is structurally inert; it pays only when extraction runs on a larger-context model (model.source: incoming). The split ships anyway — it is free, correct, and wins where it can — but do not infer a cache win from the fact that a breakpoint was placed. Watch /stats → extract.prompt_cache_read_tokens: if it stays 0 while extract.calls climbs, the breakpoint is inert on your model.

3. The agent's own KV cache, which the component must not disturb — hence freeze-and-replay and the tail-only gate for new decisions.

Rejected: reusing the agent's cached prefix¶

Appending the extraction instruction after the agent's existing cached prefix, so that extraction reads an already-cached context. Prototyped against the live gateway and rejected. It works mechanically (the extraction turn read a 103,019-token prefix from cache with no cache-write and no prefix invalidation), but cache-read is cheap, not free, and the bill scales with the whole context:

At the ~1.7M contexts this workload reaches, and with up to 4 concurrent per-output calls per turn, that is ~$2.04/turn against ~$0.016 — the opposite of this issue's direction. Paying 1.7M cache-read tokens to answer a question about one tool output is structurally wrong regardless of rate. Two further reasons, each independently sufficient: it risks a cache-write on the agent's own prefix (11.5× a read — exactly the mistake this workstream exists to avoid), and it couples the compaction model to the agent model. Re-open only if a provider prices in-context follow-up questions at a flat rate.

Metrics¶

/stats gains an extract block (purely additive — every pre-existing field keeps its name).

Per-component, in components.extract_llm: acted counts free replays. A frozen decision re-spliced on a later turn saves tokens and costs nothing, and it landed in the same counter as the call that derived it — acted: 239 beside reapplied_same_session: 2,291 was read as 239 paid extractions. Use acted_fresh (paid work) and acted_replay (free) instead.

Both extraction components measure a saving the same way, and they did not always. extract_llm booked the candidate minus the model's PROJECTION, while the text written is the projection plus the summary segment, the marker and the recovery hint — so its figure overstated by all three, with the summary the dominant term. Since the summary is a model output, the overstatement varied per candidate rather than averaging out, and two arms of a comparison read side by side were not measuring the same thing (#195). Both components now subtract the message that was actually sent, so the row is the number of tokens the requests it counts genuinely shrank by. It is not the bill delta, for two reasons that predate that fix and are unchanged by it: RecordExtractionSaving counts each distinct compaction once, so when single-flight hands two concurrent requests the same result two messages shrink and one saving is booked; and replays feed gross_value_usd but never this row, so the tokens are fresh removals only while the dollars beside them are fresh plus replay — the same acted_fresh / acted_replay split, one column over. Take the money question to gross_value_usd and net_value_usd.

The basis matters beyond reporting: the same figure feeds the ratio tracker the economic gate spends against, so an optimistic saving argued for making more calls.

Plus, at the top level of /stats: llm_truncated — replies that stopped at the model's output cap. That is the worst outcome available, full price for zero result, and it used to be invisible because a truncated program parses as nothing, exactly like a model that declined to compact. A non-zero count means the reply budget is too small for what the model is writing.

Before → After¶

Captured live through the proxy (pipeline: [extract_llm], strategy: code, model.source: config → aws/claude-haiku-4-5, economic_gate: false to force the call). The query was "find the auth timeout error and nearby context"; the model kept the error plus a few surrounding requests and elided ~118 repetitive successful-request lines:

before:  2024 GET /users/0 200 12ms          ← 60 near-identical lines
         … 58 more …
         2024 GET /users/59 200 12ms
         ERROR auth timeout on token refresh
         2024 GET /items/0 200 8ms            ← 60 more near-identical lines
         … 59 more …

after:   2024 GET /users/58 200 12ms
         2024 GET /users/59 200 12ms
         ERROR auth timeout on token refresh
         2024 GET /items/0 200 8ms
         2024 GET /items/1 200 8ms
         [auth timeout error + context; repetitive successful requests elided]
         <<cg:923fff04ab267215>> [full output: call context_guru_expand]

Note the reduction is real and useful — the problem was never output quality, it was whether the call was worth its price on a caching backend.

Lossiness¶

Lossy but reversible — the original is stashed and recovered via context_guru_expand / GET /expand. The default rewrite: true mode is verified only to the extent that the result must DERIVE from the input (≥90% of it traceable, in order, to the body; see above) plus sanity and strictly-smaller; rewrite: false gives the full deletion-only (character-subsequence) guarantee. The one-line SUMMARY spliced next to the marker is clipped to 120 runes, because an over-long one used to abandon the whole reduction rather than truncate.

Configuration¶

Context guard¶

Every call sends one tool output in its own prompt, so the size risk here is a single prompt exceeding the extraction model's window — not a conversation that grew too long (nothing older is ever dropped, and user messages are never touched: only tool-role messages are candidates). Before each call the component checks that

(shown body + prompt overhead) × 1.65  +  4096 (reply) + 512  ≤  input limit

fits, where the shown body is the bounded head+tail the prompt actually carries (a 200k-token log still travels as a ~8k-token sample), × 1.65 covers the extraction model tokenizing the same bytes more heavily than our own o200k_base counter (measured 2026-08-19: 6,396 o200k billed as 8,222 on claude-haiku-4-5 and 10,574 on aws/claude-sonnet-5, i.e. 1.29x and 1.65x — the old 1.15 under-stated both, and this margin's only job is keeping a request inside a window, where under-counting puts a prompt on the wire the upstream may reject), and 4096 is the max_tokens the cheap clients send — most APIs bound input+output against the same window.

The input limit is resolved as data, never a constant: model_max_input_tokens if pinned → the window of the config-pinned model from internal/modelinfo's table → the host-resolved window of the proxied model when model.source: incoming → otherwise a conservative 32768. model.source: config hides the cheap model's id from the component, so it takes the conservative default; pin model_max_input_tokens if its real window is smaller (or larger, to stop the guard from declining calls it could make).

A candidate that cannot fit is left verbatim — no truncation, no dropped messages, no request on the wire — and the refusal is counted as components.extract_llm.gates.over_model_context at /stats. A non-zero count on a workload that should be compacting means the extraction model's window (or the pin) is too small for the outputs being seen.

Extraction-model pricing for the gate comes from CHEAP_MODEL_PRICE_IN, _OUT, _CACHE_WRITE, _CACHE_READ (dollars per MTok; defaults are claude-haiku-4-5 list rates).

The value of a saved token is no longer a constant either. It is read from the request's own rate card (Ctx.SelfRates, resolved from the same provider price table the dashboard uses); the agentFreshPerMTok / agentCacheReadPerMTok / agentCacheWritePerMTok literals are only the fallback for when the table said nothing. This settles a 27% disagreement that sat under every cold-turn decision: the code said a cold-turn token was worth $3.75/MTok and docs/results/measured-2026-08.md §9 said $4.75/MTok. Neither is right here — $3.75 is 1.25x claude-sonnet-4-5's $3.00 list rate and $4.75 is the opus-5-era figure, while this gateway bills aws/claude-sonnet-5 at $2.00 in / $10.00 out per MTok (derived 2026-08-19 by solving the recorded cost_usd and token-tier columns of two captured corpora simultaneously), making a cold-turn token worth $2.50/MTok. A literal cannot be right for every operator.

When it shines¶

Non-caching backends — every removed token is a direct saving at the full input rate, so the break-even is ~10× easier. Also: very large single outputs (>~12k tokens) even under caching; recurring content that amortizes one call across many replays; and novel prose/log shapes no deterministic rule anticipates — this is the only component that can compress those.

When it's inert¶

Output below the derived floor, low context pressure, suppressed by the economic gate (the common case on a caching backend), throttled out this turn, result served from the global cache, projection not smaller, or no model available.

See also: extract · Components overview · Choose a preset