What can and cannot be measured here¶

This page exists because the hardest part of evaluating compaction turned out not to be building it, but establishing what any given experiment is capable of showing. Several results in this repo were invalid for reasons that had nothing to do with the components — sample sizes with no power, a harness blind to a whole class of defect, and yields that measured a component's economic throttle rather than its capability.

Read this before designing an arm or believing a number.

1. Statistical power — reward at small n shows nothing¶

Reward on LOCA is binary per task, so a paired comparison uses McNemar's exact test on discordant pairs. That test is brutal at small n.

Clusters give the power; seeds make each cluster trustworthy¶

This section's own title overstates the case and this subsection is the correction. "Reward at small n shows nothing" is not what the arithmetic says. Fifteen clusters detected iteration 024's effect at p = 0.0078, and the bar is reachable: a two-sided exact sign test over 15 paired task means clears α = 0.05 at 6 better and 0 worse.

Seeds do not add clusters. Five seeds of the same fifteen tasks is still fifteen clusters, because the seeds are correlated observations of the same task. What they buy is different and it is not power: they make each cluster's mean stable. Iteration 023 measured arm C scoring 0.200 and then 0.800 on a byte-identical config, and the iteration 026 probe ran one task at 75, 32 and 16 steps across three passes. At one seed per task a cluster is close to a coin flip; at five it is a mean. That stability is what let iteration 024's clusters come out 8-and-0 rather than as noise, which is what made the sign test significant.

So the design reads: clusters set the ceiling on what can be detected, seeds decide whether the clusters are worth testing. Getting that backwards produces two specific errors, both of which have been made here:

1. Reading a null as underpowered. Iteration 025 returned 6 better, 3 worse, p = 0.5078. That is a coin, not a suppressed signal, and the cause was that the mechanism fired 34 times against iteration 024's 151 — a treatment problem, whose remedy is not more seeds. More seeds would have changed nothing.
2. Buying seeds for power. They do not supply it. If more power is needed the answer is more tasks, which means a different LOCA set or a different workload, not more repetitions of the same fifteen.

Where fifteen clusters genuinely does bind is the harm gate, and that is a property of the decision rule rather than of the sample: a Clopper-Pearson 95% upper bound on the worsened proportion is 21.8% at 0 of 15, 31.9% at 1 of 15, and 48.1% at 3 of 15. Against a 25% veto that passes only on a perfect zero — so a result of 9 better and 1 worse is significant at p = 0.0215 and vetoed anyway. Any run using that bound as a blocker should say so in advance, because the rule and not the evidence is doing the work.

Headroom: n is not the only thing that limits detection¶

Before any n calculation, ask how many tasks are able to show the effect at all. Direction matters, and the two pools are disjoint:

• Harm can only appear on tasks that currently pass. A task that already fails cannot be broken visibly — it fails either way.
• Improvement can only appear on tasks that currently fail. A task that already passes cannot pass more.

Measured pools, from the matched 15-task comparison in iteration 008:

At 64k the whole harm signal had to come from three tasks. At a 10% harm rate that is 0.3 expected visible events, which is why iteration 007 could bound harm only at ≤26% — there was almost nothing for harm to act on. Nor were the 9 failing tasks useful: they failed under format alone, which is lossless, so they fail for reasons no compaction component can influence. They cost full price and return a tie.

Why ~50% is the optimum rather than merely "better". For a binary outcome the available variance is p(1-p), maximal at p = 0.5:

For the same underlying effect, the observable signal is largest near 50%. The whole band table in section 6 is this one fact:

This bites hardest on the two-sided claim below. If deferring a wipe can raise accuracy, that needs a pool of currently-failing-but-recoverable tasks — and at a 25% base rate the large failing pool was mostly tasks failing for unrelated reasons.

The headroom limit was ours, not the benchmark's¶

Worth recording as a reasoning error, not just a result. The conclusion drawn at 64k was "LOCA lacks the headroom for this" — a claim about the benchmark. The 32k run used the identical 15 tasks, same classes and seeds, differing only in data volume (600 games / 6 teams instead of 1200 / 8), and scored 53%.

Same benchmark, same tasks, one config knob, 25% → 53%. The limitation was a property of the chosen configuration that had been generalised into a property of the benchmark. That is the same shape as the three rig artifacts in section 7 (chunked bodies, EAGAIN, the 128k "collapse"): each looked like a fact about the world and was a fact about the setup.

Standing rule: treat "the benchmark cannot do X" as a hypothesis about your own configuration until a matched run says otherwise. A matched run is cheap — this one cost $85 and saved several hundred.

(Caveat kept visible: the 64k figure is 3/12 because three tasks errored on the broken shim, so its true rate carries some uncertainty. The 25%-vs-53% gap is far too wide for that to account for.)

iteration 004b, 12 tasks:

What n=12 can detect at all:

So compaction must flip ≥6 of 12 tasks (50 percentage points) to register. Nothing smaller is detectable in either direction, which means a null result at this size is not evidence of safety.

What a realistic effect needs:

~200–350 tasks per arm. LOCA has 75, so even the full set detects only ~20pp effects.

Two consequences that keep being forgotten¶

Zero discordance is stronger evidence than a p-value. The deterministic arm's per-task outcome string was byte-identical to the baseline — 010000101010, same tasks solved, same tasks failed. That is not "no significant difference", it is "no difference", and it is the most reassuring number in this work. Still n=12, but qualitatively unlike a 4-vs-5 split.

"Reward-neutral" is a non-inferiority claim. It needs more power than showing a difference, because failing to find an effect is not finding its absence. Every reward statement here should be read as unmeasured, not confirmed.

The effect is two-sided: compaction can raise accuracy, not only risk it¶

The non-inferiority framing below is necessary but incomplete, and taking it as the whole story understates what compaction can do.

"coref can only harm accuracy" is true against a baseline that keeps the full context. That is not the baseline in any run here. LOCA's native trimmer drops whole messages once a request outgrows its limit — that is precisely why loca_repair_shim.py has to repair tool_use/tool_result pairing at all. So at any band where the trimmer fires, the baseline is itself lossy, and selective removal that keeps what is still referenced can preserve more usable context than a blunt drop. That is a mechanism for coref to be better, not merely not-worse.

The same argument applies to summarization, and that is the core of the deferral claim. Replacing a blunt context wipe with summarize does not remove the phenomenon — a summary is lossy too, and lossy in a way that is arguably worse for co-reference: a trim removes messages wholesale, so what is gone is at least knowable, whereas a summary paraphrases exact identifiers into prose, silently corrupting the literal tokens Tier-1 matching depends on. Summarizing early therefore destroys exactly what selective removal would have kept, which is why deferring it is expected to help rather than merely cost less.

Two consequences for design:

1. Use a two-sided test, not a one-sided harm bound. A two-sided test has more power at the same n, and a one-sided non-inferiority frame cannot register a gain even when one occurs.
2. The baseline arm must be named precisely. "vs baseline" is ambiguous between full context (where only harm is possible) and lossy default trim or summary (where gain is possible). These are different experiments with different expected signs, and conflating them makes any result uninterpretable.

The affordable version of the claim, priced¶

Superiority on a binary reward is the expensive claim; bounding harm is the cheap one, and is usually what is actually being asked. Priced below at $7.59 per task per arm, which was wrong by 5× — the true figure is $1.52 per run (iteration 010 amendment 1). Divide every dollar figure in this table by ~5; the ratios between rows, which are the point, are unaffected:

Compare the superiority cost at the observed ~10% discordance rate: 119 pairs at a realistic 80/20 split, i.e. $1,800, to detect an effect nobody claimed. The margin must be declared before the run and the budget chosen to buy it — otherwise the run silently purchases the top row.

Contrast: token measurements are near-exact¶

Component savings (format at 92–99% of total, extract_llm 38 firings/494k tokens, coref 61/638k) are counts over a fixed request set from deterministic components, not sampled estimates. They need no significance testing. The asymmetry is the point: we know precisely what is removed and almost nothing about what it costs.

2. Replay is not validation¶

The /compact endpoint runs a pipeline and returns the rewritten body without forwarding it upstream. No provider ever validates it. So replay can tell you what a component removed and is structurally incapable of telling you whether the result is a sendable request.

This blind spot hid three provider-rejecting defects in summarize (iteration 005), each masked by the previous one, found only by sending live traffic one at a time. It silently covered every /compact-based result here: density, the eval-box pass, component gating, and iteration 002 — whose deferral figure came from a pipeline that 400s in production.

Mitigated, not solved. schema.ValidateShape + the all-presets test now catch this class statically in 0.07s (verified against the real defects). Live runs remain the other half, but no longer carry the whole burden.

3. Eligible ≠ acted — yield measures the throttle, not the capability¶

coref acts only after clearing four economic gates, all on by default:

So a reported yield is "what coref could remove and justify", never "what coref could remove". Three numbers must be reported separately or the result is uninterpretable:

• eligible mass — from class_* gates: what the index judged dead
• acted mass — what survived all four gates
• refused-for-economics — batch_too_small + break_even + rewrite_budget

Why it changes the conclusion. If acted-yield is negligible but refused-for-economics is large, the finding is not "coref does not work" — it is "coref is economically throttled". Those demand opposite responses, and without the gate breakdown they are indistinguishable.

4. Break-even is sometimes vacuous¶

S × T > 11.5 × W prices a cache-write the mutation causes. When the provider's cache has already expired, the prefix is rewritten regardless — the mutation costs nothing incremental and needs no break-even test at all. Those moments arise whenever the inter-turn gap exceeds the cache TTL: a slow tool, a queue, a human thinking.

Combined with §3: coref's ceiling is set by break-even, and break-even is sometimes wrong. Its real capability is higher than any measured yield, by exactly the mass refused for economics at moments when the rewrite was already free.

Measured on LOCA (deploy/harbor/cache_opportunity.py, from per-step cache_read_input_tokens): zero such moments — LOCA drives local mock MCP servers, so turns land seconds apart and a 5-minute TTL never lapses. A property of the benchmark, not an answer. See the proposal's design notes (docs/proposals/coref-compaction.md).

5. Cost model, measured¶

Per-task cost at the 64k band, from real runs — not estimates:

Deterministic arms cost less than the baseline — 31% less in one measured case — because compaction reduces the agent's own token bill and makes no model calls. coref is deterministic, so only the two extract_llm arms carry LLM spend.

Six arms ≈ $690 at k=1. Earlier I quoted "$315/arm", which was actually the three-arm total and ignored that deterministic arms undercut the control.

But removing tokens is not saving money. The prefix-reach arm removed 20.3% and cost more than baseline ($22.64 vs $21.34): model calls plus pipeline overhead outweighed the saving.

6. Benchmark suitability¶

The unclosable gap: nothing is both long-context and Tier-1-rich. LOCA is long but Tier-2/3-heavy (adverse to an exact-match detector); SWE-bench is Tier-1-rich but short. So a null result on LOCA cannot be generalised, and that limit should be stated in advance rather than discovered.

LOCA band behaviour — three of these five rows were never actually measured:

All three bad rows were my rig, not the benchmark. The 32k and 96k probes died with [Errno 11] write could not complete without blocking — the EAGAIN pipe trap in the table below — with messages: 0, i.e. before the agent ever ran. The 128k row was collected while repair_shim.py was silently dropping chunked bodies, so its zero floor is not established.

Consequence for design: "64k is the only band with pressure and headroom" was never tested against its neighbours. Solve rate falls from 1.0 at 8k to 0.20 at 64k, so an intermediate band plausibly has both, and the headroom problem in section 1 may be an artifact of having picked 64k. Measuring 32k on the fixed rig is the cheapest available move, and should precede any decision to buy more pairs at 64k or to move the agent to a larger model.

Two properties of LOCA that cap what any reward arm here can conclude¶

~~group_by_seed silently divides your n by five.~~ FALSE — and the real trap is the opposite one. It defaults to True and is genuinely not exposed as a CLI flag, but it groups for reporting only: all 75 configs execute either way. The actual trap is on the reading side — state0…state4 are the 5 seeds, so globbing tasks/*/state0/eval.json silently reads 15 of 75 completed runs and overstates per-run cost by 5×. That is what iterations 007 and 008 did (iteration 010 amendment 1).

What actually limits reward power here is 15 independent tasks, not n. 75 runs are 5 seeds of 15 tasks, so they are clustered: extra seeds buy precision within a task and do not add independent observations. Any harm bound must be quoted on the task-clustered end (~≤18% at zero events), and tightening it needs more distinct tasks, which LOCA's 32k set does not have.

The 64k base solve rate is ~33%, and accuracy is binary. (Measured over all 75 runs; the "20%" first reported came from reading state0 only.) Every accuracy value is exactly 0.0 or 1.0, so there is no partial-credit signal to recover power from. A 20% ceiling means most tasks fail for reasons no context-management component can affect — they cannot register improvement or degradation, so they consume budget while contributing nothing but a tie. Trajectories were 9–53 tool calls, well short of the ~106 ceiling, and the agent terminated on its own: these are genuine task failures, not truncation.

Together these are why iteration 007 was stopped rather than completed.

7. Rig traps that produced valid-looking wrong numbers¶

Each of these yielded a numerically plausible result with a broken cause. Check them before believing any benchmark output.

The meta-lesson: a broken environment produces a plausible transcript and a real-looking score. Before believing a benchmark number, confirm the tools worked — per-task eval.json, gate counters, and a transparency assertion on the off arm (0 saved, no component acted).

See also: the proposal (docs/proposals/coref-compaction.md) · experiment log · selection experiment

The benchmark's compaction never fired, on any iteration, because the gateway does not forward it¶

The most expensive rig trap recorded here, because it silently invalidates the word "band" in every LOCA iteration from 008 onward.

Every LOCA run is launched with --use-clear-tool-uses --clear-trigger-tokens <band>, and each iteration is indexed by that band ("32k band", "64k band"). The intent is that the agent behaves as if it had that window, compacting as it approaches. clear_tool_uses is not client-side: it is an Anthropic API beta (clear_tool_uses_20250919) sent as a context_management request parameter and applied server-side.

It never applied. Measured 2026-09-09 against the benchmark gateway:

And the drop is localised to the gateway, not to our proxy and not to the model:

• context-guru preserves it. A request carrying context_management plus the beta header, put through the proxy with an echo upstream, arrives upstream with the parameter and the header byte-intact — the "byte-lossless splice" holds for a field no Go code in this repo has ever heard of.
• The gateway shape-checks it and then discards it. A malformed shape is rejected with context_management: Input does not match the expected shape. Received Model Group=…, which is litellm's error format, not Anthropic's. But a bogus edit type (this_is_not_a_real_edit_type) returns 200 — and Anthropic would reject an unknown member of a discriminated union. So the edits are never seen by the API, and the {"applied_edits": []} echoed in every response is synthesised gateway-side.

Consequences, which reach past one iteration. Nothing enforced the band on any run. Iteration 025 measured requests to 612,290 tokens against a declared 64,000 — 9.6x — with 50% of all sweep decisions taken past the window, where estimateTurnsRemaining returns 0 and the econ trigger cannot authorise anything. The pressure regime in every iteration was whatever the tasks happened to produce, not what the design selected, and a declared window is a scale for computing pressure, never a constraint.

What was tried and did not fix it. --use-clear-thinking was the leading hypothesis: thinking blocks are never reclaimed by default (Clear thinking: False), so once that mass passes the trigger, clearing tool uses cannot get back under it. A paired 4-task probe with the flag on ($22.18 control, $30.71 treatment) moved the share of decisions past the band from 54% to 55% — nothing — and per-session shrink detection found 1 real shrink in 97 consecutive-turn transitions. The flag was not the problem because no clearing was reaching the API at all.

How to actually hold a band, in preference order: run the benchmark against an endpoint that forwards context management; or implement clearing client-side in the harness; or accept that the band cannot be enforced and choose it to fit the workload rather than to constrain it — at a 64k declared band the median request was 63,687, i.e. the threshold sat exactly on the median, which is the worst available choice. Whatever is chosen, report the share of decisions that actually fell inside the band before treating the band as the independent variable.

A window that failed to resolve became a confident 1,000,000, and nothing said so¶

The trap that voided iteration 024 outright, and it is a different one from the clear_tool_uses failure above — that broke the agent's compaction, this broke the proxy's view of the window.

The rig serves the band as a model-window document over HTTP and points the proxy at it with MODEL_INFO_URL. Iteration 024's file was present, correct, and 64,000. The proxy resolved 1,000,000 on all 2,207 requests, across ten passes and six hours.

Three independent silences composed into it:

The rig contributed the fourth: pgrep -f "http.server 6980" || start one adopts any listener on that port regardless of which directory it serves, so a stale server from another session serves 404s for a file that exists.

What it cost, and why nothing caught it. Every fraction-based threshold in the pipeline was evaluated against a window nobody configured:

• summarize's trigger, 0.78, became 780,000. The largest request either arm produced was 345,996, so summarize never fired once — 4,015 invocations, all declined, in both arms. Iteration 024 had no compaction of any kind.
• the econ trigger's horizon is linear in the window, so it came out ~16x too long: haveTurns never dropped below 32 and ran to 11,611. It authorised 626 asks. Repriced at 64,000, 74 of its 203 real-horizon firings still repay — the gate was never in a position to decline.
• the run completed with cost_source: component, healthy counters, and a full results table. Nothing in the artifact said the configuration was not in effect.

The lesson is not "check your URLs". It is that a fallback which converts "I cannot resolve this" into a plausible number is worse than one that fails, because the failure mode has no symptom. Four fixes, in descending order of value:

1. model_info_unresolved in /stats — the run's own artifact now says whether its thresholds meant anything. This is the cheapest possible detection: one field in a file already collected per pass.
2. cg_model_info_unresolved_total, alertable, for a deployment rather than a benchmark.
3. An explicitly-configured MODEL_INFO_URL is probed synchronously at startup and the proxy refuses to run if it does not load, matching the treatment MODEL_PRICES already had. Iteration 024 would not have started.
4. The rig asserts the RESOLVED window, off the proxy's own log after traffic, not just the file's reachability — a document that fetches fine but does not name the run's model id falls through to the fallback just as silently.

The general form, worth stating separately because it recurs: any threshold expressed as a fraction has a denominator that can be silently wrong, and a wrong denominator produces a component that looks configured, runs, and measures nothing. Prefer an assertion on the resolved value over one on the input that was supposed to produce it.

A long transcript is not a large request, and picking a fixture on the wrong axis manufactures a null¶

Caught before it cost anything, on iteration 027's mechanism probe, and recorded because the mistake is natural and the failure is a false negative — the most expensive kind, because a null result ends an investigation.

The probe had to show whether a size-gated component (min_pressure: 0.20, i.e. 12,800 tokens of a 64k band) fires at all. The two tasks first chosen were "the cheapest long transcripts": 51 and 27 steps. Their mean request sizes are 8,865 and 7,827 tokens. Neither reaches the floor. The probe would have fired nothing, and the pre-registered stop rule for "fires nothing" is stop.

Step count and request size are nearly uncorrelated on LOCA. Measured across iteration 024's five seeds:

A task can run fifty steps against small requests (many short tool results) or twelve against enormous ones (a few large ones). Both are "long conversations"; only the second exercises a fraction-of-window gate.

The general rule: select the fixture on the axis the mechanism is gated on, not on a proxy that feels related. Where a gate reads a request's token mass, rank candidate tasks by measured mean request size before anything else — and where no such measurement exists yet, that measurement is the first thing the probe should produce, before any decision is read from it.

Untracked scratch tooling is the least-reviewed code in the measurement path¶

repair_shim.py — an ~80-line HTTP hop between LOCA and the gateway, living only in /tmp on the eval box, never committed, never reviewed, never tested — silently dropped the body of every request the client happened to send with Transfer-Encoding: chunked, and forwarded chunked together with Content-Length: 0. Result: intermittent 400 Bad request HTML that was attributed first to a component under test and then to the benchmark, across two iterations.

Two structural lessons, not one:

1. Error counts that move opposite to the amount of machinery indict the harness. The arm running format+coref produced one such error; the arm running only format produced three, and succeeded on none of the three tasks the other arm passed. No component-caused defect has that shape. That signal was in the data before the root cause was found.
2. A permissive component will hide a protocol violation until it reaches a strict one. A local Python echo server accepted the contradictory framing with a 200. Only the real gateway rejected it. Testing the shim against a lenient stand-in would have "passed."

The shim is scratch tooling by necessity (it repairs bodies for replay), but everything in the measurement path deserves the same suspicion as the thing being measured — and it is the code not in the repo that gets none.