Keep an idle prompt cache warm¶

The provider's prompt cache has a five-minute lifetime. A session idle longer than that loses its whole cached prefix, and the next turn re-bills every token of it at the cache-creation rate.

On this service, measured over 19,805 requests and $3,139.97 of spend:

A TTL-expired request costs 8.5x a request that hit, and against each row's own counterfactual the penalty is 11.35x, of which 91.2% is pure re-write. $584.83 — 18.6% of all spend — sits in misses whose idle gap was under an hour.

What the keep-alive does¶

Two sentences from the provider's documentation make this fixable:

By default, the cache has a 5-minute lifetime. The cache is refreshed for no additional cost each time the cached content is used.

The lifetime is measured from the start of the request that writes or reads the cache entry, not from the end of its response.

So a cache read refreshes the lifetime, and a read costs 0.1x base input where re-creating the prefix costs 1.25x. After a session has been idle for X seconds, the proxy re-sends that session's last request with max_tokens: 1 — a pure cache read, which refreshes the entry. One ping buys back about 11.5 of itself.

Nothing else about the request changes. The prefix hash is cumulative over tools → system → messages, so a ping that altered any of it would create an entry at 1.25x instead of refreshing one at 0.1x — the one way this mechanism costs money instead of saving it. Only max_tokens and stream are touched, and neither is inside the hashed prefix.

Turning it on¶

It is off by default, because it spends the caller's own money without the caller asking. On the Settings page: "Keep my prompt cache warm while I am away." In a configuration document:

cache:
  keepalive: true
  keepalive_idle_seconds: 280          # X — must be under 300
  keepalive_max_pings: 2               # K
  keepalive_min_prefix_tokens: 20000   # the gate
  keepalive_max_usd_per_ping: 0.25

These are the measured optimum, not defaults picked by feel. Replayed through the shipped decision function over the production window:

The gate is the whole reason this is deployable: 9.8× fewer pings for 1.7% less money. What it drops is the near-free pings on tiny prefixes — ping cost is bimodal, p50 $0.0004 against a p99 of $0.2275 — so 8,000 requests of real gateway load disappear and almost none of the value does. On a path that already returned 180 HTTP 429s in the same window, that matters as much as the dollars.

X = 280 beats 240 because it wastes fewer pings on gaps that would have hit anyway. K peaks at 2–3 and falls away after: a session that has ended looks exactly like one that is thinking, so every additional ping is spent partly on sessions that will never come back. K=2 ships because K=2 and K=3 are within noise here while K=2 sends 26% fewer requests.

Never use a per-session ping cap. It truncates exactly the long large-prefix sessions holding the value: capping the window's pings per session at 20 drops the net to $92.34, and at 10 to $54.04. The guard that works is the per-ping cost budget above.

What it will not do¶

• It will not ping a session with thinking.type: enabled. Such a request requires max_tokens above the thinking budget (32,000 on this traffic), so the ping cannot be made cheap, and altering the thinking parameters invalidates the message-level prefix. adaptive is unaffected, and is 81% of the addressable traffic.
• It will not crowd out a real request. A ping only ever uses a tenant's slack — a quarter of the rate and concurrency budget is reserved — and is skipped rather than queued when there is none.
• It will not retry. A failed ping is logged and forgotten. A 4xx, or a ping whose usage reports more written than read, stops that session being pinged again.

Where the money shows up¶

Every ping is a dashboard row marked keepalive, with its own tokens, its own cost, and its own session — so an account can audit what was spent on its behalf while nobody was at the keyboard. The Overview ledger carries both halves:

Ping rows are excluded from every other aggregate (Filter.WithKeepAlive). A ping is a request we made on the user's behalf, not traffic their agent sent, so counting it would inflate the request count and drag every per-request average towards a one-token response. They are still listed on the Requests tab, flagged, because that list is the audit trail. Spend figures do include them: the money was really spent on the caller's key.

The saving is credited only where a ping of ours preceded the request, the gap exceeded the provider's lifetime, the provider read more than it wrote, and the amount is capped at the tokens that ping actually refreshed. It is a ceiling, not a floor: the provider's cache is keyed on content, so another session sending the same prefix would have refreshed the entry for free.

The honest downside¶

The policy is a very small tax on almost every session it touches, funding a large rebate for a few dozen. Over the production window:

The gate does not make the tax rarer per session touched — it makes it fall on 119 sessions instead of 3,809, leaving the other 3,690 untouched entirely. That is the fairness argument, and it is independent of the dollars: a session that is never pinged cannot lose.

Sessions that lose are short ones and abandoned ones. Sessions that win are long ones with large prefixes and human-paced gaps. An account whose work is many brief sessions should leave this off. Given this shape, the opt-in default and the per-account ledger are not optional: a user has to be able to see that they are one of the ~70% paying the tax rather than one of the few collecting the rebate, and turn it off.

Operator kill switch¶

CONTEXT_GURU_KEEPALIVE=off

Set in the environment, read at startup. An environment variable rather than a control-plane route on purpose: the thing an operator needs at 3am is one that works when the control plane is what is broken. /stats publishes keepalive with pings, skipped, failed, wrote_instead_of_read and spend_usd; wrote_instead_of_read above zero is a bug, not an expense.

What is held in memory, and for how long¶

To ping a session the proxy keeps that session's last request body, and — where the upstream is caller-pays, which is the default — the caller's own provider credential. There is no alternative: the ping happens when no request is in flight, so nothing else can authenticate it or supply the byte-identical prefix a cache read requires, and pinging on the operator's key would bill the wrong party.

The body is conversation content, and enabling the keep-alive is what consents to holding it. It is up to 8 MiB of that session's last request — system prompt, tools, and the message history — held for the length of the idle gap. This is deliberately not covered by the transcript-storage consent, which is a different promise about a different thing: that one governs writing content to disk, where a manager can read it and a retention window applies. This is memory only, masked, overwritten on release, never persisted, and gone within about 14 minutes. It cannot be reduced further — the prefix hash covers the whole tools → system → messages sequence, so a ping that trimmed the body would miss the cache and cost 12.5x instead of saving 11.5x, which is the mechanism inverted.

Seven controls, because this is the one place the service holds a caller's credential beyond the life of a request:

Bounded on volume too: 128 MiB across at most 512 sessions, and a single body over 8 MiB is not held.

What the masking does and does not buy. A heap or core dump, a crash report, a /proc/<pid>/mem read or a strings pass over a snapshot yields masked bytes rather than a working key — the accidental-capture class, which is the realistic one. It does not stop an attacker with code execution in this process: the mask is in the same address space. It is obfuscation at rest, not encryption. And the credential is necessarily plaintext for the duration of the ping itself, because net/http's header map holds strings and a Go string cannot be overwritten — so the window is one request instead of the whole idle hold, which is the part that was worth closing.

Gating pings on stop_reason (opt-in, per strategy)¶

Every ping above fires unconditionally — pingable() never asks why the previous turn ended, deliberately, because excluding end_turn (the naive reading of "the turn looks over") would exclude 83.7% of the recoverable dollars. What that decision never added is the other half: docs/results/kv-ttl-predictor-features.md and docs/results/kv-ttl-predictor-arms.md measured that tool_use/stop_sequence-preceded gaps land in the addressable 5m–1h band only 0.0–0.6% of the time (20x under the ~8% one-ping break-even), while end_turn/max_tokens/refusal-preceded gaps land there 11.7–43.3% of the time — and the unconditional schedule pings on all of them equally.

A manager-controlled strategy (the Strategies tab; see docs/superpowers/specs/2026-08-25-keepalive-strategies-design.md for the underlying model) can now name a predictor: predictor_id: "stop-reason-gated" plus a predictor_threshold adds one more condition on top of the strategy's own windows — a ping only fires when the request that would be refreshed itself ended on end_turn, max_tokens, or refusal. Measured pooled at +1.54% vs fixed-5m (CI95 [0.60%, 2.79%]) — real, and statistically indistinguishable from a trained logistic regression on the same window, which is why it ships as a rule (kvcache.ClusterOf, a plain string-set lookup) rather than a model: the whole production-safe predictor class this repo supports is a switch of small functions like this one, never an embedded model in the ping's hot path.

Opt-in, not a fleet-wide default. A strategy with no predictor_id — which is every strategy that predates this field — behaves exactly as before. Setting one is a per- strategy decision a manager makes deliberately, the same audited, DCO'd path every other strategy field goes through. Per-tenant heterogeneity is real (0.76%–41.4% band rate even conditioned on end_turn alone) — the pooled gain in the arms study is carried almost entirely by a few tenants, so scope a gated strategy to the tenants it actually helps rather than applying it blind, until a per-tenant-tuned version exists.

Related¶

• Measuring savings — how the ledger separates the provider's cache from ours.
• apply/headttl.go — the mixed 1h/5m TTL, which is implemented, off, and worth $0 on the models this service runs. Measured live: aws/claude-haiku-4-5 grants the 1-hour tier (36,251 of 36,574 written tokens), aws/claude-sonnet-5 silently downgrades it to 5 minutes (0 of 48,212). The ttl field reaches the provider; Bedrock's 1h support covers the Claude 4.5 family, and this service's spend is Opus 5, Opus 4.8, Sonnet 5 and Opus 4.6. cache_write_1h on each row is what says when that changes.