Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Keeping the Cache Warm Pays: Keepalive Economics for Agentic Workloads

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Client-side keepalives for LLM prompt caches are individually rational and pay only inside a per-provider band bounded by eviction time and a break-even horizon.

desk verdict Measured cross-provider keepalive economics with a genuinely useful policy — but the star OpenAI saving is an upper bound, not a tested number. read the letter →

arxiv 2607.19214 v2 pith:UZUFVG7X submitted 2026-07-21 cs.DC

Maxim Khailo This is my paper
classification cs.DC
keywords promptcachingcachekeepaliveagenticworkloadsLLMinferenceevictionTTLcostoptimizationtoken-hourpricing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Agentic workloads repeatedly pause for tools or approval, long enough for LLM providers to evict the cached prompt prefix and charge full prefill price on the next request. The paper claims that a client-side keepalive—replaying the prefix on a timer—holds the cache warm through these gaps and is individually rational on every provider whose cold re-prefill is expensive enough to insure. The economics reduce to one identity: the keepalive saves only when the pause is longer than the provider's measured eviction point and shorter than Imax = τ(w/r − 1), roughly 46 minutes at Anthropic and 36 at OpenAI. It also shows that the common 30-second ping convention spends about 8x more than needed; the economical interval is the provider's own retention minus a margin. A reader should care because it converts a folklore trick into a measured, per-provider policy and warns that universal adoption will push providers to meter cache residency directly.

What carries the argument

The load-bearing object is the keepalive: re-sending the exact prompt prefix on a timer so each cached read refreshes the entry's lifetime and recency in the eviction policy. The companion identity is the break-even condition Imax = τ(w/r − 1), where r is the cached-read price ratio, w is the re-prefill price ratio, and τ is the ping interval; it converts the provider's retention curve into a paying band with a hard ceiling. The same identity shows why intervals past the provider's retention turn toxic: each ping lands on a dead cache and pays the full re-prefill it was meant to prevent.

What would settle it

Run the same 30-minute pause experiment with a 240s keepalive while many clients ping concurrently; if the follow-up request arrives cold (cached fraction below 90%) or the provider's effective retention drops so far that no interval under the documented lifetime holds warmth, the central claim is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that keepalives work everywhere measured: a 30s ping kept ≥98% of prompt tokens cached across four providers through gaps that evicted idle baselines, cutting the post-pause request cost up to 12.5x. The strategic claim is sharper: keepalive cost falls as the interval grows, so the optimum is the largest interval safely under the provider's own retention (about 240s at Anthropic and DeepSeek, 480s at OpenAI), not the 30s convention. The saving exists only inside a paying band bounded by the provider's eviction point and Imax ≈ τ(w/r − 1), about 6–46 minutes for Anthropic and 25–36 for OpenAI; measured savings at a 30-minute pause are 1.56x and 2.45x. D

Load-bearing premise

The whole argument rests on the assumption that reading a cached prefix refreshes its lifetime and recency, and that the retention curves measured at one client's load remain valid when every client runs keepalives; if either fails, the paying band and the measured savings collapse.

Editorial extensions

If this is right

  • The 30-second keepalive convention is roughly an 8x overspend; the right interval is the provider's measured retention minus a margin, about 4 minutes at Anthropic and 8 at OpenAI.
  • Keepalives pay only inside the band: below the eviction point they are waste, past Imax they exceed the re-prefill, and past the provider's retention each ping re-prefills a dead cache at full price.
  • At a 30-minute pause, measured marginal savings are 1.56x on Anthropic and 2.45x on OpenAI, with the post-pause request cost cut by up to 12.5x.
  • For DeepSeek and Google the cost band is empty: DeepSeek's re-prefill is too cheap to insure, and Google's cache never reliably evicts, so keepalives buy latency only.
  • Universal rational adoption will make the shared cache tier degrade toward first-in-first-out-of-luck, forcing providers to meter cache residency per token-hour; one provider already does.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests the same keepalive logic applies beyond tool pauses—to any scheduled idle inside the band, including human-approval waits and overnight batch gaps, provided the user accepts the per-read rent.
  • A testable extension: under fleet-wide keepalive load, effective retention curves should shorten; operators should re-measure their provider's eviction point before fixing the interval, otherwise the band may close.
  • If providers adopt per-token-hour residency pricing, the break-even formula would gain a residency-cost term, shifting the optimal interval and making speculative warmth unprofitable; that suggests operators should prefer the bounded keepalive policy even before pricing changes.
  • The per-provider interval policy suggests multi-provider agent frameworks should treat cache keepalive as a routing parameter, choosing ping intervals and providers based on measured retention rather than a single global default.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper measures prompt-cache retention and client-side keepalive efficacy across Anthropic, DeepSeek, Google, and OpenAI. It derives a cost model I_max = τ(w/r − 1), recommends pinging at the largest interval safely under each provider's measured retention (≈240 s for Anthropic/DeepSeek, ≈480 s for OpenAI) rather than the common 30 s convention, and reports that at a 30-minute pause a keepalive saves 1.56× on Anthropic and up to 2.45× on OpenAI, while DeepSeek and Google only get latency benefits. It further argues that universal rational adoption would degrade shared prompt-cache tiers and push providers toward metering cache residency.

Significance. The paper addresses a practical, timely problem and is methodologically careful: timing-integrity gates, warm-reference sanity checks, separate baseline/keepalive time blocks, per-run Fisher exact tests with Bonferroni correction, explicit threats-to-validity, and a transparent, self-contained derivation of the break-even horizon. The cross-provider retention curves and the per-provider policy prescription are useful contributions. However, the central quantitative claims are not yet established at the advertised precision: the cost model treats provider retention as deterministic, the paper's own DeepSeek data contradict that model, and the headline OpenAI saving rests on one small, unreplicated run. The errors are local and fixable, so the paper is a credible major revision rather than a rejection.

major comments (4)
  1. [§5.3, Table 4 (OpenAI τ=480s)] The claimed 2.45× OpenAI saving assumes every keepalive ping is a cached read at price ratio r. But §5.1/§5.2 describe OpenAI as a 'soft, lossy cache' with gradual decay, and the τ=480s column is a single n=6 run. Under the paper's own §7 within-run-correlation caveat, 6/6 warm cells cannot tightly bound a per-ping miss probability. With per-ping miss probability q, normalized w=1, r=0.1, three pings plus the final read have expected cost 4(q + (1−q)·0.1); break-even is q≈0.17. A miss rate of 0.1 reduces the saving to ≈1.3×, and q=0.2 makes it a loss. The paper reports no per-ping miss counts. The headline saving is therefore an upper bound, and τ*=480s for OpenAI is not established. Please add per-ping hit/miss data at candidate intervals or present a stochastic cost model with confidence intervals.
  2. [Table 4 (DeepSeek) vs. §5.3] The measured DeepSeek costs contradict the cost model in §5.3. Using Table 1's r=0.10, w=1.00, the 1800s/τ=240s cell should cost roughly 8–9 read-priced pings ≈ $0.018–0.019 per 100k tokens, below the $0.022 re-prefill, so the model predicts a small saving. Table 4 reports $0.108, i.e., a large loss. The paper does not reconcile this or report whether the pings actually hit. Either DeepSeek's effective read price on this backend is not 0.1×, or pings miss at a substantial rate. Since this model is the basis for I_max and the 'too cheap to insure' conclusion, the discrepancy must be explained; otherwise the model is not predictive for a provider where it fails.
  3. [Table 1 vs. §5.4/Abstract] There is an internal inconsistency in OpenAI's τ* and I_max. Table 1 lists OpenAI τ*≈240s and I_max≈36min, and the abstract repeats the ~36min figure. Section 5.4 and Table 4 recommend τ*≈480s for OpenAI. With τ=480s, the formula gives I_max=480·(1/0.1 −1)=4320s=72min, more than double the stated band. The paying band is a central claim; the paper must adopt one τ* for OpenAI and recompute the abstract, Table 1, and §5.3 consistently.
  4. [Table 1 (Anthropic 1-hour tier)] For the Anthropic 1-hour tier, Table 1 gives w=2.00, r=0.10, τ*≈50min, and I_max≈3.3h. But the formula I_max=τ(w/r −1) yields 19τ = 19·50min ≈15.8h, not 3.3h. This is inconsistent with the formula displayed in the same table and with its use for the 5-minute tier. Please correct the arithmetic and adjust any discussion that relies on the paid tier's break-even horizon.
minor comments (5)
  1. [Abstract] The 'up to 12.5×' reduction on post-pause request cost is simply the list-price ratio w/r (1.25/0.1). It should be labeled as a price-ratio ceiling, not a measured end-to-end saving, to avoid confusion with the whole-strategy savings reported in Table 4.
  2. [Figure 2] The retention curves are plotted as point estimates without confidence intervals. Given n=6 per cell and the paper's own 'lottery cache' characterization of Google, adding binomial CIs would materially strengthen the 'never converges' claim.
  3. [Tables 3 and 4] The n annotations differ between Table 3 (240s column n=4) and Table 4 (τ=240s n=8). Please clarify whether these are separate runs and how the per-run Fisher-exact bar was applied to the 1800s paying-band run, which appears to be reported without formal statistical tests.
  4. [§6] The 'universal adoption is rational' equilibrium and the degradation argument are flagged in §7 as extrapolations, but the Discussion would benefit from an explicit statement that the retention curves were measured at single-client load and may not describe a keepalive-saturated tier.
  5. [References] References [2] and [3] appear to point to the same Anthropic prompt-caching documentation under different URLs; please consolidate or distinguish them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the break-even horizon and paying band follow algebraically from public price ratios and independently measured retention curves; reported savings are direct measurements, not fitted predictions.

full rationale

The paper's derivation chain is self-contained. The central formula Imax = τ(w/r − 1) is derived algebraically from the stated cost model in §5.3: keeping a prefix alive through idle I costs (I/τ + 1)r per input token, while letting it die costs w once; equating the two gives I_max. The parameters r and w come from public list prices, and τ is chosen from documented TTLs via the monotonicity argument r/τ decreasing in τ. The measured eviction points in §5.2 are independent experimental inputs, not outputs of the model. The headline savings in Table 4 are direct measured median costs (baseline vs. pings + warm reqB), not values computed from the fitted model, so no fitted parameter is renamed as a prediction. The paper acknowledges practitioner work proposing keepalives and 4-minute intervals ([1], [3], [17]), but its claims rest on its own measurements and public pricing, not on those citations as load-bearing justification. The limitations in §7—within-run correlation, Google quota, backend pinning, load extrapolation—are validity caveats, not circular steps: they identify conditions under which the measured numbers may not generalize, but they do not show that any result is equivalent to its inputs by construction. No self-definitional, fitted-input-as-prediction, self-citation, uniqueness-import, or ansatz-smuggling pattern is present. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted: r and w are public list prices, tau is picked from measured retention, and I_max is derived algebraically. The load-bearing inputs are provider semantics (refresh-on-read) and the assumption that single-client retention curves extrapolate to the congested regime, which the paper itself flags rather than measures.

assumptions (4)
  • domain assumption Provider caches refresh an entry's TTL and eviction recency on every cached read.
    Table 1 caption: 'the keepalive's whole mechanism is the refresh-on-read semantics in the last row.' All measurements and the economic derivation rest on this semantics.
  • domain assumption Cached reads are billed at the documented read ratio r and re-prefill at ratio w, with no other per-ping fee or quota.
    §2 and §5.3: the cost model 'keepalive cost falls monotonically in the interval' uses only list prices and per-read billing; a metered-residency or per-account quota regime would invalidate the current policy.
  • domain assumption Retention measured at one client's load approximates retention under fleet-wide keepalive adoption.
    The paper explicitly flags this in §6 and §7: 'Retention timescales are measured at the load one measurement client generates; provider tiers under fleet-wide keepalive pressure are the subject of §6.'
  • domain assumption A warm reqB requires >=90% of prompt tokens cached; warm/cold is bimodal.
    §4: 'we report the warm rate: the fraction of valid samples whose reqB cached ≥90% of prompt tokens.' The bimodality claim is an empirical regularity of the data, not an established fact for all providers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Keeping the Cache Warm Pays: Keepalive Economics for Agentic Workloads." pith.science (2026). https://pith.science/paper/UZUFVG7X

@misc{pith2026260719214,
  author       = {Pith},
  title        = {Pith review of: Keeping the Cache Warm Pays: Keepalive Economics for Agentic Workloads},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZUFVG7X}},
  note         = {Machine review of arXiv:2607.19214}
}
read the original abstract

Frontier LLM providers cache a prompt's processed prefix so that a follow-up request sharing it pays ~10% of the input price and skips most of the prefill latency. Agentic workloads systematically destroy this benefit: the agent sends a request, runs a tool or waits for approval for minutes, and by the time the follow-up is sent the cached prefix has been evicted, so the agent pays the full prefill again. A client-side keepalive, replaying the prefix on a timer during the pause, prevents this, and it is individually rational: across Anthropic, OpenAI, Google, and DeepSeek we show that a keepalive holds the prefix warm through gaps where idle baselines are evicted, cutting the post-pause request cost by up to 12.5x. The strategic question is the ping frequency, and it has a clean answer: keepalive cost falls monotonically in the interval, so the economical choice is the largest interval safely under the provider's TTL, about 4 minutes at Anthropic's 5-minute TTL rather than the 30-second convention, and the strategy breaks even against a re-prefill at idle ~tau(w/r - 1) (~46 min for Anthropic, ~36 min for OpenAI and DeepSeek). Because the benefit is real and bounded only by each user's own bill, rational adoption is universal adoption; and since cache residency is priced per read rather than per token-hour, a keepalive-saturated tier gives LRU eviction nothing to rank. We argue this externality will push providers to meter cache residency directly, and one already does. We derive the operator's policy until then.

Figures

Figures reproduced from arXiv: 2607.19214 by the authors.

Figure 1
Figure 1. Per-provider warm rate vs. idle (fraction of valid samples whose post-idle reqB cached ≥90% of prompt tokens; 100k prefix; 40k shows the same pattern on Anthropic, DeepSeek, and OpenAI; the 1800 s points come from the paying-band run of [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Retention curve: idle baseline warm rate out to 40 minutes (no keepalive, 100k prefix). No cache evicted before its documented TTL: Anthropic falls off a cliff between 300 and 360 s (its 5-minute TTL, no grace period), DeepSeek collapses by 540–600 s, and OpenAI outlives its documented 5–10 minutes, decaying gradually to fully cold by 1800 s. Google never converges (a routing lottery, not a retention curve). A study… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 2 linked inside Pith

  1. [1]

    Aider change history (v0.53.0: cache keepalive)

    Aider AI. Aider change history (v0.53.0: cache keepalive). https://github.com/Aider-AI/aider/blob/ main/HISTORY.md, 2024. Accessed 2026

    2024

  2. [2]

    Prompt caching

    Anthropic. Prompt caching. https://docs.anthropic. com/en/docs/build-with-claude/prompt-caching,

  3. [3]

    Prompt caching: keeping the cache warm

    Anthropic. Prompt caching: keeping the cache warm. https://platform.claude.com/docs/en/ build-with-claude/prompt-caching, 2026. Accessed 2026

    2026

  4. [4]

    claude-code-cache-keepalive (plugin, with break-even analysis)

    Yujia Chen. claude-code-cache-keepalive (plugin, with break-even analysis). https://github.com/ yujiachen-y/claude-code-cache-keepalive, 2026. Ac- cessed 2026

    2026

  5. [5]

    Cacheprobe: Auditing prompt cache isolation in gateway apis.arXiv preprint arXiv:2605.30613, 2026

    Ryan Fahey. Cacheprobe: Auditing prompt cache isolation in gateway apis.arXiv preprint arXiv:2605.30613, 2026

    arXiv 2026

  6. [6]

    Prompt cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-seob Lee, et al. Prompt cache: Modular attention reuse for low-latency inference. InProceedings of Machine Learning and Systems (MLSys), 2024

    2024

  7. [7]

    Context caching

    Google. Context caching. https://ai.google.dev/ gemini-api/docs/caching, 2024. Accessed 2026

    2024

  8. [8]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM Sympo- sium on Operating Systems Principles (SOSP), 2023

    2023

Show all 21 references
  1. [9]

    Don’t break the cache: An evaluation of prompt caching for long-horizon agentic tasks.arXiv preprint arXiv:2601.06007, 2026

    EliasLumer, FaheemNizar, AkshayaJangiti, KevinFrank, Anmol Gulati, et al. Don’t break the cache: An evaluation of prompt caching for long-horizon agentic tasks.arXiv preprint arXiv:2601.06007, 2026

    2026

  2. [10]

    Prompt caching

    OpenAI. Prompt caching. https://platform.openai. com/docs/guides/prompt-caching, 2024. Accessed 2026

    2024

  3. [11]

    Feature: prompt cache keep-warm pings (is- sue #62475)

    OpenClaw. Feature: prompt cache keep-warm pings (is- sue #62475). https://github.com/openclaw/openclaw/ issues/62475, 2026. Accessed 2026

    2026

  4. [12]

    Openrouter api

    OpenRouter. Openrouter api. https://openrouter.ai/ docs, 2024. Accessed 2026

    2024

  5. [13]

    Prompt caching best practices

    OpenRouter. Prompt caching best practices. https: //openrouter.ai/docs/guides/best-practices/ prompt-caching, 2025. Accessed 2026

    2025

  6. [14]

    Prompt cache thrashing: a multi-tenant noisy- neighbor billing scenario

    Tian Pan. Prompt cache thrashing: a multi-tenant noisy- neighbor billing scenario. https://tianpan.co/blog/ 2026-04-28-prompt-cache-thrashing-multi-tenant-noisy-neighbor , April 2026. Accessed 2026

    2026

  7. [15]

    Efficiently scaling transformer inference.Proceedings of Machine Learning and Systems (MLSys), 2023

    ReinerPope, SholtoDouglas, AakankshaChowdhery, etal. Efficiently scaling transformer inference.Proceedings of Machine Learning and Systems (MLSys), 2023

    2023

  8. [16]

    Cachewise: Understanding workloads and optimizing kvcache man- agement for efficiently serving llm coding agents.arXiv preprint arXiv:2606.16824, 2026

    Shubham Tiwari, Tapan Chugh, Nash Rickert, Simon Peter, Ratul Mahajan, and Haiying Shen. Cachewise: Understanding workloads and optimizing kvcache man- agement for efficiently serving llm coding agents.arXiv preprint arXiv:2606.16824, 2026

    2026

  9. [17]

    We taught our ai agents to take coffee breaks

    Veritas Supera. We taught our ai agents to take coffee breaks. https://vsits.co/ coffee-break-cache-keepalive/, 2026. Accessed 2026

    2026

  10. [18]

    Automatic prefix caching (design document)

    vLLM project. Automatic prefix caching (design document). https://docs.vllm.ai/en/v0.9.1/design/ automatic_prefix_caching.html, 2025. Accessed 2026

    2025

  11. [19]

    Kvcache cache in the wild: Charac- terizing and optimizing kvcache reuse at a large cloud provider

    Jiahao Wang et al. Kvcache cache in the wild: Charac- terizing and optimizing kvcache reuse at a large cloud provider. InProceedings of the USENIX Annual Technical Conference (ATC), 2025

    2025

  12. [20]

    Anthropic prompt cache ttl and cost mechanics

    Brandon Wie. Anthropic prompt cache ttl and cost mechanics. https://brandonwie.dev/posts/ anthropic-prompt-cache-ttl, 2026. Accessed 2026

    2026

  13. [21]

    Batchllm: Optimizing large batched llm inference with global prefix sharing and throughput- oriented token batching.arXiv preprint arXiv:2412.03594, 2024

    Zhen Zheng et al. Batchllm: Optimizing large batched llm inference with global prefix sharing and throughput- oriented token batching.arXiv preprint arXiv:2412.03594, 2024. 7

    2024 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.