Pith. sign in

REVIEW 2 major objections 1 minor 1 cited by

CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs

T0 review · 2 major / 1 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Routing through a gateway API with shared credentials creates global prompt cache sharing across unrelated users.

desk verdict The paper applies Gu et al.'s audit to OpenRouter but supplies no data showing the method works through the gateway. read the letter →

arxiv 2605.30613 v1 pith:IVKJHQ77 submitted 2026-05-28 cs.CR cs.LG

Ryan Fahey This is my paper
classification cs.CRcs.LG
keywords promptcachingcacheisolationAPIgatewaysLLMsecuritytimingattacksside-channel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether an intermediary gateway that routes requests using shared organizational credentials can break the per-organization prompt cache isolation that LLM providers implement. It applies an existing audit technique to detect whether one user's cached prompt becomes available to other users on the same gateway. If the claim holds, then the isolation guarantees providers advertise do not survive passage through such gateways. A reader would care because many production LLM workloads now rely on prompt caching for speed and cost, yet the security boundary may be drawn at the wrong layer.

What carries the argument

Application of a timing-and-metadata cache audit to requests that traverse the gateway instead of direct provider endpoints.

What would settle it

Running the same audit with two separate gateway accounts on unrelated prompts and observing no cross-account cache hits would show the isolation is preserved.

Watch

Extended reading notes

Core claim

Routing requests through the gateway with shared credentials produces cache hits between distinct users, showing that provider-level per-account or per-organization isolation is bypassed and replaced by global sharing at the gateway level.

Load-bearing premise

The cache audit technique developed for direct provider endpoints still correctly detects cache state even after the gateway may rewrite headers or alter request handling.

Editorial extensions

If this is right

  • Provider isolation guarantees become ineffective for any traffic that passes through the gateway.
  • Cache-based side channels that were contained within one organization now become available to every gateway user.
  • Existing per-provider isolation policies must be re-evaluated when requests are mediated by shared-credential gateways.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Other gateways that aggregate traffic under shared credentials are likely to exhibit the same global sharing behavior.
  • Future isolation mechanisms may need to be enforced at the gateway layer rather than only at the model provider.
  • Audits that worked on direct endpoints should be re-validated whenever an intermediary is introduced.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces CacheProbe to audit prompt cache isolation in gateway APIs. It claims that routing requests through OpenRouter using shared organizational credentials creates global prompt cache sharing across all OpenRouter users, bypassing the per-organization isolation guarantees implemented by underlying LLM providers. The work applies and extends the timing/metadata audit technique from Gu et al. (ICML 2025) to intermediary gateway architectures.

Significance. If the central claim is substantiated with appropriate controls, the result would be significant for LLM API security, as it identifies a previously unexamined attack surface introduced by shared-credential gateways. The approach builds directly on prior audit methods and could inform isolation requirements for intermediaries. No machine-checked proofs or parameter-free derivations are present, but the topic is timely given the rapid adoption of prompt caching.

major comments (2)
  1. [Abstract] Abstract: The central claim that OpenRouter creates global cross-user cache sharing depends on the Gu et al. timing/metadata audit remaining effective when requests pass through the gateway. No controls, equivalence experiments, or discussion of potential gateway modifications (header rewriting, prompt normalization, or inserted caching layers) are supplied to verify that the isolation measurement distinguishes provider-level per-org caching from gateway-level sharing. This assumption is load-bearing for the reported vulnerability.
  2. [Abstract] The manuscript provides no experimental data, methodology details, or results to support the claim that the audit distinguishes the two caching regimes. Without such evidence the central claim cannot be evaluated.
minor comments (1)
  1. [Abstract] The abstract references Gu et al. (ICML 2025) but does not include a full citation or clarify whether any modifications to their method were required for the gateway setting.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful review and for identifying the need to strengthen validation of the audit through the gateway. We address each comment below and commit to revisions that improve clarity without altering the core claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that OpenRouter creates global cross-user cache sharing depends on the Gu et al. timing/metadata audit remaining effective when requests pass through the gateway. No controls, equivalence experiments, or discussion of potential gateway modifications (header rewriting, prompt normalization, or inserted caching layers) are supplied to verify that the isolation measurement distinguishes provider-level per-org caching from gateway-level sharing. This assumption is load-bearing for the reported vulnerability.

    Authors: We agree that explicit controls would make the argument more robust. The manuscript applies the Gu et al. technique but does not present dedicated equivalence experiments. In the revision we will add a new subsection with side-by-side timing measurements for direct provider access versus OpenRouter routing under matched conditions, plus a short discussion of why common gateway operations (header rewriting, normalization) preserve the observable cache signals in our traces. revision: yes

  2. Referee: [Abstract] The manuscript provides no experimental data, methodology details, or results to support the claim that the audit distinguishes the two caching regimes. Without such evidence the central claim cannot be evaluated.

    Authors: Sections 3 and 4 of the full manuscript already contain the methodology and timing distributions that separate per-organization provider caching from the global sharing observed via OpenRouter. To address the referee's concern we will expand the abstract with a one-sentence summary of the distinguishing evidence and insert an additional figure caption and paragraph that explicitly contrasts the two regimes. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; empirical application of independent external method

full rationale

The paper applies the audit technique developed by Gu et al. (ICML 2025) to test prompt cache isolation on OpenRouter. No equations, fitted parameters, self-definitional constructs, or self-citations appear in the abstract or described claims. The derivation consists of an empirical investigation of a gateway intermediary using an externally developed method; the central claim does not reduce to its own inputs by construction. This matches the default expectation of a self-contained paper with no load-bearing self-referential steps.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review supplies no equations, parameters, or modeling choices; ledger remains empty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs." pith.science (2026). https://pith.science/paper/IVKJHQ77

@misc{pith2026260530613,
  author       = {Pith},
  title        = {Pith review of: CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IVKJHQ77}},
  note         = {Machine review of arXiv:2605.30613}
}
read the original abstract

Over the past year, prompt caching in Large Language Models (LLMs) has become increasingly more popular across inference APIs. Prompt caching helps save precious compute resources and speeds up response times by reusing parts of the KV cache of a specific prompt for another request. However, many implementations of prompt caching are not secure against timing attacks or even basic metadata disclosure. Gu et al. (ICML 2025) develop a method to audit prompt caching in LLMs. This paper investigates whether OpenRouter's API gateway architecture introduces prompt caching vulnerabilities that bypass provider-level prompt cache isolation guarantees. Most LLM inference providers implement per-account or per-organization prompt caching to prevent data leaks, but does routing through OpenRouter with shared organizational credentials inadvertently create global cache sharing across all OpenRouter users?

Figures

Figures reproduced from arXiv: 2605.30613 by the authors.

Figure 1
Figure 1. Comparison of cross-account cache detection across routing configurations. Cache sharing is detected via OpenRouter default credentials but not via [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Metadata disclosure rates showing cached token percentages across scenarios. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Groq TTFT boxplots across all scenarios. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Groq TTFT histograms across all scenarios. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Fireworks TTFT boxplots across all scenarios. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Fireworks TTFT histograms across all scenarios. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: OpenAI TTFT boxplots across all scenarios. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: OpenAI TTFT histograms across all scenarios. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Keeping the Cache Warm Pays: Keepalive Economics for Agentic Workloads

    cs.DC 2026-07 conditional novelty 5.0 of 10

    Keepalive pings at the largest interval safely under a provider's cache TTL keep agentic prompts warm and profitable, with savings bounded by ~tau(w/r - 1), while universal adoption creates an unpriced congestion externality.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Auditing prompt caching in language model apis,

    C. Gu, X. L. Li, R. Kuditipudi, P. Liang, and T. Hashimoto, “Auditing prompt caching in language model apis,” inF orty-second International Conference on Machine Learning, ICML 2025, V ancouver , BC, Canada, July 13-19, 2025, ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaf...

    2025

  2. [2]

    In Gim, Guojun Chen, Seung seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong

    I. Gim, G. Chen, S. seob Lee, N. Sarda, A. Khandelwal, and L. Zhong, “Prompt cache: Modular attention reuse for low-latency inference,” 2024. [Online]. Available: https://arxiv.org/abs/2311.04934

    work page arXiv 2024

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.