Pith. sign in

REVIEW 4 major objections 6 minor 22 references

Query-agnostic compression paired with caching beats all tested strategies on 16 of 16 configurations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:06 UTC pith:G6RNZFW5

load-bearing objection Solid cost-model idea with real production data, but the flagship 16/16 dominance result is unauditable until the authors name and release the black-box Compress function. the 4 major comments →

arxiv 2607.15516 v1 pith:G6RNZFW5 submitted 2026-07-17 cs.LG cs.AI

Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

Yan Song This is my paper
classification cs.LG cs.AI
keywords prompt compressionprompt cachingcache-aware compressiontwo-tier cachecost modelquery-agnostic compressionprefix cachingLLM API cost
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that prompt compression and prompt caching have been treated as independent cost levers, even though the dominant query-aware compression family produces a different prefix per query and thereby invalidates prefix-strict caches on every call. Using measured cache hit rates on a production LLM API, it derives a cost model with a crossover threshold: because small cached prefixes hit at only about 0.83 and cost 12.5 times more to write than read, naive caching stops beating query-aware compression once the compression ratio reaches about 6. The proposed remedy, Cache-Aware Prompt Compression (CAPC), compresses the static document once, caches that compressed prefix, and bounds the compression ratio so the prefix stays in the large persistent tier rather than the small hot tier. On LongBench-v2 this is the cheapest strategy in every tested document-size and ratio cell, saving 49% over cache-only, 64% over query-aware compression, and 90% over vanilla at quality within 0.05 of the uncompressed baseline. Production validations on a 94k-token tool-schema prefix, two knowledge-graph RAG codebases, and a judge-free public benchmark reproduce the ordering.

Core claim

The paper's central discovery is that a production LLM cache behaves as a two-tier system: prefixes larger than roughly 3,500 tokens reliably hit at a rate near 1.0, while smaller prefixes plateau at about 0.83 across 30-call sessions. Because query-aware compression regenerates the compressed prefix on every query, it turns a potential cache read into a cache write that costs 12.5 times more. The paper converts this into a provider-agnostic cost model with the crossover formula rho_cross(r) = (alpha - 1/r) / (alpha - beta), then uses it to motivate CAPC: compress the static document once with query-agnostic compression, mark that compressed block with cache_control, and choose r so the comp

What carries the argument

The load-bearing machinery is the measured cache-hit-rate surface rho(N,|P|), whose two-tier step near 3,500 tokens lets the paper replace the literature's implicit rho=1.0 assumption with a realistic curve. On top of that sits the crossover identity rho_cross(r) = (alpha - 1/r) / (alpha - beta), which says cache-only stops beating query-aware compression once the compression ratio reaches about 6, and the tier-preserving bound r_max = floor(|D|/3500), which prevents compression from pushing the cached prefix into the hot tier where the hit rate drops to about 0.83. CAPC itself is a three-step procedure: compress the static document once, cache the compressed block with an explicit cache mar

Load-bearing premise

The paper never names, describes, or releases the query-agnostic compressor used for every CAPC and query-aware result, so if that compressor cannot actually hit the stated ratios with acceptable quality, the 16/16 dominance, the Pareto-results, and the tau-bench penalty are artifacts of that compressor choice rather than properties of the CAPC framework.

What would settle it

Run the same 16 LongBench-v2 document-size/ratio cells with a different publicly available query-agnostic sentence-selection compressor: if CAPC is not the cheapest in every cell, or if quality drops more than 0.05 on the 28k-token document, the dominance claim fails. A second falsifier is to measure the 2.4k-token prefix hit rate across a 50-call session; if it reaches a plateau near 1.0 instead of about 0.89, the two-tier model and the crossover predictions collapse.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Query-aware compression can be more expensive than no compression at all on cache-heavy workloads: tau-bench shows +40.1% over vanilla, the reverse of the usual assumption.
  • At compression ratios of 6 and above, naive cache-only stops beating query-aware compression, so the conventional winner of the caching-versus-compression trade-off changes at high ratios.
  • A fixed compressed prefix cached once retains quality within 0.05 of the uncompressed baseline while cutting cost by 49% over cache-only and 90% over vanilla across all 16 LongBench-v2 configurations.
  • On a 94k-token production tool-schema prefix, CAPC at ratio 3 cuts cost by 51.7% while preserving tool-selection quality essentially equal to cache-only.
  • For knowledge-graph RAG, caching a compressed structural layer and fetching source snippets per query delivers a 9.3x cost reduction on one codebase and 2.4x on another, at roughly 85% cache hit rate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 3,500-token threshold is an empirical particular of the measured provider and model; if other providers shift the threshold, the tier-preserving bound should be recomputed from local pricing, though the shape of the two-tier curve and the CAPC ordering may persist.
  • If a provider ever introduces semantic prefix matching that tolerates small perturbations, CAPC's advantage would shrink to pure token reduction and the crossover formula would need a mutated-fraction parameter instead of a binary hit/miss assumption.
  • The paper's two production agent workloads sit at opposite ends of a spectrum: the tau-bench penalty and the enterprise-assistant saving both come from the same query-aware mechanism, suggesting that the cost effect of query-aware compression is monotone in how much of the cached prefix it mutates.
  • The two-layer CAPC pattern for knowledge-graph RAG should generalize to other pointer-emitting indexers such as code-search systems and schema catalogues, because the indexer supplies structure while the prompt must still deliver content on demand.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies the interaction between prompt caching and prompt compression in LLM APIs. It reports an empirical characterization of Anthropic Sonnet 4.6's prefix cache, claiming a two-tier architecture with a hot-tier plateau ρ≈0.83 and a persistent tier at ρ=1.0 above ~3,500 tokens; a cost model (Eqs. 1–6) yielding a crossover ratio above which cache-only loses to query-aware compression; and CAPC, a query-agnostic compression + cache_control algorithm with a tier-preserving ratio bound rmax=floor(P/T). It claims 16/16 cost dominance on LongBench-v2, large savings on an enterprise tool-using workload, a graphify RAG case study, and a τ-bench validation in which CAPC matches vanilla reward at −7.9% cost while query-aware compression costs +40.1%. The cost-model algebra is simple and internally valid, and the paper is unusually candid about limitations. However, several load-bearing empirical claims are not verifiable as written: the compressor is never named, the reported costs do not reconcile with the model equations, and the crossover confirmation contradicts the paper's own ρ=1.0 characterization.

Significance. If the empirical claims hold, this is a useful and timely systems result: it quantifies a real failure mode of query-aware compression (cache-busting), provides a parametric crossover rule (Eq. 6) that practitioners can instantiate with their own prices, and contributes a simple, mechanistically grounded design rule (rmax). Strengths: the cost-model algebra is transparent and correct; the paper explicitly separates framework claims from model-version-specific measurements (Section 1); the τ-bench evaluation uses deterministic DB-state reward; the total spend ($98.96) makes full reproduction feasible; and Section 7 is candid about implicit tools= caching, the judge fallback, and in-sample confirmation. The main weaknesses are that 'predictions confirmed' is weakened by fitting and validating on the same API, and that the headline dominance and negative-ROI results rest on an unnamed compressor and mutually inconsistent ρ measurements. The framework contributions survive these issues; the empirical snapshot claims need repair before the strongest claims can be accepted.

major comments (4)
  1. [§5.1 (Listing 1); §§6.1–6.5] Every empirical headline—Table 7's 16/16 dominance, the graphify Pareto results, and the τ-bench +40.1% penalty—is computed with Compress(D,r), which §5.1 defines as 'any query-agnostic sentence-selection compressor' but never instantiates. Neither CAPC's compressor nor the Strategy-C query-aware compressor is named, described, or released. Since the paper claims CAPC is compressor-agnostic, the validation must demonstrate robustness across at least two independent compressors; as written, the results are unauditable and may reflect the particular compressor chosen. Specify the algorithm and hyperparameters, release the code, and re-run at least the LongBench grid and τ-bench with a second compressor.
  2. [§3.1 vs §6.1/§6.4 (Tables 2, 7; Fig. 8)] Section 3.1's characterization and Section 6.1's crossover 'confirmation' contradict each other. Table 2 reports ρ=1.00 for prefixes ≥4,096 tokens 'from the first subsequent call'; the 12k–24k LongBench docs therefore have ρ_B=1.0 by the paper's own measurement. Substituting ρ=1 into Eq. (5) gives ρ_cross(6)=0.942 < ρ_B, so B must beat C at r=6. Yet §6.1 reports C cheaper than B in 4/4 r=6 cells and Table 7 shows C<B (0.0092 vs 0.0096 on the 12,191-token doc). Separately, Fig. 8 shows 5k–260k prefixes converging to ~85% hit rate, also contradicting Table 2. The confirmation only goes through if deployed ρ_B ≈ 0.85–0.92, i.e., if the two-tier characterization is wrong for production-style sessions. Report session-level ρ for the actual LongBench/graphify prefixes and state which regime Table 2 refers to.
  3. [§6.1, Table 7 vs Eqs. (1)–(4)] The Table 7 entries do not reproduce from Eqs. (1)–(4) with the stated prices. For the 12,191-token doc, Eq. (1) (A=$0.0447) implies a common per-query overhead Dq·pin+O·pout of roughly $0.006–0.008. With ρ=1, Eq. (2) then gives costB≈$0.0096–0.0118, while Eq. (3) at r=6 gives costC≈$0.0120–0.0142—above costB; Table 7 reports the opposite ordering (0.0092 vs 0.0096). No single ρ value reconciles all rows (e.g., costD at r=2, a 6,096-token persistent prefix, is predicted near $0.0077 but reported as $0.0061). Publish per-configuration token and ρ values (|D|/r, Dq, O, ρ_B, ρ_D) so the cost table can be checked against the model.
  4. [Abstract; §6.1; §7.4] The abstract's 'quality within 0.05 of the uncompressed baseline at tier-preserving ratios' is unverifiable from reported data. §6.1 gives only per-strategy means (A 0.73, C 0.67, D 0.66) and notes that at r=2 all strategies are within 0.04; no per-cell quality table covers the 16 cells, so the 0.05 bound at tier-preserving ratios (rmax=floor(|D|/3500), e.g., r≤3 for the 12,191-token doc) cannot be checked, and the mean D-vs-A gap is 0.07. The §7.4 admission that judge calls silently fell back to a 0.5 score during one run compounds this. Also, τ-bench's 'p=1.00' (z=0 on identical counts 36/50) is non-significance, not equivalence; with N=50 the Wilson CI is [0.58, 0.83]. Report per-cell quality and audit the judge calls affected by the fallback.
minor comments (6)
  1. [§6.3 vs §6.6] The EA simulator validation is described as '15 representative queries' at $9.91 (Table 10), but §6.6 itemizes the EA simulator validation as '40 queries × 4 strategies' at $9.57. Reconcile query counts and amounts.
  2. [Table 4] The rows labeled 'Ncross (ideal ρ=1)' are misleading: under ρ=1, B beats C for all r≤10, so no crossover N exists. These values appear to be the N at which the measured hot-tier curve crosses ρ_cross; relabel them accordingly.
  3. [Table 10] The column header 'Quality vs Vanilla' mixes a set-IoU score (0.680) with a cost delta (−51.7%). Use separate columns for quality and cost.
  4. [§3.1; Eq. (7)] T≈3,500 is interpolated between the 2,053-token (ρ=0.83) and 4,096-token (ρ=1.00) measurements with no intermediate data, yet Eq. (7) treats it as an exact bound. Report sensitivity of rmax and the §6.2 cost-rebound claim to T ∈ [3,000, 4,000].
  5. [Fig. 2 vs §3.1] The Fig. 2 caption says ρ=1.0 'from N=2 onward', while §3.1 says 'from the first subsequent call' (N=1). Clarify. This is a symptom of the larger ρ inconsistency flagged in Major Comment 2.
  6. [§6.5, Table 14] Query-aware compression attains the highest reward (38/50) despite its +40.1% cost. Given the paper's cost focus this is fine, but the text should acknowledge that reward differences are within noise rather than implying C is quality-inferior.

Circularity Check

0 steps flagged

No significant circularity: the crossover, dominance, and production-validation claims rest on independent API-spend measurements rather than on the model's definitions.

full rationale

The paper's derivation chain is not circular. The empirical characterization in Section 3 measures ρ(N,|P|) and the ~3,500-token threshold directly on the Sonnet 4.6 API. Section 4's cost model (Eqs. 1–6) is an algebraic combination of those measured parameters and published prices, and the crossover thresholds in Tables 3–4 are computed, not fitted, from them. The Section 6 experiments then measure per-query API costs and quality independently across actual calls; the '4 of 4 r=6 configurations showed C cheaper than B' result is an external cost observation, not a restatement of the model's assumptions. The tier-preserving bound rmax=⌊P/3500⌋ reuses the measured threshold, but it is a design constraint rather than a predicted output, and the paper does not present that bound itself as a validated discovery. There are no load-bearing self-citations: the references are external prior work, and no uniqueness theorem or ansatz is imported from the same authors. The two substantive limitations—the unnamed Compress(D,r) implementation that underlies the 16/16 grid and the Section 7.4 admission that judge calls sometimes silently fell back to a 0.5 score—are reproducibility and measurement-quality risks, not circularity: they affect external validity but do not make any claimed result equivalent to its inputs by construction. The τ-bench validation uses deterministic DB-state reward, so its quality-equivalence claim is independent of the judge limitation. Overall, the framework claims are self-contained and the empirical claims are externally measured; the correct finding is no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical or mechanistic entities are introduced; the 'two-tier cache' is an empirical characterization of API behavior, not a postulated entity. The load-bearing free parameters are the fitted tier threshold and the measured rho plateau, both reused inside the crossover and ratio-bound arguments.

free parameters (4)
  • T = 3,500 token persistent-tier threshold = 3500 tokens
    Set between measured rho=0.83 at 2,053 tokens and rho=1.0 at 4,096 tokens (Table 2); used in tier-preserving ratio bound rmax=floor(|D|/T).
  • hot-tier hit-rate plateau rho = ~0.83 (0.89 in Figure 1)
    Empirical plateau of cumulative hit rate for prefixes below threshold; drives crossover predictions in Section 4.4; values inconsistently reported as 0.83 vs 0.89.
  • AdaptiveCacheBoundary thresholds eps_static / eps_quasi = 0.05 / 0.30
    Hand-chosen mutation-rate thresholds in Section 5.3; no sensitivity analysis; affects adaptive-prefix savings claims.
  • graphify Layer-1 per-element token estimates = god node 25, community heading 60, inferred edge 22
    Empirically fit token budgets used by capc_graphify_profiler.py to back-solve Layer-1 size; subclaim only.
axioms (4)
  • domain assumption Prefix-strict cache key with cw > cr pricing exists on the deployed API (Anthropic Sonnet 4.6).
    Stated in Section 1 and used in the cost model; partially verified by billing reconciliation and mutation tests, but provider behavior can change.
  • domain assumption The measured rho(N,|P|) two-tier curve is stationary and applies to all workloads tested (LongBench, EA, graphify, tau-bench).
    Section 3.1 measures on synthetic prompts; Section 7.2 later shows mutation footprint changes cache costs, so the premise is workload-dependent.
  • ad hoc to paper An unnamed query-agnostic sentence-selection compressor (Compress(D,r)) exists and meets the quality and cost assumptions of CAPC.
    Listing 1 leaves Compress abstract; no implementation is cited for any experiment, so the central comparisons assume a competent compressor without specification.
  • domain assumption Haiku 4.5 self-consistency scoring is a valid proxy for answer quality.
    Used for LongBench, EA, and graphify quality; Section 7.4 flags judge-family bias and a silent 429 fallback to 0.5 scores.

pith-pipeline@v1.3.0-alltime-deepseek · 21988 in / 22948 out tokens · 221904 ms · 2026-08-01T23:06:36.109670+00:00 · methodology

0 comments
read the original abstract

Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent). The compression literature has standardized on query-aware methods that produce a different compressed prefix per query, mechanically invalidating the prefix-strict cache on every call. We characterize this cost empirically on Anthropic's Sonnet 4.6 API and find caching is far from the rho=1.0 ideal the literature assumes: Sonnet's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at rho~0.83 across 30-call sessions. Our cost model predicts, and experiments confirm, that under realistic rho, query-aware compression beats naive caching at high compression ratios (r>=6). We propose Cache-Aware Prompt Compression (CAPC), pairing query-agnostic compression with explicit cache_control plus a tier-preserving ratio bound that prevents over-compression from pushing the cached prefix into the hot tier. CAPC is the cheapest strategy in 16/16 configurations on LongBench-v2, with mean savings of 49% over cache-only, 64% over query-aware compression, and 90% over vanilla, at quality within 0.05 of the uncompressed baseline. We validate CAPC on three production workloads: an enterprise tool-using assistant with a 94k-token schema prefix (51.7% cost reduction at r=3); a graphify knowledge-graph RAG pipeline across two codebases (9.3x vs cache-all on FastAPI, 2.4x on httpx); and the public tau-bench retail benchmark (50 tasks), where CAPC is the cheapest of four strategies with reward exactly equal to vanilla (both 36/50, p=1.00) while query-aware compression is the most expensive at +40.1% over vanilla -- the first production confirmation of the crossover model's negative-ROI prediction on a public benchmark.

Figures

Figures reproduced from arXiv: 2607.15516 by Yan Song.

Figure 1
Figure 1. Figure 1: Cumulative cache hit rate ρ(N) at |P| = 2.4k cached tokens, n = 3 independent trials. Hit rate climbs from ρ ≈ 0.47 at N = 5 to a plateau of ρ ≈ 0.89 at N ≥ 40. 3.2 Prefix invalidation is token-strict, with one tokenizer caveat A correctness prerequisite is that the cache key is computed over the exact token sequence of the marked prefix. We tested this by sending the same prompt 10 times (warming the cach… view at source ↗
Figure 2
Figure 2. Figure 2: ρ(N, |P|) surface at four prefix sizes. The 2k curve (hot tier) plateaus at ρ ≈ 0.83; 4k, 6k, 8k (persistent tier) sit at ρ = 1.0 from N = 2 onward. 4.1 Notation and the strategy zoo Let |D| be the document size in tokens, r the compression ratio, N the number of queries, Dq the per-query dynamic block size, O the output size. A — Vanilla Full document and query, no cache, no compression. B — Cache-only Fu… view at source ↗
Figure 3
Figure 3. Figure 3: Adaptive boundary on real LongBench-v2 docs with synthetic version drift. Left: cost [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: AdaptiveCacheBoundary on real EA git histories ( [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-query cost vs. document length on LongBench-v2. Each panel shows the four strategies [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cost-quality Pareto frontier per document. Each panel = one document; each curve = one [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cumulative cache hit rate. All four caching strategies converge to ≈0.85 by query 10–15 (server-pool replication delay). G-D vari￾ants reach steady state slightly faster than G-B because smaller prefixes propagate sooner across the pool. training-time exposure (G-A Q=0.647), the structural metadata is largely redundant: graphify￾native quality is 0.613 — below vanilla, because the added lines dilute attent… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 12 linked inside Pith

  1. [4]

    ACM ICAIF ’25 Workshop on LLMs and Generative AI for Finance

    URLhttps://arxiv.org/abs/2510.18043. ACM ICAIF ’25 Workshop on LLMs and Generative AI for Finance. Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach to routing and cascading for llms. InInternational Conference on Learning Representations (ICLR),

    arXiv

  2. [5]

    25 Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, and Tatsunori Hashimoto

    URL https://arxiv.org/abs/2410.10347. 25 Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, and Tatsunori Hashimoto. Auditing prompt caching in language model apis,

    Pith/arXiv arXiv

  3. [6]

    Accepted ICML

    URL https://arxiv.org/abs/2502.07776. Accepted ICML

    Pith/arXiv arXiv

  4. [7]

    Llmlingua: Compressing prompts for accelerated inference of large language models, 2023a

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models, 2023a. URLhttps://arxiv.org/ abs/2310.05736. Accepted EMNLP

    Pith/arXiv arXiv

  5. [8]

    Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression, 2023b

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression, 2023b. URLhttps://arxiv.org/abs/2310.06839. Accepted ACL

    Pith/arXiv arXiv

  6. [10]

    Accepted ACL 2025 Main; University of Cambridge

    URLhttps://arxiv.org/abs/2408.03094. Accepted ACL 2025 Main; University of Cambridge. Hanbing Liu, Chunhao Tian, Nan An, Ziyuan Wang, Pinyan Lu, Changyuan Yu, and Qi Qi. Budget-constrained agentic large language models: Intention-based planning for costly tool use,

    Pith/arXiv arXiv 2025

  7. [11]

    URLhttps://arxiv.org/abs/2602.11541. Tengxiao Liu, Zifeng Wang, Jin Miao, I-Hung Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, Samira Daruki, Yi Liang, William Yang Wang, Tomas Pfister, and Chen-Yu Lee. Budget-aware tool-use enables effective agent scaling,

    arXiv

  8. [12]

    Google; 15 authors

    URLhttps: //arxiv.org/abs/2511.17006. Google; 15 authors. Elias Lumer, Faheem Nizar, Akshaya Jangiti, Kevin Frank, Anmol Gulati, Mandar Phadate, and Vamse Kumar Subbiah. Don’t break the cache: An evaluation of prompt caching for long-horizon agentic tasks,

    arXiv

  9. [13]

    Evaluation across OpenAI, Anthropic, Google on DeepResearch Bench

    URL https://arxiv.org/abs/2601.06007. Evaluation across OpenAI, Anthropic, Google on DeepResearch Bench. Alliot Nagle, Adway Girish, Marco Bondaschi, Michael Gastpar, Ashok Vardhan Makkuva, and Hyeji Kim. Fundamental limits of prompt compression: A rate-distortion framework for black-box language models,

    arXiv

  10. [14]

    URLhttps://arxiv.org/abs/2407.15504. NeurIPS

    Pith/arXiv arXiv

  11. [15]

    URLhttps://arxiv.org/abs/2406.18665. OpenAI. Automatic prompt caching,

    Pith/arXiv arXiv

  12. [16]

    Salesforce AI Research + UIUC

    URL https://arxiv.org/abs/2510.08439. Salesforce AI Research + UIUC. Luis Gaspar Schroeder, Aditya Desai, Alejandro Cuadron, Kyle Chu, Shu Liu, Mark Zhao, Stephan Krusche, Alfons Kemper, Matei Zaharia, and Joseph E. Gonzalez. vcache: Verified semantic prompt caching,

    arXiv

  13. [17]

    Accepted ICLR

    URLhttps://arxiv.org/abs/2502.03771. Accepted ICLR

    arXiv

  14. [18]

    Converts code, documentation, PDFs, and images into a NetworkX knowledge graph with Leiden community detection and LLM-extracted concept edges

    Open-source tool; v1 release. Converts code, documentation, PDFs, and images into a NetworkX knowledge graph with Leiden community detection and LLM-extracted concept edges. Zhentao Xu, Fengyi Li, Albert C. Chen, and Xiaofeng Wang. Procut: Llm prompt compression via attribution estimation. InProceedings of EMNLP 2025 Industry Track,

    2025

  15. [19]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A benchmark for tool-agent-user interaction in real-world domains

    URLhttps: //arxiv.org/abs/2508.02053. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A benchmark for tool-agent-user interaction in real-world domains. InAdvances in Neural Information Pro- cessing Systems (NeurIPS),

    arXiv

  16. [20]

    Sierra Research

    URLhttps://arxiv.org/abs/2406.12045. Sierra Research. Benchmark of multi-round tool-using LLM agents with deterministic database-state reward. Renos Zabounidis, Aditya Golatkar, Michael Kleinman, Alessandro Achille, Wei Xia, and Stefano Soatto. Re-forc: Adaptive reward prediction for efficient chain-of-thought reasoning,

    Pith/arXiv arXiv

  17. [21]

    Ivan Zakazov, Berke Argin, Oussama Gabouj, Kamel Charaf, Alexander Sharipov, Alexi Semiz, Lorenzo Drudi, Nicolas Baldwin, and Robert West

    URL https://arxiv.org/abs/2511.02130. Ivan Zakazov, Berke Argin, Oussama Gabouj, Kamel Charaf, Alexander Sharipov, Alexi Semiz, Lorenzo Drudi, Nicolas Baldwin, and Robert West. Cmprsr: Abstractive token-level question- agnostic prompt compressor,

    Pith/arXiv arXiv

  18. [22]

    "" Feed one observed document version

    URLhttps://arxiv.org/abs/2511.12281. A AdaptiveCacheBoundary Pseudocode Listing 2: AdaptiveCacheBoundary Python implementation class A d a p t i v e C a c h e B o u n d a r y : def __init__ ( self , e p s _ s t a t i c =0.05 , ep s_ qu as i =0.30 , mi n_ ca l ls =3) : self . e p s _ s t a t i c = e p s _ s t a t i c self . e ps _ qu as i = ep s_ qu as i s...

    arXiv

  19. [2023]

    Joong Ho Choi, Jiayang Zhao, Jeel Shah, Ritvika Sonawane, Vedant Singh, Avani Appalla, Will Flanagan, and Filipe Condessa

    URLhttps://arxiv.org/abs/2305.05176. Joong Ho Choi, Jiayang Zhao, Jeel Shah, Ritvika Sonawane, Vedant Singh, Avani Appalla, Will Flanagan, and Filipe Condessa. Compactprompt: A unified pipeline for prompt and data compression in llm workflows,

    Pith/arXiv arXiv

  20. [2024]

    THUDM; 503 multiple-choice questions, 8k-2M tokens

    URLhttps://arxiv.org/abs/2412.15204. THUDM; 503 multiple-choice questions, 8k-2M tokens. Sarthak Chakraborty, Suman Nath, Xuchao Zhang, Chetan Bansal, and Indranil Gupta. Generative caching for structurally similar prompts and responses,

    Pith/arXiv arXiv

  21. [2025]

    NeurIPS 2025 Poster; 83% hit rate on production prompts

    URLhttps://arxiv.org/abs/ 2511.17565. NeurIPS 2025 Poster; 83% hit rate on production prompts. Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance,

    arXiv 2025

  22. [2026]

    Zongqian Li, Yixuan Su, and Nigel Collier

    URL https://arxiv.org/abs/2604.02985. Zongqian Li, Yixuan Su, and Nigel Collier. 500xcompressor: Generalized prompt compression for large language models,

    Pith/arXiv arXiv