Pith. sign in

REVIEW 3 major objections 6 minor 51 references

InstCache: A Predictive Cache for LLM Serving

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fine-tuned LLM's negative log-likelihood ordering can precompute answers to future user requests, beating exact-match caching by up to 2.3x.

desk verdict Interesting and novel idea, but the headline 2.3x hit-rate claim is measured on random splits that leak future data; the only temporal experiment lacks an exact-match baseline, so the deployment case is not yet made. read the letter →

arxiv 2411.13820 v2 pith:HTTTMZ6P submitted 2024-11-21 cs.CL cs.DC

Longwei Zou , Yan Liu , Jiamu Kang , Tingfeng Liu , Jiangang Kong , Yangdong Deng This is my paper · ORCID
classification cs.CLcs.DC
keywords LLMservinginstructioncachingnegativelog-likelihoodspatiallocalitycachepre-populationV-arytreesearchhitratepredictionexact-match
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Instruction-response caches for LLM serving normally only help when the same request appears twice, which is rare in real traffic. This paper argues that a language model fine-tuned on past user instructions can be used to predict which never-seen instructions are likely to arrive, by treating each instruction's negative log-likelihood (NLL) as a one-dimensional address. The proposed InstCache stores every instruction whose NLL falls below a threshold, together with its precomputed response, and serves exact matches from a hash table. On deduplicated WildChat data, the cache achieves an 8.2% hit rate, about 2.3 times the exact-repeat upper bound for traditional caching, and when integrated with a serving engine it reduces time per output token by up to 42.0% and 50.0% on the LMSys and Moss datasets. If the effect holds, serving systems could answer a substantial fraction of requests without running the large model at all.

What carries the argument

The load-bearing mechanism is the use of a fine-tuned LLM's negative log-likelihood, $\operatorname{NLL}(s) = -\sum_i \log p(t_i \mid t_{<i})$, as a one-dimensional "address" for each possible instruction. The cache is the thresholded set $C = \{\langle s, r\rangle : \operatorname{NLL}(s) \le \sigma\}$, and because NLL is additive over tokens, a depth-first search over the model's $V$-ary tree can enumerate this set while pruning subtrees whose cumulative NLL already exceeds $\sigma$; keeping key-value states along the current path makes the enumeration reuse computation. A secondary piece is Theorem 1, which estimates the number of enumerated texts from power-law token probabilities, and Equation 1, which predicts hit rate as the CDF $F_N(\sigma)$ of the NLL distribution measured on validation instructions.

What would settle it

Train InstCache on the first six months of WildChat and compute its hit rate at a fixed threshold for each subsequent month: if the hit rate collapses to the exact-match repetition rate within a month or two, the NLL-locality claim is falsified. A second check is to enumerate all cached instructions and have an oracle judge how many are meaningful user requests; if the low-NLL region is mostly generic filler, the hit-rate number is an artifact of the threshold rather than evidence of prediction.

Watch

Extended reading notes

Core claim

The central claim is that a language model trained on observed instructions rearranges the space of all possible texts so that future user instructions—even ones that have never appeared before—cluster near the low-NLL head of the distribution. The paper defines the cache as the set $C = \{\langle s, r\rangle : \operatorname{NLL}(s) \le \sigma\}$, where $\sigma$ is a chosen threshold, and shows that this set can be enumerated offline by a depth-first walk of the model's $V$-ary text tree, pruning any path whose cumulative NLL exceeds $\sigma$. Under the assumption that next-token probabilities follow a power law, Theorem 1 gives the number of texts with $\operatorname{NLL}\le\sigma$ as $N \approx e^{\sigma/\alpha}(\sigma/\alpha)^{L-1}/(L-1)!$, which lets cache size and hit rate be predicted before construction. The empirical anchor is that on deduplicated WildChat the hit rate reaches 8.2% versus a 3.6% exact-repeat upper bound, a 2.3x improvement, while integrated serving reduces time per output token by up to 42.0% and 50.0% on the LMSys and Moss datasets.

Load-bearing premise

The load-bearing premise is that the NLL ranking learned from past instructions remains a reliable ranking of what users will ask next: future real requests stay concentrated in the low-NLL region, and that region does not fill up with boilerplate as the instruction distribution drifts.

Editorial extensions

If this is right

  • At a fixed threshold, the hit rate scales with cache size, so operators with large or multi-tier storage can push the hit rate well beyond the ranges reported here.
  • Because both cache size and hit rate can be predicted before construction, a serving operator can choose $\sigma$ to meet a target trade-off between storage and latency savings.
  • InstCache works at the level of whole requests and is orthogonal to token-level key-value caching, so the two can be combined: the predictive cache absorbs repeated or predictable requests, while the KV cache accelerates the generation that still occurs on misses.
  • Under higher request rates, the latency benefit grows, since each cache hit removes a full generation from the serving engine's load.
  • When the instruction distribution drifts over time, hit rates decline gradually, and the paper argues that periodic re-pre-population can restore them; the low costs of the tree search make refresh cycles practical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the NLL-locality assumption transfers, the same thresholded-likelihood idea could apply at finer granularity—caching reusable prefixes, tool-call templates, or retrieved document sets for retrieval-augmented generation—rather than only full instruction-response pairs.
  • The reported gains come from open-domain chat logs with heavy boilerplate; specialized domains with narrow, shifting vocabularies may have a much thinner low-NLL region, so the hit-rate multiplier should be re-measured per domain before deployment.
  • A cheap online adaptation would be to re-estimate the NLL CDF on a rolling window of recent traffic and adjust $\sigma$ continuously, turning the static threshold into a control knob instead of a one-time setting.
  • Because the Appendix A proof of Theorem 1 omits some simplification steps, a direct numerical enumeration of low-NLL texts for small vocabulary sizes would settle whether the closed-form count is exact or an approximation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes InstCache, an instruction-level cache for LLM serving. The cache is constructed offline by fine-tuning a small LLM on observed user instructions and then enumerating, via a V-ary tree traversal, all instruction strings whose negative log-likelihood (NLL) under this model is below a threshold sigma; responses are generated with a larger serving model and stored in a hash table. At serving time, an incoming instruction is answered from the cache on an exact-match hit. The paper reports hit rates of 8.2% on WildChat (2.3x the repetition-rate upper bound), 20.1% on LMSys, and 23.4% on Moss, and latency reductions for vLLM of up to 42-50%. It also claims that hit rate and cache size can be predicted via the NLL cumulative distribution function (Eq. 1) and a power-law text-count theorem (Theorem 1).

Significance. The underlying idea of using an LLM's NLL to induce a total order on instruction space and caching the low-NLL region is interesting and potentially useful in serving systems where a small fraction of requests dominates traffic. The authors evaluate on several real-world conversation datasets, release code, and include a temporal drift experiment, which is the right instinct. If the comparison against exact-match caching is re-run under a causal split and the advantage survives, the paper would be a solid systems contribution. However, as presented, the headline hit-rate gain is not yet established because the main evaluation uses a random split that lets both InstCache and the baseline peek at future requests.

major comments (3)
  1. [§5.1, §5.2.1, §5.2.4] The central comparison is affected by temporal leakage. In Section 5.1 the data are split 80/10/10 randomly, so the 80% used to fine-tune the prepopulation LLM contains requests that occur after the test requests, and the exact-match 'upper bound' repetition rate in Figure 3 is computed from the same random 80% (for WildChat, 3.56%). Under deployment, a cache can only be built from the past, so this repetition rate is not an upper bound for a traditional cache that knows no future. The only chronological experiment (Section 5.2.4, Figure 7) trains on the first six months and tests on later months, but it reports no exact-match baseline under the same split, so it cannot show whether the 2.3x advantage over traditional caching survives temporal drift; indeed the figure shows InstCache's own hit rate declining steadily with the gap. To support the headline claim, the authors should report both InstCache hit rate and the exact-match repetition rate on a causal split (e.g., train on months 1-6, test on month 7, then months 8-12) and show the ratio over time.
  2. [§3, Eq. (1)] The 'hit-rate prediction' in Eq. (1) is a definitional identity rather than a predictive model. Because C is defined as {<s,r> : NLL(s) <= sigma}, the equality Hit_Rate = P(N <= sigma) = F_N(sigma) holds by construction; no property of the LLM or of user behavior is used. The validation-set estimate described after Eq. (1) therefore only checks whether the NLL distribution on a held-out random sample matches the validation sample; it does not validate the paper's central assumption that low-NLL regions are stable over time. The evaluation in Figure 4 should be reframed accordingly, or supplemented with a genuine out-of-time prediction experiment in which the NLL CDF estimated from past months is used to predict hit rates on future months.
  3. [§3, Theorem 1, Appendix A] The cache-size prediction rests on an assumption that is not stated as the strong idealization it is: the proof of Theorem 1 takes the next-token probabilities to follow the same power law P(t_i) = beta * i^{-alpha} at every position independently of context. Real LLM next-token distributions are highly context-dependent, and the power-law fit in Figure 8 is an aggregate over positions and contexts, not a per-node statement. The final formula also drops beta by assuming beta is close to 1. As a result, Theorem 1 is a heuristic estimate under a stylized model, not a theorem about the LLM's text space. The authors should either prove the claim under the actual conditional distributions (or state the needed conditions) or explicitly label Theorem 1 as an approximate scaling law and validate its error across datasets, thresholds, and sequence lengths; currently the claimed 'accurate' profiling in Section 5.2.1 is supported only by the single aggregate plot in Figure 4.
minor comments (6)
  1. [§5.1, Baselines] The text says the baseline cache is built from 80% of the data and measured on the remaining 20%, but Figure 3 and the surrounding text define the repetition rate as the proportion of test instructions appearing in the training set, implying an 80/10/10 split; please make the split consistent.
  2. [§4.1, Algorithm 1] The pseudocode does not specify how the candidate next-token set T is obtained from the LLM (full vocabulary, top-k, or a probability cutoff); without this, the enumeration is not fully reproducible from the pseudocode.
  3. [References [9], [11]] References [9] (AttentionStore) and [11] (MeanCache) are both assigned arXiv:2403.19708; one of these identifiers is incorrect.
  4. [§5.2.4, Figure 7] The x-axis labels ('Apr--Sep Oct Nov Dec Jan 2024 Feb Mar Apr') are ambiguous; please label the training and test months explicitly.
  5. [§5.2.1, Table 2] The columns 'Mem Size' and 'Storage Size' are not defined in the text; clarify what each measures (e.g., in-memory hash table size vs. disk storage, or KV-cache memory vs. response storage).
  6. [Abstract] The phrase 'upper bound of traditional caching mechanisms' is imprecise: the repetition rate is an upper bound for an exact-match cache with infinite capacity on a given training set, not for caching in general; please qualify it.

Circularity Check

2 steps flagged · score 6.0 of 10

The paper's analytic 'predictions' of hit rate and cache size are restatements of the cache's own NLL-threshold construction rule or forward calculations from the same fitted power-law distribution; the headline 2.3x hit-rate advantage is measured rather than derived, but it is reported under a random split that leaks future requests into the cache and training set.

  1. self definitional [Section 3, cache definition and Eq. (1)]
    "Formally, given a threshold σ, we define the InstCache as C = {⟨s, r⟩ : NLL(s) ≤ σ, s∈ S}... The expected hit rate of InstCache is then given by: Hit_Rate = P (S ∈C) = P (N ≤σ) = FN (σ) (1) where FN is the cumulative distribution function of N ."

    The 'prediction' in Eq. (1) is exactly the definition of the cache membership rule restated in probability notation: because C is defined as the set of instructions with NLL(s) ≤ σ, the hit rate against any stream is by construction the CDF F_N(σ). Measuring F_N on a validation set and comparing it to the test hit rate (Figure 4) therefore only checks that validation and test NLL distributions coincide; it does not independently test whether low-NLL texts are the instructions users will send. No derivation connects the LLM's NLL ordering to future demand beyond this definitional identity.

  2. fitted input called prediction [Section 3, Theorem 1; Section 5.2.1 and Appendix A (power-law fit)]
    "The power-law distribution parameters α and β can be estimated based on actual next-token generation statistics. ... As illustrated in the blue lines of Figure 4, by estimating the function parameters α, β defined in Theorem 1, we can profile the number of instructions for InstCache accurately."

    The predicted instruction count is not an independent estimate of cache size: Theorem 1 assumes P(t_i)=β i^{-α}, and α,β are fitted to the same model's next-token probabilities (Appendix A). The count N ≈ e^{σ/α}(σ/α)^{L−1}/(L−1)! is then a deterministic function of those fitted parameters under the assumed distribution. Thus the 'prediction' is a forward calculation from the fitted power law; agreement with the actual pre-populated cache in Figure 4 only validates the power-law approximation, not the claim that NLL-based caching anticipates real user requests.

full rationale

Section 3 defines InstCache solely by the NLL≤σ rule, so Eq. (1) is a tautological restatement of the construction rule, and the cache-size 'prediction' is computed from α,β fitted to the model's own token distribution. These two steps are the basis of the paper's claim that hit rate and cache size 'can be predicted.' They reduce, respectively, to the definition of the cache and to the fitted power-law assumption. I have not found self-citation load-bearing or uniqueness-imported-from-authors circularity: all cited baselines and datasets are external. The central empirical comparison (2.3x hit rate versus the exact-match upper bound in Figure 3) is a measurement, not a derivation, so the paper is only partially circular and no score above 6 is warranted. Separately, and not counted as circularity, the random 80/10/10 split includes future requests in the cache construction and in the fine-tuning data, and the only chronological experiment (Figure 7) reports no exact-match baseline; this is a correctness risk for the deployment claim but does not involve a definitional reduction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The system introduces no new physical entities. The novel machinery is a modeling choice: representing text space as a V-ary tree with NLL as a one-dimensional address. The load-bearing free parameters are the NLL threshold, the fitted power-law parameters used in the cache-size estimate, and the response-length filter. The main unstated assumptions are the stability of NLL ordering over time and the completeness of the tree enumeration.

free parameters (3)
  • NLL threshold sigma = 15-21 (swept)
    Defines cache membership in Section 3; higher sigma increases cache size and hit rate (Table 2). It is chosen by the operator, not derived from first principles.
  • power-law exponent alpha and scale beta = alpha about 2.2216, beta about 0.9533 in Figure 8
    Estimated from actual next-token generation statistics and used in Theorem 1 to predict cache size, so the predicted count is a fit rather than a parameter-free derivation.
  • minimum response length filter (32 tokens) = 32 tokens
    Instructions whose chatbot responses are shorter than 32 tokens are discarded as noise in Section 5.1. This choice affects measured repetition rates and hit rates.
assumptions (4)
  • domain assumption Next-token probabilities follow a power-law distribution at every node of the V-ary tree.
    Invoked in Theorem 1 and Appendix A. Supported only by one aggregate plot (Figure 8) and not by a mechanistic argument; real LMs have position- and context-dependent distributions.
  • ad hoc to paper A fine-tuned LLM's NLL induces stable spatial locality, grouping likely future instructions near the low-NLL head of the text space.
    This is the paper's core premise, stated in Section 3 but not proven. Section 5.2.4 shows hit rate declines with temporal drift, indicating the assumption holds only approximately.
  • domain assumption The validation-set NLL distribution matches the future test distribution.
    Used when Eq. 1 estimates hit rate from the validation CDF. The distribution-shift experiment shows this assumption degrades over time.
  • ad hoc to paper Algorithm 1 enumerates every instruction with NLL below sigma.
    Needed to equate the built cache with the set {NLL <= sigma}. The algorithm does not specify how many next-token candidates T are expanded at each node, so completeness is assumed rather than shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InstCache: A Predictive Cache for LLM Serving." pith.science (2026). https://pith.science/paper/HTTTMZ6P

@misc{pith2026241113820,
  author       = {Pith},
  title        = {Pith review of: InstCache: A Predictive Cache for LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTTTMZ6P}},
  note         = {Machine review of arXiv:2411.13820}
}
read the original abstract

The revolutionary capabilities of Large Language Models (LLMs) are attracting rapidly growing popularity and leading to soaring user requests to inference serving systems. Caching techniques, which leverage data reuse to reduce computation, offer opportunities to optimize the performance of LLM inference engines. On the one hand, the low-level key-value (KV) cache working at the token level is widely adopted, albeit it incurs significant overhead as request volume grows. On the other hand, instruction-level caching, which stores full instruction-response pairs, is expected to play an increasingly crucial role. However, the high variability in the content and length of instructions make it rare for identical instructions to recur within a short time window, presenting challenges for effective caching instruction-response pairs. To address this challenge, we propose InstCache, a predictive caching mechanism for LLM serving systems. Leveraging the capability of LLMs, we can effectively reorder the representation space of instruction texts and develop a sufficient level of spatial locality. Such spatial locality enables us to predict potential instructions located in a compact region in the space, resulting in an effective caching system at runtime. Experimental results demonstrate that InstCache achieves a 2.3x higher hit rate compared to the upper bound of traditional caching mechanisms on WildChat dataset and reduces the time per output token of vLLM by up to 42.0% and 50.0% on LMSys and Moss datasets, respectively.

Figures

Figures reproduced from arXiv: 2411.13820 by the authors.

Figure 1
Figure 1. Subfigure (a) presents the token length distribution of first-turn instructions across three [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Text space can be represented as paths in a V-ary tree of depth [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Hit rates of InstCache across different datasets. The solid line represents the hit rates [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison between predicted and actual hit rates and instruction counts. The red and blue [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Serving performance of vLLM with InstCache. We use [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Cost of Pre-population on the WildChat dataset. Sub [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: The blue bars represent the probabilities of tokens at different ranks, while the red line [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Evaluation of the impact of train and test ratio. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 35 canonical work pages

  1. [1]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as ...

    arXiv 2021

  2. [2]

    Baeza-Yates and Felipe Saint-Jean

    Ricardo A. Baeza-Yates and Felipe Saint-Jean. A three level search engine index based in query log distribution. In Mario A. Nascimento, Edleno Silva de Moura, and Arlindo L. Oliveira, editors, String Processing and Information Retrieval, 10th International Symposium, SPIRE 2003, Manaus, Brazil, October 8-10, 2003, Proceedings , volume 2857 of Lecture Not...

    work page 2003

  3. [3]

    GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings

    Fu Bang. GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. In Liling Tan, Dmitrijs Milajevs, Geeticka Chauhan, Jeremy Gwin- nup, and Elijah Rippeth, editors, Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023) , pages 212–218, Singapore, December

    work page 2023

  4. [4]

    Jordan, Joseph E

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024....

    work page 2024

  5. [5]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian L...

    arXiv 2024

  6. [6]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    arXiv 2024

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany...

    arXiv 2024

  8. [8]

    Boosting the perfor- mance of web search engines: Caching and prefetching query results by exploiting historical usage data

    Tiziano Fagni, Raffaele Perego, Fabrizio Silvestri, and Salvatore Orlando. Boosting the perfor- mance of web search engines: Caching and prefetching query results by exploiting historical usage data. ACM Trans. Inf. Syst., 24(1):51–78, 2006

    work page 2006

Show all 51 references
  1. [10]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. CoRR, abs/2312.10997, 2023

    2023 arXiv

  2. [11]

    Privacy-aware semantic cache for large language models

    Waris Gill, Mohamed Elidrisi, Pallavi Kalapatapu, Ali Anwar, and Muhammad Ali Gulzar. Privacy-aware semantic cache for large language models. CoRR, abs/2403.19708, 2024

    2024 arXiv

  3. [12]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and...

    2023

  4. [13]

    Predictive caching and prefetching of query results in search engines

    Ronny Lempel and Shlomo Moran. Predictive caching and prefetching of query results in search engines. In Gusztáv Hencsey, Bebo White, Yih-Farn Robin Chen, László Kovács, and Steve Lawrence, editors, Proceedings of the Twelfth International World Wide Web Conference, WWW 2003, ...

    2003

  5. [14]

    von Riedemann, Cong Zhang, and Jiangchuan Liu

    Jiaxing Li, Chi Xu, Feng Wang, Isaac M. von Riedemann, Cong Zhang, and Jiangchuan Liu. SCALM: towards semantic caching for automated chat services with large language models. In 32nd IEEE/ACM International Symposium on Quality of Service, IWQoS 2024, Guangzhou, China, June 19-...

    2024

  6. [15]

    Three-level caching for efficient query processing in large web search engines

    Xiaohui Long and Torsten Suel. Three-level caching for efficient query processing in large web search engines. In Allan Ellis and Tatsuya Hagino, editors,Proceedings of the 14th international 11 conference on World Wide Web, WWW 2005, Chiba, Japan, May 10-14, 2005, pages 257–2...

    2005

  7. [16]

    New caching techniques for web search engines

    Mauricio Marín, Veronica Gil-Costa, and Carlos Gómez-Pantoja. New caching techniques for web search engines. In Salim Hariri and Kate Keahey, editors, Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC 2010, Chicago, Illinois, U...

    2010

  8. [17]

    Markatos

    Evangelos P. Markatos. On caching search engine query results. Comput. Commun., 24(2):137– 143, 2001

    2001

  9. [18]

    Context-based semantic caching for llm applications

    Ramaswami Mohandoss. Context-based semantic caching for llm applications. In 2024 IEEE Conference on Artificial Intelligence (CAI), pages 371–376. IEEE, 2024

    2024

  10. [19]

    Openai chatgpt, 2022

    OpenAI. Openai chatgpt, 2022

    2022

  11. [20]

    Llms for test input generation for semantic caches

    Zafaryab Rasool, Scott Barnett, David Willie, Stefanus Kurniawan, Sherwin Balugo, Srikanth Thudumu, and Mohamed Almorsy Abdelrazek. Llms for test input generation for semantic caches. CoRR, abs/2401.08138, 2024

    2024 arXiv

  12. [21]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tiona...

    2019

  13. [22]

    Fonseca, Wagner Meira Jr., Berthier A

    Patricia Correia Saraiva, Edleno Silva de Moura, Rodrigo C. Fonseca, Wagner Meira Jr., Berthier A. Ribeiro-Neto, and Nivio Ziviani. Rank-preserving two-level caching for scalable search engines. In W. Bruce Croft, David J. Harper, Donald H. Kraft, and Justin Zobel, editors, SI...

    2001

  14. [23]

    The early bird catches the leak: Unveiling timing side channels in LLM serving systems

    Linke Song, Zixuan Pang, Wenhao Wang, Zihao Wang, XiaoFeng Wang, Hongbo Chen, Wei Song, Yier Jin, Dan Meng, and Rui Hou. The early bird catches the leak: Unveiling timing side channels in LLM serving systems. CoRR, abs/2409.20002, 2024

    2024

  15. [24]

    MOSS: an open conversational large language model

    Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanji...

    2024

  16. [25]

    Sharegpt, 2023

    Team. Sharegpt, 2023

    2023

  17. [26]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  18. [27]

    O’Hallaron

    Yinglian Xie and David R. O’Hallaron. Locality in search engine queries and its implications for caching. In Proceedings of IEEE INFOCOM, pages 1238–1247. IEEE Computer Society, 2002

    2002

  19. [28]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  20. [29]

    Chunkattention: Efficient self-attention with prefix-aware KV cache and two-phase partition

    Lu Ye, Ze Tao, Yong Huang, and Yang Li. Chunkattention: Efficient self-attention with prefix-aware KV cache and two-phase partition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    2024

  21. [30]

    Performance of compressed inverted list caching in search engines

    Jiangong Zhang, Xiaohui Long, and Torsten Suel. Performance of compressed inverted list caching in search engines. In Jinpeng Huai, Robin Chen, Hsiao-Wuen Hon, Yunhao Liu, Wei- Ying Ma, Andrew Tomkins, and Xiaodong Zhang, editors,Proceedings of the 17th International Conferenc...

    2008

  22. [31]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Trista...

    2023

  23. [32]

    Wildchat: 1m chatgpt interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  24. [33]

    Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world LLM conversation dataset. In The Twelfth International Con...

    2024

  25. [34]

    kinky date

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Efficiently programming large language models using sglang. CoRR, abs/2312.07104, 2023. 13 A Proof...

    2023 arXiv

  26. [37]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The main claims in the abstract and introduction accurately reflect the paper’s contributions and scope. Guidelines: •...

  27. [38]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations in the Discussion section. Guidelines: • The answer NA means that the paper has no limitation while the answer No means th...

  28. [39]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 19 Justification: The paper provides a complete set of assumptions and includes correct and rigorous proof ...

  29. [40]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  30. [41]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code 20 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: All datasets used in thi...

  31. [42]

    • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We present the experimental d...

  32. [43]

    This experimental setup is also consistent with existing studies[12, 34]

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Our experiments are minimally affected by randomnes...

  33. [44]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide detailed inf...

  34. [45]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research conducted in the paper fully conforms to the NeurIPS Code of Ethic...

  35. [46]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the societal impacts in the Discussion Section. Guidelines: • The answer NA means that there i...

  36. [47]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does...

  37. [48]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We properly cite...

  38. [49]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets. Guidelines: • The answer NA means that the paper does not release new assets. ...

  39. [50]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  40. [51]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  41. [52]

    Answer: [NA] Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  42. [2023]

    Association for Computational Linguistics

  43. [2024]

    OpenReview.net, 2024

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.