Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

IC-Cache: Efficient Large Language Model Serving via In-context Caching

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IC-Cache claims that historical large-model response pairs, prepended as in-context examples, let small LLMs match or exceed large-model quality, yielding 1.4–5.9x higher throughput and 28–71% lower latency.

desk verdict A genuinely useful serving-system idea with a quality-parity claim that is slightly over-sold on reasoning workloads. read the letter →

arxiv 2501.12689 v3 pith:53GA4MRJ submitted 2025-01-22 cs.LG

Yifan Yu , Yu Gan , Nikhil Sarda , Lillian Tsai , Jiaming Shen , Yanqi Zhou , Arvind Krishnamurthy , Fan Lai
show 2 more authors
Henry M. Levy David Culler
This is my paper · ORCID
classification cs.LG
keywords LargelanguagemodelsLLMservingIn-contextlearningSemanticcachingRequestroutingLoadbalancingQuality-efficiencytradeoffCachemanagement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most LLM serving traffic is repetitive: the paper finds that over 70% of real-world requests have a semantically similar counterpart. IC-Cache turns that repetition into a resource by reusing historical large-model request-response pairs as in-context examples for smaller models. The paper claims this 'live capability augmentation' lets small models imitate and sometimes surpass large-model quality, so a load-aware router can offload a large fraction of requests to cheap, fast models without hurting quality. The reported end-to-end result is 1.4–5.9x higher serving throughput and 28–71% lower latency at parity quality. The system is built from three pieces: a two-stage example selector, a bandit-based request router, and a cost-aware cache manager that refines and evicts examples.

What carries the argument

The load-bearing mechanism is the in-context example set, managed by three coordinated components. The Example Selector first narrows the cache by embedding similarity and then scores candidate examples with a lightweight proxy model, adapting the number and combination of examples per request. The Request Router is a contextual multi-armed bandit, a learning algorithm that repeatedly chooses among candidate models to maximize cumulative quality; it adds a smooth load bias based on a moving average of system load, so it offloads aggressively under bursts and conservatively otherwise. The Example Manager replays selected examples offline, keeping the response that maximizes expected offloading gain, and treats the cache as a knapsack to evict low-utility entries. These components together convert the history of requests into a continuously refreshed curriculum for the small model.

What would settle it

Take a held-out reasoning benchmark whose questions resemble cached examples in wording but require a different chain of reasoning; if the small model with IC-Cache examples does not beat its un-augmented baseline in win rate, the capability-transfer claim is falsified. The paper's own Figure 4 already hints at this risk: a 3B model with curated examples scores 34.4% on Math-500-Hard versus 54.5% for a 32B model, so the end-to-end quality claim would survive only because the router sends such requests to the large model. A test that reports win rate split by reasoning difficulty would settle which claim to trust.

Watch

Extended reading notes

Core claim

The paper's central claim is that a cached response is more valuable as a demonstration than as an answer. Instead of returning a similar old response verbatim, IC-Cache prepends high-utility historical request-response pairs to a new request, then lets a contextual-bandit router decide whether the small augmented model or the large model should generate the final answer. On real-world traffic, the augmented small model wins side-by-side quality comparisons against the large model more than half the time on several datasets, meaning quality can be preserved while most requests are served by the cheaper model. The paper reports that this raises throughput by 1.4–5.9x and cuts latency by 28–71% without degrading response quality, and that the benefit survives distribution shift, model upgrades, tight cache budgets, and even differentially private synthetic example pools.

Load-bearing premise

The load-bearing premise is that prepending historical large-model request-response pairs transfers genuine reasoning competence, not just output style, to a small model for a substantial share of requests. If that gives way, the quality-parity claim collapses.

Editorial extensions

If this is right

  • Service providers can meet a fixed quality bar with substantially fewer large-model calls, because repeated and similar traffic is absorbed by the cache and served by small models.
  • Semantic caching becomes safer: rather than returning an old similar answer directly, the system uses it as guidance, which the paper shows raises the quality of cache hits by up to 28%.
  • The example pool saturates quickly; tens of thousands of examples, under 20 MB of plaintext, already approach full benefit, so the cache's memory overhead is small.
  • The router's load bias doubles as an autoscaling signal, letting a deployment react to bursty load by shifting traffic toward smaller models.
  • The approach composes with supervised fine-tuning and retrieval-augmented generation, improving both rather than replacing them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own hard-math numbers suggest the capability transfer is partial: a 3B model with curated examples reaches 34.4% on Math-500-Hard where a 32B model reaches 54.5%; if that gap holds generally, the quality-parity claim applies to the cache-rich, easier fraction of traffic rather than to all requests.
  • A direct test of whether the gain is genuine reasoning transfer or style imitation would ablate the content of the example responses while preserving their structure and format, and measure how much of the small model's win rate survives.
  • The same input-composition approach could plausibly stack with prefix caching or speculative decoding, since it only changes what tokens are prepended, but the paper does not evaluate those compounding gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes IC-Cache, a serving-layer system that stores historical request-response pairs produced by large LLMs and prepends the most useful ones as in-context examples to requests handled by smaller LLMs. The system combines a two-stage example selector, a contextual-bandit request router with load-based bias, and a cost-aware example replay/cache manager. The authors evaluate IC-Cache on millions of requests across Gemini, Qwen, Gemma, and Phi models, reporting 1.4-5.9x throughput improvements and 28-71% latency reductions while maintaining response quality relative to the larger model. The central claim is that this 'in-context caching' enables smaller models to imitate and sometimes exceed the compositional abilities of larger models, allowing safe offloading of a large fraction of traffic.

Significance. If the quality-parity claim holds, this is a significant systems contribution: it repurposes the natural stream of historical traffic as live in-context demonstrations, is complementary to prefix caching, semantic caching, and routing systems, and is evaluated at unusually large scale across several model families and workloads. The paper also gives credit for its thoughtful treatment of privacy (DP synthesis), adaptation to load bursts, and integration with vLLM, HuggingFace Runtime, and LangChain. The main weaknesses are that the 'exceed and imitate' capability claim is not supported on reasoning tasks by the paper's own data, the quality evaluation relies solely on LLM-as-a-judge without error bars, and no code or artifacts are released.

major comments (5)
  1. [§2.3, Figure 4] The load-bearing claim that small models 'imitate and even exceed' larger models on compositional abilities is not supported by the paper's own data: on Math-500-Hard, Qwen-3B with curated IC examples reaches 34.4% accuracy versus 54.5% for Qwen-32B. The end-to-end quality results in §6.2 and Figures 12-17 cover conversational and QA datasets (LMSys-Chat, MS MARCO, Natural Questions, Orca), not math or code reasoning. Please add an end-to-end evaluation on a reasoning workload that reports both the router's offloading ratio and a correctness-based quality metric (e.g., exact-match or verifier accuracy) for the small model, the large model, and IC-Cache; without it, the 'without hurting response quality' claim is only established for style-sensitive tasks.
  2. [§6.1-6.2] The quality-parity result rests entirely on LLM-as-a-judge pairwise scores, and the paper reports a single run per configuration with no error bars, seeds, or statistical tests. Since the win-rate definition counts ties as 0.5 and the system is evaluated near parity, small evaluation noise can change the conclusion. Please report multiple evaluation seeds or bootstrap confidence intervals, and include at least one objective metric (e.g., code compilation/pass rate, math exact match) on datasets with ground truth, together with the MT-Bench human-alignment numbers for the actual judge-model combination used in each experiment.
  3. [§6.2, Figure 13] The headline throughput gains (1.4-5.9x) are not reproducible from the text: the paper does not state how throughput is measured (e.g., goodput under an SLO, requests/s at what batch and concurrency), how the Microsoft trace is scaled to the 16-A100 cluster, or what fraction of requests are offloaded in the reported end-to-end runs. Please specify the measurement protocol and report offloading ratios for each dataset/model pair, especially for reasoning-heavy workloads where the capability transfer is weaker.
  4. [§4.3 (Eq. 1) and Figure 11] The replay gain formula G(e) is introduced as the basis for cost-aware replay, but the paper does not validate that this proxy predicts actual offloading benefit. Figure 11 shows that replay improves response quality, which is a different claim. Please add an ablation that compares replay selection by G(e) against alternatives (e.g., random, frequency-only, quality-only) in terms of end-to-end throughput and quality, and report the sensitivity to the constants in the formula.
  5. [Appendix A.2] The 'formal' sample-complexity analysis is not rigorous as written. Theorem 1's proof sets C=2K and applies Hoeffding without stating boundedness of utilities; Theorem 4 concludes convergence to the 'minimum-cost model j' but relies on a 'sufficiently large' λ0, which is a free parameter, so the result is conditional. Since §4.2 advertises a formal analysis, either correct these proofs or soften the claim to an empirical observation.
minor comments (6)
  1. [§6 summary bullet] The text says 'without comprising response quality'; this should be 'without compromising response quality'.
  2. [Table 1 and Figure 4] Please clarify the relationship between 'Math500-Level5' (listed as 7,500 requests in Table 1) and 'Math-500-Hard' used in §2.3; the naming and sizes are inconsistent.
  3. [Figures 3, 10, 12, 13, 16, 19] The PDF text contains glyph artifacts such as '/glyph1197atural' and '/glyph1197umber'; these need to be repaired before publication.
  4. [Figure 10 and captions] The paper uses 'MS MACRO' in some places and 'MS MARCO' in others; please use a single consistent name.
  5. [§2.3] The grounding in in-context learning theory [32,72] is invoked without stating which theorem or result implies that prepended historical Q-A pairs transfer reasoning competence rather than surface format. Please cite a specific result or present a small controlled experiment that separates reasoning transfer from formatting imitation.
  6. [§5] The paper describes an implementation but does not release code or artifacts; given the systems claims, an artifact (even a VM image or Dockerfile) would materially strengthen reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: IC-Cache is an empirical systems paper evaluated against external baselines; its components are heuristics, and no headline claim reduces to a fitted input or self-citation.

full rationale

The paper's argument chain is empirical rather than derivational: it observes request similarity, hypothesizes that historical request-response pairs can serve as in-context examples, designs an example selector, router, and manager, and then measures throughput, latency, and quality against external baselines such as RouteLLM, LongRAG, and semantic caching. No load-bearing prediction is obtained by fitting a parameter to the target metric. The replay-gain formula G(e) = (1 - normalized_response_quality) x normalized_model_cost is an explicitly stated prioritization heuristic, not a quantity fitted to reproduce the reported gains; the bandit sample-complexity theorems are standard concentration and union-bound arguments; and the load-bias theorem is a direct consequence of the softmax score definition, not evidence for the end-to-end serving claims. The paper does not rely on self-citations to justify its central premise: in-context learning is grounded in external references, and the router's benefit is demonstrated against independent systems baselines. The only mild self-referential aspect is that the proxy helpfulness model is trained on preference-style feedback resembling the evaluation signal used for quality measurement, but the paper nowhere states that the LLM-as-judge evaluation score is the proxy's training target, so this remains an evaluation-validity concern rather than a demonstrated circular step. Likewise, the paper's own Figure 4 shows the reasoning transfer is incomplete (Qwen-3B with IC examples reaches 34.4% versus 54.5% for Qwen-32B on Math-500-Hard), which weakens the claim that small models exceed larger ones, but a correctness or scope limitation is not circularity. Overall the claim structure is self-contained and externally checked.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The headline efficiency numbers come from an assembled system with many hand-set or online-tuned parameters (example count, utility threshold, load threshold, replay cutoff, decay, similarity threshold). The largest epistemic load is carried by the ICL-transfer assumption and by the absence of released artifacts, which makes it hard to separate the method's contribution from the tuning procedure.

free parameters (7)
  • Number of in-context examples = 5 in Fig. 4; adaptive utility threshold otherwise
    Hand-picked count in the motivating experiment; the online system uses a dynamic threshold whose initial value and update rule are unspecified (§4.1).
  • Utility threshold for example selection = not specified
    §4.1 selects the threshold online by sampling requests and picking the value that maximizes performance; no update rule or starting value is given.
  • Load threshold for router = not specified
    §4.2 compares EMA load to a 'desired operational threshold'; the threshold value is left open.
  • Load-bias constants λ0, γ = not specified
    Appendix A.2 score S_i(L)=μ_i−λ0 tanh(γL) C_i; proof assumes λ0 large enough to dominate utility differences.
  • Example replay cutoff = determined online
    §4.3 stops replay when potential gain falls below a cutoff 'determined online'; the computation is not specified.
  • Cache utility decay factor = 0.9 per hour
    §4.3 decays example gains with factor 0.9 per hour; arbitrary but standard.
  • Semantic-similarity threshold = 0.8
    §2.3 uses cosine similarity >0.8 to claim 'over 70% of requests have semantically similar counterparts'; threshold is arbitrary and affects the motivation.
assumptions (5)
  • domain assumption In-context learning transfers reasoning capability, not just style, from demonstrations to small models
    §2.3 grounds the approach in ICL theory [28], but the theory covers surface skills; the paper's own math results (Fig 4) show limited transfer.
  • domain assumption User preference feedback (thumbs up/down, or pairwise) is a reliable, available quality signal for router and proxy training
    §4.1 and §4.2 assume feedback is collected and sufficiently denoised; noisy delayed feedback is not modeled in the evaluation.
  • standard math The sample-complexity analysis relies on standard concentration results (Hoeffding, union bound) and Thompson sampling regret bounds [19]
    Appendix A.2 uses these to bound router convergence.
  • ad hoc to paper Replay gain G(e) = (1 - normalized_response_quality) × normalized_model_cost is an adequate proxy for future replay benefit
    §4.3 introduces this multiplicative heuristic without derivation or comparison to alternative ranking metrics.
  • domain assumption Semantic similarity via T5 embeddings is a sufficient prefilter for example relevance at scale
    §4.1 uses dense embeddings as stage-1 filter despite weak correlation with helpfulness (Fig 7); the paper argues it remains useful as a cheap filter.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IC-Cache: Efficient Large Language Model Serving via In-context Caching." pith.science (2026). https://pith.science/paper/53GA4MRJ

@misc{pith2026250112689,
  author       = {Pith},
  title        = {Pith review of: IC-Cache: Efficient Large Language Model Serving via In-context Caching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/53GA4MRJ}},
  note         = {Machine review of arXiv:2501.12689}
}
read the original abstract

Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 70% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge transfer among requests. However, naively caching and reusing past responses leads to a big quality drop. In this paper, we introduce IC-Cache, a caching system that enables live LLM capability augmentation to improve serving efficiency: by leveraging historical request-response pairs from larger models as in-context examples, IC-Cache empowers small LLMs to imitate and even exceed the compositional abilities (e.g., reasoning) of their larger counterparts, enabling selective offloading of requests to reduce cost and latency. Achieving this live augmentation at scale introduces intricate trade-offs between response quality, latency, and system throughput. For a new request, IC-Cache efficiently selects similar, high-utility examples to prepend them to the new request's input. At scale, it adaptively routes requests across LLMs of varying capabilities, accounting for response quality and serving loads. IC-Cache employs a cost-aware cache replay mechanism that refines example quality offline to maximize online cache utility and efficiency. Evaluations on millions of realistic requests demonstrate that IC-Cache improves LLM serving throughput by 1.4-5.9x and reduces latency by 28-71% without hurting response quality.

Figures

Figures reproduced from arXiv: 2501.12689 by the authors.

Figure 1
Figure 1. Quality-Efficiency Trade-off of Gemini, DeepSeek-R1, and Qwen Models. The average score represents a seven-point scale ranging from -3 (significantly worse) to 3 (significantly bet￾ter), evaluating the smaller model’s performance relative to the larger model. Self-comparison scores (large vs. large) are omitted as they would yield 0 by definition. and previously generated tokens, until reaching either a pre￾defined … view at source ↗
Figure 4
Figure 4. (a) Incorporating in-context (IC) examples improves response quality, whereas adding random examples degrades it. (b) Prepending examples introduces a slight TTFT overhead, but it remains lower than that of querying larger models. augmentation for smaller models. This insight is grounded in in-context learning theory [28], which demonstrates that LLMs can learn from high-quality examples, enabling on-the￾fly knowled… view at source ↗
Figure 5
Figure 5. IC-Cache overview and request execution flow. data distributions, optimizing for the optimal accuracy￾latency-cost trade-off in flight? 3 IC-Cache Overview Extending the prevalent semantic caching architecture in today’s LLM deployments, we introduce IC-Cache, an in￾context caching LLM serving system. It exploits historical requests for live LLM capability augmentation, adaptively offloading requests to reduce servi… view at source ↗
Figures from the paper (13 more)
Figure 7
Figure 7. Figure 7: Pearson correlation (∈ [-1, 1]) between example similarity and its helpfulness is weak. 1 2 3 0.243 0.081 0.123 0.134 0.321 0.165 1 Embed input request 4 2 Identify similar examples 3 4 Calculate utility Select examples [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 10
Figure 10. Figure 10: Example access ex￾hibits long-tail distribution. w/o distillation w/ distillation 0.6 0.4 0.2 0.0 Average Score -0.26 -0.20 -0.42 -0.19 -0.66 -0.41 Open Orca Math Reasoning Code Generation [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 12
Figure 12. Figure 12: IC-Cache enables offloading more requests to small models, improving serving throughput (a)-(b), while achieving better response latency (c)-(d) and response quality (e)-(h). Task Dataset Example Size Request Size Conversation Alpaca[66] 32,392 1,800 lmsys-chat-1m [79…
Figure 15
Figure 15. Figure 15: IC-Cache augments supervised fine-tuning (SFT) and RAG deployments. 1 2 3 4 5 6 Normalized Throughput 45 50 55 60 Win Rate(%) MS MACRO IC-Cache IC-Cache w/o Router IC-Cache w/o (Router & Retriever) 1 2 3 4 5 6 Normalized Throughput 46 48 50 52 Alpaca IC-Cache IC-Cache…
Figure 17
Figure 17. Figure 17: IC-Cache improves the quality of generation across different tasks for Gemini, Qwen, and DeepSeek series models. 2.43 Gemma-2 -2B Gemma-2 -27B Gemma-2-2B w/ IC-Cache 2 4 6 8 Zero-load latency (s) 2.66 2.57 8.94 0.06 0.02 0.06 1.154 Gemma-2 -2B Gemma-2 -27B Gemma-2-2B …
Figure 19
Figure 19. Figure 19: IC-Cache delivers improvement under different the example cache sizes. Gemma-2 -2b Gemma-2 -2b + IC Gemma-2 -27b 0 10 20 30 40 Request Completion Time(s) Light Load Medium Load Heavy Load [PITH_FULL_IMAGE:figures/full_fig_p012_19.png]
Figure 21
Figure 21. Figure 21: IC-Cache with DP synthetic example pool brings marginal quality degradation. 0 5 10 15 20 25 30 Time (m) 0 10 20 30 40 50 60 70 80 Number of Requests [PITH_FULL_IMAGE:figures/full_fig_p012_21.png]
Figure 23
Figure 23. Figure 23: System Prompt without IC-Cache for conversational tasks. SYSTEM PROMPT WITH IC-CACHE [System] You are a helpful AI Assistant that follows users’ instructions carefully. Write a response that appropriately completes the request. Provide necessary details or explanation…
Figure 24
Figure 24. Figure 24: System Prompt with IC-Cache for conversational tasks. Proof: A failure occurs if any of the 𝑘 (𝑁 −𝑘) critical pairs (one model from the top-k, one from outside) are incorrectly ordered. Using a union bound, the required error rate for any single pair is 𝛿/(𝑘 (𝑁 − 𝑘)).…
Figure 25
Figure 25. Figure 25: Autorater system prompt for side-by-side quality evaluation. Theorem 4: Let the score (logit) for selecting model 𝑖 be defined by its utility and a load-dependent cost penalty: 𝑆𝑖(𝐿) = 𝜇𝑖 − 𝜆0 tanh(𝛾𝐿)𝐶𝑖 (9) where 𝜆0,𝛾 > 0 are constants. Let the selection policy be a …
Figure 26
Figure 26. Figure 26: An example of how IC-Cache works to improve response quality of Gemma-2-2B using a retrieved example from Gemma-2-27B. not being semantically identical though, provide related in￾formation from different perspectives to answer the question even better than the large m…
Figure 27
Figure 27. Figure 27: Model quality comparison on five text generation tasks with three different model families. With IC-Cache, the quality of a smaller model can be significantly boosted. 3 2 1 0 1 2 3 Average Score 0.0 0.5 1.0 1.5 Density Natural Questions Baseline IC-Cache [PITH_FULL_…
Figure 28
Figure 28. Figure 28: IC-Cache improves response score on nat￾ural question using Phi-3. comparable to GPT-4, further supporting the reliability of our LLM-as-a-Judge policy. B More Evaluation Results B.1 Response Quality Improvement IC-Cache brings significant improvements in response qua…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems

    cs.MA 2025-09 conditional novelty 6.0 of 10

    CORRECT distills recurring multi-agent failure patterns into reusable error schemas and retrieves them at inference time to localize the decisive error step more accurately than judging or fine-tuning baselines.

Reference graph

Works this paper leans on

92 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://developers.google.com/ search/docs/appearance/ai-overviews

    AI Overviews and Your Website. https://developers.google.com/ search/docs/appearance/ai-overviews

  2. [2]

    https://aws.amazon.com/codewhisperer/

    Amazon codewhisperer. https://aws.amazon.com/codewhisperer/

  3. [3]

    https://claude.ai/

    Anthropic claude. https://claude.ai/

  4. [4]

    https://character.ai/

    Character ai. https://character.ai/

  5. [5]

    https://openai.com/index/ introducing-deep-research/

    ChatGPT: Introducing deep research. https://openai.com/index/ introducing-deep-research/

  6. [6]

    https://api-docs.deepseek.com/guides/ kv_cache

    DeepSeek Context caching. https://api-docs.deepseek.com/guides/ kv_cache

  7. [7]

    https: //github.com/deepseek-ai/open-infra-index/blob/main/ 202502OpenSourceWeek/day_6_one_more_thing_ deepseekV3R1_inference_system_overview.md

    DeepSeek-V3/R1 Inference System Overview. https: //github.com/deepseek-ai/open-infra-index/blob/main/ 202502OpenSourceWeek/day_6_one_more_thing_ deepseekV3R1_inference_system_overview.md

  8. [8]

    https: //developers.googleblog.com/en/gemini-15-flash-8b-is-now- generally-\available-for-use/

    Gemini 1.5 Flash-8B is now production ready. https: //developers.googleblog.com/en/gemini-15-flash-8b-is-now- generally-\available-for-use/

Show all 92 references
  1. [9]

    https://ai.google.dev/gemini-api/docs/ caching?lang=python

    Gemini Context caching. https://ai.google.dev/gemini-api/docs/ caching?lang=python

  2. [10]

    https://github.com/features/copilot/

    Github copilot. https://github.com/features/copilot/

  3. [11]

    https://www

    Helicone:Open source LLM observability platform. https://www. helicone.ai/status/provider/Google

  4. [12]

    https://huggingface.co/spaces/lmarena-ai/chatbot-arena- leaderboard

    https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leaderboard. https://huggingface.co/spaces/lmarena-ai/chatbot-arena- leaderboard

  5. [13]

    https://huggingface.co/docs/ api-inference/index

    HuggingFace Serverless Inference API. https://huggingface.co/docs/ api-inference/index

  6. [14]

    https://github

    LangChain: Build context-aware reasoning applications. https://github. com/langchain-ai/langchain

  7. [15]

    https://microsoft.github.io/msmarco/

    MS MARCO. https://microsoft.github.io/msmarco/

  8. [16]

    https: //github.com/explosion/spaCy

    spaCy: Industrial-strength Natural Language Processing (NLP). https: //github.com/explosion/spaCy

  9. [17]

    https://www.databricks.com/blog/building-cost-optimized- chatbot-semantic-caching, 2024

    Databricks: Building a cost-optimized chatbot with semantic caching. https://www.databricks.com/blog/building-cost-optimized- chatbot-semantic-caching, 2024

    2024

  10. [18]

    Sarathi: Efficient llm in- ference by piggybacking decodes with chunked prefills

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhar- gav S Gulavani, and Ramachandran Ramjee. Sarathi: Efficient llm in- ference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023

    2023 arXiv

  11. [19]

    Analysis of thompson sampling for the multi-armed bandit problem

    Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Shie Mannor, Nathan Srebro, and Robert C. Williamson, editors, Proceedings of the 25th Annual Conference on Learning Theory, volume 23 of Proceedings of Machine Learning Resear...

  12. [20]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023

    2023 arXiv

  13. [21]

    Gptcache: An open-source semantic cache for llm applica- tions enabling faster answers and cost savings

    Fu Bang. Gptcache: An open-source semantic cache for llm applica- tions enabling faster answers and cost savings. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 212–218, 2023

    2023

  14. [22]

    Findings of the 2016 conference on machine translation

    Ondˇrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, IC-Cache: Efficient Large Language Model Serving via In-context Caching SOSP ’25, October 13–16, 2025, Seoul, Republic of Korea Varvara Logach...

    2025

  15. [23]

    JAX: compos- able transformations of Python+NumPy programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: compos- able transformations of Python+NumPy programs, 2018

    2018

  16. [24]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  17. [25]

    Are more llm calls all you need? towards scaling laws of compound inference systems

    Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more llm calls all you need? towards scaling laws of compound inference systems. In NeurIPS, 2024

    2024

  18. [26]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023

    2023 arXiv

  19. [27]

    Learning semantic similarity in a continuous space

    Michel Deudon. Learning semantic similarity in a continuous space. In NeurIPS, 2018

    2018

  20. [28]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning. In Arxiv: 2301.00234, 2023

    2023 arXiv

  21. [29]

    Gemini: A family of highly capable multimodal models

    Gemini Team Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  22. [30]

    Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Miao Liu, Michael L

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2025 arXiv

  23. [31]

    Apple intelligence foundation lan- guage models

    Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap, Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Peebles, Ke Ye, Mark Lee,...

    2024

  24. [32]

    A theory of emergent in-context learning as implicit structure induction

    Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction. In arXiv: 2303.07971, 2023

    2023 arXiv

  25. [33]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181, 2024

    2024 arXiv

  26. [34]

    An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers

    Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers. arXiv preprint arXiv:2403.02839, 2024

    2024 arXiv

  27. [35]

    Evaluation of best-of-n sampling strategies for language model alignment

    Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of best-of-n sampling strategies for language model alignment. In arXiv: 2502.12668, 2025

    2025 arXiv

  28. [36]

    Active retrieval augmented generation

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983 , 2023

    2023 arXiv

  29. [37]

    MegaScale: Scaling large language model training to more than 10,000 GPUs

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi...

  30. [38]

    Billion-scale similarity search with GPUs

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 2019

    2019

  31. [39]

    Tanh works better with asymmetry

    Dongjin Kim, Woojeong Kim, and Suhyun Kim. Tanh works better with asymmetry. In NeurIPS, 2023

    2023

  32. [40]

    Tree of clarifications: Answering ambiguous ques- tions with retrieval-augmented large language models

    Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. Tree of clarifications: Answering ambiguous ques- tions with retrieval-augmented large language models. arXiv preprint arXiv:2310.14696, 2023

    2023 arXiv

  33. [41]

    Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  34. [42]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023

    2023

  35. [43]

    Auto-gda: Automatic domain adaptation for efficient grounding verification in retrieval aug- mented generation

    Tobias Leemann, Periklis Petridis, Giuseppe Vietri, Dionysis Manousakas, Aaron Roth, and Sergul Aydore. Auto-gda: Automatic domain adaptation for efficient grounding verification in retrieval aug- mented generation. arXiv preprint arXiv:2410.03461, 2024

    arXiv

  36. [44]

    Retrieval-augmented generation for knowledge- intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Sys- t...

    2020

  37. [45]

    Dpsynthe- sizer: differentially private data synthesizer for privacy preserving data sharing

    Haoran Li, Li Xiong, Lifan Zhang, and Xiaoqian Jiang. Dpsynthe- sizer: differentially private data synthesizer for privacy preserving data sharing. VLDB, 2014

    2014

  38. [46]

    Schapire

    Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommen- dation. In WWW, 2010. IC-Cache: Efficient Large Language Model Serving via In-context Caching SOSP ’25, October 13–16, 2025, Seoul, Republic of Korea

    2010

  39. [47]

    Gon- zalez, and Ion Stoica

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gon- zalez, and Ion Stoica. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In OSDI, 2023

    2023

  40. [48]

    Adaserve: Slo- customized llm serving with fine-grained speculative decoding

    Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang, Zhihao Zhang, Zhuom- ing Chen, Sean Lai, Xupeng Miao, and Zhihao Jia. Adaserve: Slo- customized llm serving with fine-grained speculative decoding. In arXiv: 2501.12162, 2025

    2025 arXiv

  41. [49]

    Openorca: An open dataset of gpt augmented flan reasoning traces

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/ OpenOrca, 2023

    2023

  42. [50]

    Parrot: Efficient serving of llm-based applications with semantic variable

    Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Parrot: Efficient serving of llm-based applications with semantic variable. In OSDI, 2024

    2024

  43. [51]

    Andes: Defining and enhancing quality-of- experience in llm-based text streaming services

    Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. Andes: Defining and enhancing quality-of- experience in llm-based text streaming services. 2024

    2024

  44. [52]

    In-context learning with retrieved demonstrations for language models: A survey

    Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. In-context learning with retrieved demonstrations for language models: A survey. arXiv preprint arXiv:2401.11624, 2024

    2024 arXiv

  45. [53]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  46. [54]

    MS MARCO: A human generated machine reading comprehension dataset

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268, 2016

    2016 arXiv

  47. [55]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route llms with preference data. In arXiv: 2406.18665, 2024

    2024 arXiv

  48. [56]

    Training language models to follow instruc- tions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. Advances in neural information processing systems, 35:...

    2022

  49. [57]

    Splitwise: Efficient gen- erative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient gen- erative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) , pages 118–...

    2024

  50. [58]

    Conserve: Harvesting gpus for low-latency and high-throughput large language model serving

    Yifan Qiao, Shu Anzai, Shan Yu, Haoran Ma, Yang Wang, Miryung Kim, and Harry Xu. Conserve: Harvesting gpus for low-latency and high-throughput large language model serving. In arXiv: 2410.01228, 2024

    2024 arXiv

  51. [59]

    Modserve: Scal- able and resource-efficient large multimodal model serving

    Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Inigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ramachandran Ramjee, and Rodrigo Fonseca. Modserve: Scal- able and resource-efficient large multimodal model serving. In arXiv: 2502.00937, 2025

    2025

  52. [60]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Explor- ing the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020

    2020

  53. [61]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Informa- tion Retrieval, 3(4):333–389, 2009

    2009

  54. [62]

    Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. arXiv preprint arXiv:2305.15294, 2023

    2023 arXiv

  55. [63]

    Gonzalez, and Ion Stoica

    Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. Fairness in serving large language models. In OSDI, 2024

    2024

  56. [64]

    A statistical interpretation of term specificity and its application in retrieval

    Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation , 28(1):11–21, 1972

    1972

  57. [65]

    Hygen: Efficient llm serving via elastic online-offline request co-location

    Ting Sun, Penghan Wang, and Fan Lai. Hygen: Efficient llm serving via elastic online-offline request co-location. In arXiv: 2501.14808, 2025

    2025

  58. [66]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford_alpaca, 2023

    2023

  59. [67]

    Gemma 2: Im- proving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mes- nard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Im- proving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  60. [68]

    Jaylen Wang, Daniel S. Berger, Fiodar Kazhamiaka, Celine Irvene, Chaojie Zhang, Esha Choukse, Kali Frost, Rodrigo Fonseca, Brijesh Warrier, Chetan Bansal, Jonathan Stern, Ricardo Bianchini, and Ak- shitha Sriraman. Designing cloud servers for lower carbon. In ISCA, 2024

    2024

  61. [69]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebas- tian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  62. [70]

    Fast dis- tributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast dis- tributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023

    2023 arXiv

  63. [71]

    dLoRA: Dynamically orchestrating requests and adapters for LoRA LLM serving

    Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. dLoRA: Dynamically orchestrating requests and adapters for LoRA LLM serving. In OSDI, 2024

    2024

  64. [72]

    Why in-context learning models are good few-shot learners? In ICLR, 2025

    Shiguang Wu, Yaqing Wang, and Quanming Yao. Why in-context learning models are good few-shot learners? In ICLR, 2025

    2025

  65. [73]

    Powerinfer-2: Fast large language model inference on a smart- phone

    Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. Powerinfer-2: Fast large language model inference on a smart- phone. arXiv preprint arXiv:2406.06282, 2024

    2024 arXiv

  66. [74]

    Cacheblend: Fast large language model serving with cached knowledge fusion

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving with cached knowledge fusion. arXiv preprint arXiv:2405.16444, 2024

    2024 arXiv

  67. [75]

    Generating data for symbolic language with large language models

    Jiacheng Ye, Chengzu Li, Lingpeng Kong, and Tao Yu. Generating data for symbolic language with large language models. arXiv preprint arXiv:2305.13917, 2023

    2023 arXiv

  68. [76]

    Compositional exemplars for in-context learning

    Jiacheng Ye, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. ICML, 2023

    2023

  69. [77]

    Orca: A distributed serving system for{Transformer- Based} generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for{Transformer- Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022

    2022

  70. [78]

    Longrag: A dual-perspective retrieval- augmented generation paradigm for long-context question answering

    Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, and Jie Tang. Longrag: A dual-perspective retrieval- augmented generation paradigm for long-context question answering. EMNLP, 2024

    2024

  71. [79]

    Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv: 2309.11998, 2024

    2024 arXiv

  72. [80]

    Judging llm-as-a-judge with mt-bench and chatbot arena.NeurIPS, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.NeurIPS, 2023. SOSP ’25, October 13–16, 2025, Seoul, Republic of Korea Yu et al

    2023

  73. [81]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Effi- cient execution of structured language model programs. In ASPLOS, 2023

    2023

  74. [82]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401.09670, 2024

    2024 arXiv

  75. [83]

    Distillspec: Improving speculative decoding via knowledge distillation, 2024

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Francois Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation, 2024. IC-Cache: Efficient Large Language Model Serving via I...

    2024

  76. [84]

    We can bound this with the union bound: 𝑃(ˆ𝑖𝑇 ≠ 1)≤ 𝑁∑︁ 𝑖=2 𝑃(𝜇𝑖 >𝜇1) (2)

    Union Bound: The overall probability of failure is the probability that any suboptimal model 𝑖’s estimated utility 𝜇𝑖 is greater than the best model’s estimated utility𝜇1. We can bound this with the union bound: 𝑃(ˆ𝑖𝑇 ≠ 1)≤ 𝑁∑︁ 𝑖=2 𝑃(𝜇𝑖 >𝜇1) (2)

  77. [85]

    We can state this more formally for the number of comparisons,𝑚𝑖(𝑇), for a sufficiently large T: 𝑚𝑖(𝑇)≥ 𝐾 log(𝑇) Δ2 𝑖 (3) where𝐾 is a positive constant

    Number of Samples: For Thompson sampling, each suboptimal model 𝑖 is sampled approximately 𝑂(log(𝑇)/Δ2 𝑖) times in𝑇 rounds [19]. We can state this more formally for the number of comparisons,𝑚𝑖(𝑇), for a sufficiently large T: 𝑚𝑖(𝑇)≥ 𝐾 log(𝑇) Δ2 𝑖 (3) where𝐾 is a positive constant

  78. [86]

    Let the em- pirical difference be ˆΔ𝑖(𝑚) = 𝜇1−𝜇𝑖 after𝑚 compar- isons, whose true mean is the utility gap Δ𝑖 =𝑈1−𝑈𝑖

    Applying Hoeffding’s Inequality: Let’s analyze the probability of a single error, 𝑃(𝜇𝑖 > 𝜇1). Let the em- pirical difference be ˆΔ𝑖(𝑚) = 𝜇1−𝜇𝑖 after𝑚 compar- isons, whose true mean is the utility gap Δ𝑖 =𝑈1−𝑈𝑖. The error event 𝜇𝑖 > 𝜇1 is equivalent to ˆΔ𝑖(𝑚) < 0. This can be w...

  79. [87]

    Now we substitute the bound for the number of samples 𝑚𝑖(𝑇): 𝑃(𝜇𝑖 >𝜇1)≤ exp −2 𝐾 log(𝑇) Δ2 𝑖 Δ2 𝑖 (5) Which simplifies to: 𝑃(𝜇𝑖 >𝜇1)≤ 𝑒−2𝐾 log(𝑇) =𝑒log(𝑇−2𝐾) =𝑇−2𝐾 (6)

  80. [88]

    Substitut- ing this result back into the union bound from step 1 gives the final bound

    Setting𝐶 = 2𝐾 ,we have𝑃(𝜇𝑖 >𝜇1)≤ 𝑇−𝐶. Substitut- ing this result back into the union bound from step 1 gives the final bound. Theorem 2: To identify the best model with probability at least 1−𝛿, the hybrid Thompson sampling approach requires: 𝑇 =𝑂 𝑁 Δ2 min log 𝑁 𝛿 ! (7) compar...

    2025

  81. [89]

    Limiting Behavior of Tanh:As the load𝐿→∞ , the hyperbolic tangent term approaches its maximum value: lim𝐿→∞ tanh(𝛾𝐿) = 1

  82. [90]

    Ratio of Probabilities: Consider the ratio of probabili- ties between any model𝑘 ≠ 𝑗 (where𝐶𝑘 >𝐶𝑗) and the minimum-cost model 𝑗: 𝑃𝑘(𝐿) 𝑃𝑗(𝐿) = exp(𝑆𝑘(𝐿)) exp(𝑆𝑗(𝐿)) = exp(𝑆𝑘(𝐿)− 𝑆𝑗(𝐿)) (10) Substituting the score definition: 𝑆𝑘(𝐿)− 𝑆𝑗(𝐿) =(𝜇𝑘−𝜇𝑗)− 𝜆0 tanh(𝛾𝐿)(𝐶𝑘−𝐶𝑗) (11)

  83. [91]

    Asymptotic Limit: We take the limit of this difference as𝐿→∞ : lim 𝐿→∞ (𝑆𝑘(𝐿)− 𝑆𝑗(𝐿)) =(𝜇𝑘−𝜇𝑗)− 𝜆0(𝐶𝑘−𝐶𝑗) (12) Since𝐶𝑘 >𝐶𝑗, the term(𝐶𝑘−𝐶𝑗) is a positive constant. For a sufficiently large cost-sensitivity parameter 𝜆0, the negative cost term−𝜆0(𝐶𝑘−𝐶𝑗) will dominate the consta...

  84. [92]

    As the exponent approaches−∞, the ratio of probabili- ties approaches zero: lim 𝐿→∞ 𝑃𝑘(𝐿) 𝑃𝑗(𝐿) = 0 for all𝑘 ≠ 𝑗 (13) Since Í𝑁 𝑖=1𝑃𝑖(𝐿) = 𝑃𝑗(𝐿)+ Í 𝑘≠𝑗𝑃𝑘(𝐿) = 1, and the ratio of every other model’s probability to model 𝑗’s probability goes to zero, it must be thatlim𝐿→∞𝑃𝑗(𝐿) =...

    2025

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.