Pith. sign in

REVIEW 1 major objections 46 references

WISERouter: LLM Routing with Workload Budget Constraint

T0 review · 1 major / 0 minor · reviewed 2026-07-30 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM routing under a single workload budget can be cast as a constrained contextual bandit and solved with adaptive linear programming, meeting the global cost cap without per-budget retraining.

desk verdict Useful workload-budget routing with a real but incomplete end-to-end regret story—the ordering lemma is fine; equating it to known-stats ALP is not. read the letter →

arxiv 2607.23765 v1 pith:TYHCSVL6 submitted 2026-07-26 cs.LG cs.AI

Yifei Li , Zihui Gao , Laks V.S. Lakshmanan This is my paper · ORCID
classification cs.LGcs.AI
keywords LLMroutingworkloadbudgetconstrainedcontextualbanditadaptivelinearprogrammingonlinelearningcost-awareinferenceregretbound
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using the strongest language model for every query is too expensive at scale, yet fixed per-query cost caps and dense supervised training leave money on the table and require expensive data. This paper argues that routing should instead maximize total quality subject to one expected cost budget over the whole workload. It turns that problem into a constrained contextual bandit: each query embedding is a context, each model is an arm, and quality and cost are the observed reward and spend. After clustering embeddings into a finite context set, Adaptive Linear Programming converts remaining budget into a per-step average and picks a model without retraining when the budget changes. An offline version uses historical statistics; an online version explores under the same shared budget and is proved to have sublinear regret. On RouterBench and SWE-Bench the offline router beats prior selectors at tight budgets and stays closer to the cap, while the online router matches them with far less training data.

What carries the argument

Adaptive Linear Programming (ALP): at each step it solves a small LP that turns remaining budget b over remaining rounds τ into the average constraint b/τ, then samples a model from the resulting probabilities; WR-Online wraps ALP in an ε-first exploration phase that estimates both rewards and costs under the same budget B.

What would settle it

On a held-out stream with the same models and a fixed workload budget B, measure whether WR-Offline’s total spend stays within B while average quality exceeds FrugalGPT, CascadeRouting, MetaLLM, and SingleBest at the tightest budgets; if it overspends or underperforms those baselines, or if raising intra-cluster reward variance collapses the gains, the central claim fails.

Watch

Extended reading notes

Core claim

Workload-constrained LLM routing is a constrained contextual bandit that Adaptive Linear Programming can solve: WR-Offline estimates reward and cost once from history and re-solves a lightweight LP at each step so any new global budget is met without retraining, and WR-Online jointly learns those statistics under one shared budget with end-to-end regret O(√T) (O(log T) off boundary points) when exploration is long enough to recover the true reward-cost ordering.

Load-bearing premise

After embedding and clustering, queries in the same cluster are treated as sharing the same expected quality and cost for each model, and those clusters arrive independently from a fixed distribution over a known finite horizon.

Editorial extensions

If this is right

  • Operators can change monthly API budgets at inference time without recollecting responses or retraining a router.
  • A single shared budget can cover both exploration and deployment, so online routers need far less than a full query–model response matrix.
  • At tight budgets, allocating expensive models only to hard clusters raises average quality more than any fixed per-query cap.
  • End-to-end regret of order √T gives a concrete sample-complexity target for how long exploration must run before exploitation.
  • The same ALP template applies to non-monetary costs such as latency or energy once they are normalized per query.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If query traffic drifts by time of day or product surface, periodically re-estimating only the cluster frequencies (not full model statistics) may restore budget adherence without a full re-exploration.
  • The same discretization-plus-ALP pattern could route among speculative-decoding, quantized, or cascade configurations rather than whole models.
  • When several models sit on nearly identical cost tiers, the ratio-ordering test becomes the practical bottleneck; cheaper pairwise A/B probes on the hardest clusters may shorten exploration more than uniform least-played sampling.
  • Workloads with strong query dependence (multi-turn chats) would need a contextual or restless extension before the stated regret bound can be claimed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The paper formulates workload-budget-constrained LLM routing as a constrained contextual bandit (CCB): query embeddings are clustered into a finite context set, and Adaptive Linear Programming (ALP, Wu et al. 2015) converts the remaining workload budget into a per-round constraint at each step. Two variants are proposed: WR-Offline, which estimates cluster-level reward/cost statistics from a dense historical dataset and adapts to new budgets without retraining; and WR-Online, an ε-first scheme that jointly estimates rewards and costs under the same shared budget, then exploits via ALP. The main theoretical claim is Theorem 5.3: end-to-end regret of O(log T) (non-boundary) and O(√T) (boundary) under a sufficient exploration condition (Lemma 5.2), whose proof uses a cross-product sign test to avoid random denominators. Experiments on RouterBench and SWE-Bench show WR-Offline competitive or better than FrugalGPT, CascadeRouting, MetaLLM, and SingleBest at tight budgets, with near-exact budget utilization and ~5,900× lower training latency; WR-Online matches offline methods with ~90% less supervision data.

Significance. If the theory is repaired, the contribution is meaningful: the workload-level (rather than per-query) budget formulation is well-motivated for API-quota deployments, the ALP reduction gives a practical, retraining-free mechanism for adapting to new budgets, and the extension of ε-first ALP to jointly unknown rewards and costs under a single shared budget — including the cross-product concentration argument in Lemma 5.2 and explicit failure-probability bookkeeping — is a genuine technical step beyond Wu et al. [40], which assumes known costs. The empirical package is solid for a systems-oriented claim: two public benchmarks, fixed 50:50 splits, five seeds with means±std, budget-utilization analysis, latency comparisons, a static-LP ablation isolating the adaptive constraint, robustness ablations over J, embeddings, and clustering algorithms, and a controlled simulation. The 90% supervision reduction for WR-Online and the 5,900× training-latency improvement are practically significant. However, the headline theoretical claim (Theorem 5.3) currently has a load-bearing gap, detailed in Major Comment 1, which tempers the overall assessment until addressed.

major comments (1)
  1. §5, proof sketch of Theorem 5.3 (and its reliance on Theorem 5.1/D.1): the inference 'correct ordering of ξ ⇒ exploitation incurs the regret of Theorem 5.1' does not hold as stated, and the missing term appears to be O(T), not negligible. Theorem 5.1 is explicitly for ALP with KNOWN statistics: its regret is E[v(ρ)] − E[v(b/τ)] where v is evaluated at the true (u,c). Under Lemma 5.2's good event, estimates concentrate only to the constant radius t = L/(8√2) (exploration is O(log T), so per-pair error does not vanish with T). With correct ordering, the greedy threshold structure is preserved, but the fractional boundary allocation p̃ = (b/τ − Q̃_ĩ)/(π_j c̃) and the cumulative masses Q̃_i depend on the VALUES of c̄, not only their order, and Algorithm 2 freezes (ū,c̄) after exploration. A within-radius perturbation therefore shifts the per-round realized expected reward by O(t·Δη/(π_min·Δc

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: regret and empirical claims are not forced by definition or self-citation chains.

full rationale

WISERouter applies external constrained-contextual-bandit machinery (ALP / ε-first ALP from Wu et al. [40], non-overlapping authors) to LLM routing. The oracle upper bound bU(T,B)=Tv(ρ) is the standard LP relaxation of any feasible policy’s expected reward under E[∑St]≤B (Appendix C), not a quantity defined from the algorithm’s own outputs. Regret is the gap to an oracle with true (uj,k,cj,k); WR-Online’s Lemma 5.2 only certifies recovery of the ξ-ordering from finite samples, and Theorem 5.3 claims inheritance of ALP’s known-statistics bound—whether that inheritance is fully rigorous is a correctness/gap question (estimation error in values, not only order), not circularity. Offline/online statistics are estimated from historical or exploration feedback and evaluated on held-out RouterBench/SWE-Bench scores and fixed API prices; nothing is fitted to equal the claimed optimum by construction. Cluster means as context-action statistics and i.i.d. πj are modeling assumptions, not self-definitional loops. No load-bearing uniqueness theorem or ansatz is imported from the present authors. Empty steps; score 0.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

The load-bearing content is a constrained-bandit reduction plus clustering approximation, not new physical entities. Theory inherits standard concentration and ALP structure; practice adds embedding/clustering choices and a hand-set exploration schedule because the paper’s own confidence test is too conservative for LLM cost gaps.

free parameters (4)
  • Number of clusters J = 16 (main experiments)
    Discretization granularity for ALP’s finite context space; chosen on validation (J=16). Ablations show similar reward for J in {8,12,16,20} but the finite-J model is still a fitted structural choice.
  • Exploration length ε(T) = 1×–4× |train| (experiments); condition (5) in theory
    Theory gives a sufficient lower bound in terms of unknown Δc_min, Δξ_min, π_min; experiments instead use fixed 1×–4× training-set size because the confidence-level test is impractical.
  • MetaLLM Lagrangian penalty p (baseline) = grid-searched per budget level
    Tuned per budget on held-out validation so comparisons depend on this grid search; not part of WR itself but affects claimed margins.
  • Embedding model and dimension = OpenAI text-embedding-3-small, d=1536
    text-embedding-3-small (d=1536) defines the geometry that clustering and contexts inherit; ablated vs Qwen3-Embedding-0.6B with optional PCA.
assumptions (6)
  • domain assumption Contexts (queries) are i.i.d. from a stationary discrete distribution {π_j} over a known finite horizon T.
    Stated in Sections 3–4 and Appendix A; required for Hoeffding bounds on N_j and for regret vs LPT,B.
  • domain assumption For each LLM k, all queries in cluster j share the same expected reward u_{j,k} and expected cost c_{j,k}.
    Section 4.1 homogeneity assumption that makes finite-context ALP well-defined after embedding clustering.
  • domain assumption Optimize expected cumulative reward subject to expected cumulative cost ≤ B (not pathwise almost-sure budget).
    Section 3.1; follows prior LLM selection work but allows realized spend to fluctuate around B.
  • ad hoc to paper c_{j,k1} ≠ c_{j,k2} for k1 ≠ k2 so reward-cost ratios ξ are well-defined; costs/rewards in [0,1] after normalization.
    Used in Lemma 5.2 and ratio ordering; realistic for distinct price tiers but fails if two models tie on expected cost in a cluster.
  • standard math Standard Hoeffding/Azuma concentration and the ALP regret structure of Wu et al. (2015) for known statistics.
    Appendices D–F build the unknown-cost extension on these classical inequalities and the known-cost ALP theorem.
  • domain assumption Bandit feedback only: (Y_t, S_t) observed solely for the chosen model; quality scores available offline or during exploration.
    Section 3.1 and Appendix A; quality may be human preference, unit tests, or proxies—assumed usable without blocking.
invented entities (2)
  • WISERouter (WR-Offline / WR-Online) framework
    purpose: Name the full pipeline: embed→cluster→estimate (u,c)→solve adaptive LP under remaining b/τ, with optional ε-first exploration under shared B.
    Engineering composition of known ALP and bandit ideas specialized to LLM routing; not a new physical entity. independent_evidence is empirical benchmark performance, not an external discovery handle.
  • Confidence Level Test (CLT) for stopping exploration
    purpose: Certify that empirical ξ orderings match true orderings w.h.p. before exploitation (Algorithm 3, Appendix G).
    Derived sufficient test; authors report it is too conservative for near-tied LLM prices and do not use it in main experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WISERouter: LLM Routing with Workload Budget Constraint." pith.science (2026). https://pith.science/paper/TYHCSVL6

@misc{pith2026260723765,
  author       = {Pith},
  title        = {Pith review of: WISERouter: LLM Routing with Workload Budget Constraint},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYHCSVL6}},
  note         = {Machine review of arXiv:2607.23765}
}
abstract

Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale. LLM routing exploits diversity in model capability and cost by assigning each query to a suitable model to balance utility and budget. Current methods have two limitations: (i) they either use heuristics that do not always enforce the budget constraint or impose a fixed per-query budget that cannot adapt across the workload and leads to suboptimal performance; (ii) they require supervised learning on a dense dataset with statistics for every query-model pair, which is expensive to collect. To address these challenges, we formulate LLM routing as a constrained contextual multi-armed bandit problem and introduce WISERouter (WR for short), a framework that supports offline learning from historical interactions as well as online learning with exploration. We further prove that WR-Online achieves a sublinear regret bound of $O(\sqrt{T})$ over a time horizon $T$. Empirical results on RouterBench and SWE-Bench demonstrate that (i) WR-Offline surpasses existing baselines in performance under a fixed budget and adheres more closely to budget constraints, and (ii) WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.

Figures

Figures reproduced from arXiv: 2607.23765 by the authors.

Figure 1
Figure 1. Comparison of WISERouter with previous LLM selection methods. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of WISERouter workflow. 4.1 Context Discretization For the CCB formulation to be meaningful, the context must encode query-level features that explain the variation in LLM performance across queries. Since different LLMs can perform unevenly across query semantics [38], we use a pretrained encoder g(·) to map each query q ∈ Q to an embedding vector x = g(q) ∈ R d . However, ALP assumes a finite context spac… view at source ↗
Figure 3
Figure 3. Remaining budget ratios at each budget level. Error bars show [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Budget consumption vs. reward gap for the static variant on SWE-Bench (5 runs); left￾most point is under the largest budget constraint. 200 400 600 Total Cost ($) 0.0 0.2 0.4 0.6 0.8 1.0 Average Reward per Query 200 300 400 Total Cost ($) SingleBest FrugalGPT CascadeRo…
Figure 5
Figure 5. Figure 5: (a) WR-Online vs. offline baselines on RouterBench; vertical dashed lines mark offline [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison under end-to-end budget constraints on SWE-Bench. Comparison [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 3 linked inside Pith

  1. [40]

    Srikant, Xin Liu, and Chong Jiang

    Huasen Wu, R. Srikant, Xin Liu, and Chong Jiang. Algorithms with logarithmic or sublinear regret for constrained contextual bandits. InProceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 433–441, Cambridge, MA, USA, 2015. MIT Press

    2015

  2. [1]

    Claude Team

    Anthropic. Claude Team. https://claude.com/pricing/team, 2026. Accessed: 2026-04- 28

    2026

  3. [2]

    Finite-time analysis of the multiarmed bandit problem.Mach

    Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem.Mach. Learn., 47(2–3):235–256, May 2002. ISSN 0885-6125. doi: 10.1023/A: 1013689704352. URLhttps://doi.org/10.1023/A:1013689704352

    doi:10.1023/a: 2002

  4. [3]

    Resourceful contextual bandits

    Ashwinkumar Badanidiyuru, John Langford, and Aleksandrs Slivkins. Resourceful contextual bandits. In Maria Florina Balcan, Vitaly Feldman, and Csaba Szepesvári, editors,Proceedings of The 27th Conference on Learning Theory, volume 35 ofProceedings of Machine Learning Research, pages 1109–1134, Barcelona, Spain, 13–15 Jun 2014. PMLR

    2014

  5. [4]

    PyramidKV: Dynamic KV cache compression based on pyramidal information funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. InSecond Conference on Language Modeling, 2025. URLhttps://openreview.net/forum?id=ayi7qezU87

    2025

  6. [5]

    Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    arXiv 2023

  7. [6]

    FrugalGPT: How to use large language models while reducing cost and improving performance.Transactions on Machine Learning Research,

    Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance.Transactions on Machine Learning Research,

  8. [7]

    Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: an open platform for evaluating llms by human preference. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

Show all 46 references
  1. [8]

    A unified approach to routing and cascading for LLMs

    Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach to routing and cascading for LLMs. InForty-second International Conference on Machine Learning, 2025. URLhttps://openreview.net/forum?id=AAl89VNNy1

    2025

  2. [9]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V . S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality- aware query routing. InThe Twelfth International Conference on Learning Representations,

  3. [10]

    Dujian Ding, Ankur Mallick, Shaokun Zhang, Chi Wang, Daniel Madrigal, Mirian Del Car- men Hipolito Garcia, Menglin Xia, Laks V . S. Lakshmanan, Qingyun Wu, and Victor Rühle. BEST-route: Adaptive LLM routing with test-time optimal compute. InForty-second Inter- national Confere...

    2025

  4. [11]

    Multi-armed bandit with budget constraint and variable costs

    Wenkui Ding, Tao Qiny, Xu-Dong Zhang, and Tie-Yan Liu. Multi-armed bandit with budget constraint and variable costs. InProceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, AAAI’13, page 232–238. AAAI Press, 2013

    2013

  5. [12]

    Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J. Mach. Learn. Res., 23(1), January 2022. ISSN 1532-4435

    2022

  6. [13]

    Language model cascades: Token-level uncertainty and beyond

    Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.ne...

    2024

  7. [14]

    Hassibi, D.G

    B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain surgeon and general network pruning. InIEEE International Conference on Neural Networks, pages 293–299 vol.1, 1993. doi: 10.1109/ICNN.1993.298572. 10

    1993

  8. [15]

    Routerbench: A benchmark for multi-llm routing system.CoRR, abs/2403.12031, 2024

    Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system.CoRR, abs/2403.12031, 2024. doi: 10.48550/ARXIV .2403.12031. URL https://doi.org/10.48550...

    2024 doi arXiv

  9. [16]

    Thriftllm: On cost-effective selection of large language models for classification queries.Proc

    Keke Huang, Yimin Shi, Dujian Ding, Yifei Li, Yang Fei, Laks Lakshmanan, and Xiaokui Xiao. Thriftllm: On cost-effective selection of large language models for classification queries.Proc. VLDB Endow., 18(11):4410–4423, September 2025. ISSN 2150-8097. doi: 10.14778/3749646. 374...

    2025

  10. [17]

    Ensemble learning for heterogeneous large language models with deep parallel collaboration

    Yi-Chong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Bing Qin, and Ting Liu. Ensemble learning for heterogeneous large language models with deep parallel collaboration. In Neural Information Processing Systems, 2024. URL https://api.semanticscholar.org/ CorpusID:269282634

    2024

  11. [18]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In2018 IEEE/CVF Conference on Computer Vision and Pattern R...

    2018

  12. [19]

    LLM-blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational ...

    2023 doi

  13. [20]

    SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/fo...

    2024

  14. [21]

    S3: increasing gpu utilization during generative inference for higher throughput

    Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. S3: increasing gpu utilization during generative inference for higher throughput. InProceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY , USA, 2023. Curran Ass...

    2023

  15. [22]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989

    1989

  16. [23]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  17. [24]

    Schapire

    Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. InProceedings of the 19th International Conference on World Wide Web, WWW ’10, page 661–670, New York, NY , USA, 2010. Association for Computing...

    2010

  18. [25]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas,...

    2023

  19. [26]

    Routing to the expert: Efficient reward-guided ensemble of large language models

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the 11 Nort...

    2024

  20. [27]

    Some methods of classification and analysis of multivariate observations

    James B McQueen. Some methods of classification and analysis of multivariate observations. InProc. of 5th Berkeley Symposium on Math. Stat. and Prob., pages 281–297, 1967

    1967

  21. [28]

    Omnirouter: Budget and performance controllable multi-llm routing.SIGKDD Explor

    Kai Mei, Wujiang Xu, Minghao Guo, Shuhang Lin, and Yongfeng Zhang. Omnirouter: Budget and performance controllable multi-llm routing.SIGKDD Explor. Newsl., 27(2):107–116, December 2026. ISSN 1931-0145. doi: 10.1145/3787470.3787480. URL https://doi.org/ 10.1145/3787470.3787480

    2026

  22. [29]

    Nguyen, Thinh Dao, Duy C

    Quang H. Nguyen, Thinh Dao, Duy C. Hoang, Juliette Decugis, Saurav Manchanda, Nitesh V . Chawla, and Khoa D. Doan. Metallm: A high-performant and cost-efficient dynamic framework for wrapping llms, 2025. URLhttps://arxiv.org/abs/2407.10834

    2025 arXiv

  23. [30]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.n...

    2025

  24. [31]

    API Pricing

    OpenAI. API Pricing. https://openai.com/api/pricing/, 2026. Accessed: 2026-04-28

    2026

  25. [32]

    Adaptive LLM routing under budget constraints

    Pranoy Panda, Raghav Magazine, Chaitanya Devaguptapu, Sho Takemori, and Vishal Sharma. Adaptive LLM routing under budget constraints. In Christos Christodoulopoulos, Tan- moy Chakraborty, Carolyn Rose, and Violet Peng, editors,Findings of the Association for Computational Ling...

    2025 doi

  26. [33]

    Lui, and Jinhang Zuo

    Manhin Poon, XiangXiang Dai, Xutong Liu, Fang Kong, John C.S. Lui, and Jinhang Zuo. Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 2026

    2026

  27. [34]

    Le, Geoffrey E

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V . Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In5th International Conference on Learning Representations, ICLR 2017, Toulon, Fran...

    2017

  28. [35]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249

    2018

  29. [36]

    Jennings

    Long Tran-Thanh, Archie Chapman, Enrique Munoz de Cote, Alex Rogers, and Nicholas R. Jennings. Epsilon–first policies for budget–limited multi-armed bandits.Proceedings of the AAAI Conference on Artificial Intelligence, 24(1):1211–1216, Jul. 2010. doi: 10.1609/aaai. v24i1.7758

    2010 doi

  30. [37]

    Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao. Improving the speed of neural networks on cpus. InDeep Learning and Unsupervised Feature Learning Workshop, NIPS 2011, 2011

    2011

  31. [38]

    MixLLM: Dynamic Routing in Mixed Large Language Models

    Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. MixLLM: Dynamic Routing in Mixed Large Language Models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Proceedings of the 2025 Conference of the Nations of the Ame...

    2025

  32. [39]

    Livebench: A challenging, contamination-free LLM benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and M...

    2025

  33. [41]

    Large language model cascades with mixture of thought representations for cost-efficient reasoning

    Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=6okaSfANz...

    2024

  34. [44]

    TX t=1 St # = JX j=1 KX k=1 E

    proposes UCB-ALP for unit-cost systems where all arms have a uniform cost and ϵ-first ALP for heterogeneous-cost systems, though rely on cost statistics being known a priori. Another line of work studies stochastic costs under budget constrained MAB using UCB-style frameworks ...

  35. [45]

    For k1, if there exists another action k2 such that ηj,k1 ≤η j,k2 and uj,k1 ≤u j,k2, then p∗ j,k1 = 0

  36. [46]

    Intuitively, the first part of Lemma E.1 shows that if an action has small normalized and original expected reward, then it can be removed

    For k1, if there exist two actions k2 and k3 such that ηj,k2 ≤η j,k1 ≤η j,k3, uj,k2 ≥u j,k1 ≥ uj,k3, and uj,k1 −uj,k3 cj,k1 −cj,k3 ≤ uj,k2 −uj,k3 cj,k2 −cj,k3 ,thenp ∗ j,k1 = 0. Intuitively, the first part of Lemma E.1 shows that if an action has small normalized and original ...

    2000

  37. [2024]

    URLhttps://openreview.net/forum?id=02f3mUtqnM

  38. [2025]

    ISBN 979-8-89176-189-6

    Association for Computational Linguistics. ISBN 979-8-89176-189-6

Pith tools

Reviewed July 30, 2026 · model on record in the stance chip above.