Pith. sign in

REVIEW 1 minor 1 cited by

From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing

T0 review · 0 major / 1 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Single LLM responses give noisy supervision for router training, while distribution-aware labels from multiple samples and query variants produce more reliable routing policies.

desk verdict The paper's main point is that single-shot labels add real noise to LLM router training because of generation stochasticity, and DARS tries to replace them with labels built from distributions over query variants and outputs. read the letter →

arxiv 2606.06924 v1 pith:HIDA7ZRP submitted 2026-06-05 cs.LG

Guannan Lai , Haoran Hu , Long Chen , Zhenguo Li , Han-Jia Ye This is my paper
classification cs.LG
keywords LLMroutingdistribution-awaresupervisioncapabilitydistributionsstochasticgenerationmodelselectionqueryreformulationDARS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that using one generated response as a capability label for a query-model pair only captures a noisy observation because LLM generation is stochastic. This noise systematically degrades the quality of learned routing policies. DARS instead builds supervision by sampling multiple semantically equivalent query formulations and multiple generations per formulation to estimate capability distributions. Experiments across tasks show that single-shot labels mislead model selection while the distributional approach yields stabler labels and better routing performance. The work argues that routing supervision should be grounded in query-level capability distributions rather than point observations.

What carries the argument

DARS (Distribution-Aware Routing Supervision), the framework that replaces single-response labels with aggregated performance across semantically equivalent queries and stochastic generations to estimate model capability distributions.

What would settle it

An experiment in which routers trained on single-shot labels achieve equal or higher accuracy than DARS-trained routers when evaluated on held-out queries and models.

Watch

Extended reading notes

Core claim

Existing LLM routing methods treat a model's single response to a query as its capability label, but stochastic generation makes this only a noisy observation rather than a reliable estimate. DARS constructs supervision from a distributional view by considering uncertainty from both input reformulations and output generations, producing more stable signals that improve learned routing behavior over single-response baselines.

Load-bearing premise

Aggregating performance across multiple semantically equivalent query formulations and stochastic generations produces a more accurate estimate of underlying model capability than any single observation.

Editorial extensions

If this is right

  • Routers trained on distribution-aware labels select models more accurately than those trained on single responses.
  • Single-shot supervision introduces systematic noise that can be reduced by distributional aggregation.
  • Capability estimates become more stable when both query formulation uncertainty and generation stochasticity are modeled.
  • Routing performance improves across diverse tasks when supervision reflects query-level capability distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same distributional approach to supervision could apply to other LLM training settings that currently rely on single sampled outputs for labels.
  • Routers might eventually output full capability distributions rather than point predictions for downstream decisions.
  • This framing suggests re-examining single-sample evaluation practices in model capability assessment more broadly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 1 minor

Summary. The paper claims that single-shot supervision for LLM routing is noisy and unreliable due to the inherent stochasticity of LLM generation, providing only a point observation rather than a true capability estimate. It introduces DARS (Distribution-Aware Routing Supervision), a framework that constructs labels from distributions over semantically equivalent query reformulations (input-side uncertainty) and multiple stochastic outputs (output-side uncertainty). The abstract asserts that experiments across diverse tasks demonstrate single-shot labels can be misleading while distribution-aware supervision yields more stable labels and better routing performance, advocating a shift to query-level capability distributions.

Significance. If the empirical claims hold, the work could meaningfully influence LLM routing research and practice by identifying a systematic source of supervision noise and offering a practical alternative grounded in distributional observations. This addresses a real deployment issue in cost-aware model selection and could lead to more robust routers. The conceptual framing is coherent and the distinction between point and distributional supervision is clearly articulated without internal contradictions.

minor comments (1)
  1. Abstract: the claim that 'experiments across diverse tasks show' the superiority of distribution-aware supervision is stated without any reference to specific tasks, datasets, metrics, baselines, or quantitative results, which limits the ability to assess the strength of the central empirical claim from the provided material.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their summary and for recognizing the conceptual coherence of our framing around point versus distributional supervision in LLM routing. The report does not list any specific major comments, so we have no point-by-point responses or revisions to propose at this stage. We remain available to address any additional questions the referee may have.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; derivation is self-contained

full rationale

The paper advances a conceptual proposal (DARS) that single-shot labels are noisy due to stochastic generation while distributional supervision over query variants and outputs yields stabler signals. No equations, fitted parameters, or derivations appear in the abstract or described structure that reduce by construction to inputs, self-citations, or renamed empirical patterns. The central distinction between point observations and capability distributions is presented as an empirical modeling choice rather than a mathematical identity or load-bearing self-reference. The argument remains independent of any uniqueness theorem or ansatz imported from prior author work.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review; no free parameters, axioms, or invented entities are specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing." pith.science (2026). https://pith.science/paper/HIDA7ZRP

@misc{pith2026260606924,
  author       = {Pith},
  title        = {Pith review of: From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HIDA7ZRP}},
  note         = {Machine review of arXiv:2606.06924}
}
read the original abstract

Existing LLM routing methods typically treat a model's single response to a query as its capability label for training routers. However, because LLM generation is inherently stochastic, such single-shot supervision provides only a noisy observation of a query-model pair's behavior rather than a reliable capability estimate. We show that this assumption introduces systematic noise into routing supervision, making learned routing policies less reliable. To address this issue, we propose DARS (Distribution-Aware Routing Supervision), a framework that constructs routing supervision from a distributional view of model behavior. Instead of relying on a single generated response, DARS considers uncertainty from both the input side and the output side, capturing how semantically equivalent query formulations and stochastic generations affect model performance. Based on these distribution-aware observations, DARS builds more reliable supervision signals for routing. Experiments across diverse tasks show that single-shot labels can be misleading for model selection, while distribution-aware supervision provides more stable labels and improves learned routing behavior. Our results suggest that reliable LLM routing should move beyond single-response observations and be grounded in query-level model capability distributions.

Figures

Figures reproduced from arXiv: 2606.06924 by the authors.

Figure 1
Figure 1. Illustration of the single-shot label issue in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Diagnostic analysis of single-shot routing supervision. (a) Single-shot labels are unstable at the outcome, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed DARS framework. Conventional single-shot supervision obtains only one [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Further analysis of DARS. (a) Sample efficiency analysis evaluates how the number of repeated observa [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    RouteJudge introduces an open platform for preference-based evaluation of LLM routers via pairwise user comparisons, along with the ORBIT toolbox for standardized routing workflows.

Reference graph

Works this paper leans on

168 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abril and Robert Plant

    Patricia S. Abril and Robert Plant. The patent holder's dilemma: Buy, sell, or troll?. Communications of the ACM. doi:10.1145/1188913.1188915

    work page doi:10.1145/1188913.1188915

  2. [2]

    Deciding equivalances among conjunctive aggregate queries

    Sarah Cohen and Werner Nutt and Yehoshua Sagic. Deciding equivalances among conjunctive aggregate queries. doi:10.1145/1219092.1219093

    work page doi:10.1145/1219092.1219093

  3. [3]

    When routing collapses: On the degenerate convergence of llm routers.arXiv preprint arXiv:2602.03478,

    When Routing Collapses: On the Degenerate Convergence of LLM Routers , author=. arXiv preprint arXiv:2602.03478 , year=

    work page arXiv

  4. [4]

    Special issue: Digital Libraries. 1996

    1996

  5. [5]

    Understanding Policy-Based Networking

    David Kosiur. Understanding Policy-Based Networking

  6. [8]

    doi:10.1007/3-540-09237-4

    The title of book two. doi:10.1007/3-540-09237-4

    work page doi:10.1007/3-540-09237-4

  7. [9]

    Asad Z. Spector. Achieving application requirements. Distributed Systems. doi:10.1145/90417.90738

    work page doi:10.1145/90417.90738

  8. [10]

    Douglass and David Harel and Mark B

    Bruce P. Douglass and David Harel and Mark B. Trakhtenbrot. Statecarts in use: structured analysis and object-orientation. Lectures on Embedded Systems. doi:10.1007/3-540-65193-4_29

    work page doi:10.1007/3-540-65193-4_29

Show all 168 references
  1. [11]

    Donald E. Knuth. The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd. ed.)

  2. [12]

    Donald E. Knuth. The Art of Computer Programming

  3. [13]

    Structured Variational Inference Procedures and their Realizations (as incol)

    Dan Geiger and Christopher Meek. Structured Variational Inference Procedures and their Realizations (as incol). Proceedings of Tenth International Workshop on Artificial Intelligence and Statistics, The Barbados

  4. [14]

    Stan W. Smith. An experiment in bibliographic mark-up: Parsing metadata for XML export. Proceedings of the 3rd. annual workshop on Librarians and Computers

  5. [15]

    Catch me, if you can: Evading network signatures with web-based polymorphic worms

    Matthew Van Gundy and Davide Balzarotti and Giovanni Vigna. Catch me, if you can: Evading network signatures with web-based polymorphic worms. Proceedings of the first USENIX workshop on Offensive Technologies

  6. [16]

    Predicate Path expressions

    Sten Andler. Predicate Path expressions. Proceedings of the 6th. ACM SIGACT-SIGPLAN symposium on Principles of Programming Languages. doi:10.1145/567752.567774

    doi

  7. [17]

    LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER

    David Harel. LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER

  8. [18]

    Anisi , title =

    David A. Anisi , title =

  9. [19]

    Clarkson

    Kenneth L. Clarkson. Algorithms for Closest-Point Problems (Computational Geometry)

  10. [20]

    Introduction to Bayesian Statistics

    Harry Thornburg. Introduction to Bayesian Statistics. 2001

    2001

  11. [21]

    CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11

    Rafal Ablamowicz and Bertfried Fauser. CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11. 2007

    2007

  12. [22]

    Stats and Analysis

    Poker-Edge.Com. Stats and Analysis. 2006

    2006

  13. [23]

    A more perfect union

    Barack Obama. A more perfect union

  14. [24]

    The fountain of youth

    Joseph Scientist. The fountain of youth

  15. [25]

    Solder man

    Dave Novak. Solder man. ACM SIGGRAPH 2003 Video Review on Animation theater Program: Part I - Vol. 145 (July 27--27, 2003). doi:10.945/woot07-S422

    2003

  16. [26]

    Interview with Bill Kinder: January 13, 2005

    Newton Lee. Interview with Bill Kinder: January 13, 2005. Comput. Entertain. doi:10.1145/1057270.1057278

    2005 doi

  17. [27]

    The Enabling of Digital Libraries

    Bernard Rous. The Enabling of Digital Libraries. Digital Libraries

  18. [29]

    (new) Finding minimum congestion spanning trees , journal =

    Werneck, Renato and Setubal, Jo\. (new) Finding minimum congestion spanning trees , journal =. doi:10.1145/351827.384253 , acmid = 384253, publisher =

    doi

  19. [31]

    and Mei, Alessandro , title =

    Conti, Mauro and Di Pietro, Roberto and Mancini, Luigi V. and Mei, Alessandro , title =. Inf. Fusion , volume =. 2009 , issn =. doi:10.1016/j.inffus.2009.01.002 , acmid =

    2009 doi

  20. [32]

    and Hutchful, David K

    Li, Cheng-Lun and Buyuktur, Ayse G. and Hutchful, David K. and Sant, Natasha B. and Nainwal, Satyendra K. , title =. CHI '08 extended abstracts on Human factors in computing systems , year =. doi:10.1145/1358628.1358946 , acmid =

    doi

  21. [33]

    , title =

    Hollis, Billy S. , title =. 1999 , isbn =

    1999

  22. [34]

    Goossens, Michel and Rahtz, S. P. and Moore, Ross and Sutor, Robert S. , title =. 1999 , isbn =

    1999

  23. [35]

    and Rosenberg, Arnold L

    Buss, Jonathan F. and Rosenberg, Arnold L. and Knott, Judson D. , title =. 1987 , source =

    1987

  24. [36]

    CHI '08: CHI '08 extended abstracts on Human factors in computing systems , year =

    , note =. CHI '08: CHI '08 extended abstracts on Human factors in computing systems , year =

  25. [37]

    Algorithms for Closest-Point Problems (Computational Geometry) , year =

    Clarkson, Kenneth Lee , advisor =. Algorithms for Closest-Point Problems (Computational Geometry) , year =

  26. [38]

    SIGCOMM Comput. Commun. Rev. , year =

  27. [39]

    2004 , isbn =

    IEEE TCSC Executive Committee , booktitle =. 2004 , isbn =. doi:http://dx.doi.org/10.1109/ICWS.2004.64 , acmid =

    2004 doi

  28. [40]

    Distributed systems (2nd Ed.) , year =

  29. [41]

    , title =

    Petrie, Charles J. , title =. 1986 , source =

    1986

  30. [42]

    Donald E. Knuth. Seminumerical Algorithms. 1981

    1981

  31. [43]

    E-commerce and cultural values , year =

    Kong, Wei-Chang , Title =. E-commerce and cultural values , year =

  32. [44]

    E-commerce and cultural values , year =

    Kong, Wei-Chang , type =. E-commerce and cultural values , year =

  33. [45]

    Chapter 9 , booktitle =

    Kong, Wei-Chang , editor =. Chapter 9 , booktitle =

  34. [46]

    E-commerce and cultural values , editor =

    Kong, Wei-Chang , title =. E-commerce and cultural values , editor =. 2003 , isbn =

    2003

  35. [47]

    E-commerce and cultural values - (InBook-num-in-chap) , chapter =

    Kong, Wei-Chang , editor =. E-commerce and cultural values - (InBook-num-in-chap) , chapter =. 2004 , address =

    2004

  36. [48]

    E-commerce and cultural values (Inbook-text-in-chap) , chapter =

    Kong, Wei-Chang , editor =. E-commerce and cultural values (Inbook-text-in-chap) , chapter =. 2005 , address =

    2005

  37. [49]

    E-commerce and cultural values (Inbook-num chap) , chapter =

    Kong, Wei-Chang , editor =. E-commerce and cultural values (Inbook-num chap) , chapter =. 2006 , address =

    2006

  38. [50]

    Microelectron

    Mehdi Saeedi and Morteza Saheb Zamani and Mehdi Sedighi , title =. Microelectron. J. , volume =. 2010 , pages =

    2010

  39. [51]

    Mehdi Saeedi and Morteza Saheb Zamani and Mehdi Sedighi and Zahra Sasanian , title =. J. Emerg. Technol. Comput. Syst. , volume =

  40. [52]

    Kirschmer, Markus and Voight, John , title =. SIAM J. Comput. , issue_date =. 2010 , issn =. doi:https://doi.org/10.1137/080734467 , acmid =

    2010 doi

  41. [53]

    Hoare, C. A. R. , title =. Structured programming (incoll) , editor =. 1972 , isbn =

    1972

  42. [54]

    History of programming languages I (incoll) , editor =

    Lee, Jan , title =. History of programming languages I (incoll) , editor =. 1981 , isbn =. doi:http://doi.acm.org/10.1145/800025.1198348 , acmid =

    1981 doi

  43. [55]

    , title =

    Dijkstra, E. , title =. Classics in software engineering (incoll) , year =

  44. [56]

    , title =

    Wenzel, Elizabeth M. , title =. Multimedia interface design (incoll) , year =. doi:10.1145/146022.146089 , acmid =

    doi

  45. [57]

    , title =

    Mumford, E. , title =. Critical issues in information systems research (incoll) , year =

  46. [58]

    and Golden, Donald G

    McCracken, Daniel D. and Golden, Donald G. , title =. 1990 , isbn =

    1990

  47. [59]

    The analysis of linear partial differential operators

    H. The analysis of linear partial differential operators. 1985 , PAGES =

    1985

  48. [60]

    IEEE", address =

    A. Adya and P. Bahl and J. Padhye and A.Wolman and L. Zhou , title =. Proceedings of the IEEE 1st International Conference on Broadnets Networks (BroadNets'04) , publisher = "IEEE", address = "Los Alamitos, CA", year =

  49. [61]

    I. F. Akyildiz and W. Su and Y. Sankarasubramaniam and E. Cayirci , title =. Comm. ACM , volume = 38, number = "4", year =

  50. [62]

    I. F. Akyildiz and T. Melodia and K. R. Chowdhury , title =. Computer Netw. , volume = 51, number = "4", year =

  51. [63]

    ACM", address =

    P. Bahl and R. Chancre and J. Dungeon , title =. Proceeding of the 10th International Conference on Mobile Computing and Networking (MobiCom'04) , publisher = "ACM", address = "New York, NY", year =

  52. [64]

    8 (Special Issue on Sensor Networks)

    D. Culler and D. Estrin and M. Srivastava , title =. IEEE Comput. , volume = 37, number = "8 (Special Issue on Sensor Networks)", publisher = "IEEE", address = "Los Alamitos, CA", year =

  53. [65]

    Natarajan and M

    A. Natarajan and M. Motani and B. de Silva and K. Yap and K. C. Chua , title =. Network Architectures , editor =. 960935712

  54. [66]

    Tzamaloukas and J

    A. Tzamaloukas and J. J. Garcia-Luna-Aceves , title =

  55. [67]

    Zhou and J

    G. Zhou and J. Lu and C.-Y. Wan and M. D. Yarvis and J. A. Stankovic , title =

  56. [68]

    Mapping Powerlists onto Hypercubes

    Jacob Kornerup. Mapping Powerlists onto Hypercubes. 1994

    1994

  57. [69]

    Automatic Parallelization for Distributed-Memory Multiprocessing Systems

    Michael Gerndt. Automatic Parallelization for Distributed-Memory Multiprocessing Systems

  58. [70]

    J. E. Archer, Jr. and R. Conway and F. B. Schneider. User recovery and reversal in interactive systems. ACM Trans. Program. Lang. Syst

  59. [71]

    D. D. Dunlop and V. R. Basili. Generalizing specifications for uniformly implemented loops. ACM Trans. Program. Lang. Syst

  60. [72]

    Heering and P

    J. Heering and P. Klint. Towards monolingual programming environments. ACM Trans. Program. Lang. Syst

  61. [73]

    Donald E. Knuth. The book

  62. [74]

    Korach and D

    E. Korach and D. Rotem and N. Santoro. Distributed algorithms for finding centers and medians in networks. ACM Trans. Program. Lang. Syst

  63. [75]

    : A Document Preparation System

    Leslie Lamport. : A Document Preparation System

  64. [76]

    F. Nielson. Program transformations in a denotational setting. ACM Trans. Program. Lang. Syst

  65. [77]

    AAAI , year=

    Capability instruction tuning: A new paradigm for dynamic llm routing , author=. AAAI , year=

  66. [78]

    arXiv preprint arXiv:2601.17814 , year=

    MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing , author=. arXiv preprint arXiv:2601.17814 , year=

  67. [79]

    LLM -Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

    Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen. LLM -Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. ACL. 2023

    2023

  68. [80]

    2025 , journal=

    LLMRank: Understanding LLM Strengths for Model Routing , author=. 2025 , journal=

    2025

  69. [81]

    arXiv preprint arXiv:2502.20576 , year=

    OmniRouter: Budget and Performance Controllable Multi-LLM Routing , author=. arXiv preprint arXiv:2502.20576 , year=

  70. [82]

    IEEE Transactions on Mobile Computing , year=

    Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts , author=. IEEE Transactions on Mobile Computing , year=

  71. [83]

    arXiv preprint arXiv:2501.01818 , year=

    Rerouting llm routers , author=. arXiv preprint arXiv:2501.01818 , year=

  72. [84]

    arXiv preprint arXiv:2407.10834 , year=

    Metallm: A high-performant and cost-efficient dynamic framework for wrapping llms , author=. arXiv preprint arXiv:2407.10834 , year=

  73. [85]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    arXiv

  74. [86]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    arXiv

  75. [87]

    arXiv preprint arXiv:2506.05176 , year=

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models , author=. arXiv preprint arXiv:2506.05176 , year=

    arXiv

  76. [88]

    Sentence-

    Reimers, Nils and Gurevych, Iryna , booktitle =. Sentence-

  77. [89]

    AAAI , year=

    Film: Visual reasoning with a general conditioning layer , author=. AAAI , year=

  78. [90]

    NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models , year=

    LLM routing with benchmark datasets , author=. NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models , year=

    2023

  79. [91]

    NeurIPS , year=

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face , author=. NeurIPS , year=

  80. [92]

    arXiv preprint arXiv:2405.02134 , year=

    Optimising calls to large language models with uncertainty-based two-tier selection , author=. arXiv preprint arXiv:2405.02134 , year=

  81. [93]

    arXiv preprint arXiv:2310.13561 , year=

    Cache & distil: Optimising api calls to large language models , author=. arXiv preprint arXiv:2310.13561 , year=

  82. [94]

    NeurIPS , year=

    AutoMix: Automatically mixing language models , author =. NeurIPS , year=

  83. [95]

    arXiv preprint arXiv:2505.19435 , year=

    Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection , author=. arXiv preprint arXiv:2505.19435 , year=

  84. [96]

    arXiv preprint arXiv:2502.16696 , year=

    Dynamic llm routing and selection based on user preferences: Balancing performance, cost, and ethics , author=. arXiv preprint arXiv:2502.16696 , year=

  85. [97]

    Proceedings of the 17th ACM International Conference on Web Search and Data Mining , pages=

    Fly-swat or cannon? cost-effective language model choice via meta-modeling , author=. Proceedings of the 17th ACM International Conference on Web Search and Data Mining , pages=

  86. [98]

    ICML , year=

    BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute , author=. ICML , year=

  87. [99]

    Brian K. Reid. A high-level approach to computer document formatting. Proceedings of the 7th Annual Symposium on Principles of Programming Languages

  88. [100]

    and Abdelzaher, Tarek F

    Zhou, Gang and Wu, Yafeng and Yan, Ting and He, Tian and Huang, Chengdu and Stankovic, John A. and Abdelzaher, Tarek F. , title =. ACM Trans. Embed. Comput. Syst. , issue_date =. doi:10.1145/1721695.1721705 , acmid = 1721705, publisher =

    doi

  89. [101]

    Institutional members of the Users Group

  90. [102]

    Boris Veytsman , title =

  91. [103]

    Robin Schneider , title =

  92. [104]

    and Peterson, Larry L

    Bowman, Mic and Debray, Saumya K. and Peterson, Larry L. , title =. ACM Trans. Program. Lang. Syst. , volume =. 1993 , doi =

    1993

  93. [105]

    TUGboat , volume =

    Braams, Johannes , title =. TUGboat , volume =

  94. [106]

    Post Congress Tristesse

    Malcolm Clark. Post Congress Tristesse. TeX90 Conference Proceedings

  95. [107]

    ACM Trans

    Herlihy, Maurice , title =. ACM Trans. Program. Lang. Syst. , volume =. 1993 , doi =

    1993

  96. [108]

    Salas and Einar Hille

    S.L. Salas and Einar Hille. Calculus: One and Several Variable. 1978

    1978

  97. [109]

    Publication quality tables in

    Simon Fear , month =. Publication quality tables in

  98. [110]

    Using the amsthm Package , organization =

  99. [111]

    2019 , url =

    R: A Language and Environment for Statistical Computing , author =. 2019 , url =

    2019

  100. [112]

    Sam Anzaroot and Andrew McCallum , title =

  101. [113]

    Brad and Haunschild, Robin , title =

    Bornmann, Lutz and Wray, K. Brad and Haunschild, Robin , title =

  102. [114]

    2014 , archivePrefix =

    Sam Anzaroot and Alexandre Passos and David Belanger and Andrew McCallum , title =. 2014 , archivePrefix =. 1403.1349 , doi =

    2014 arXiv

  103. [115]

    Proceedings of the 20th International Colloquium on Automata, Languages and Programming , series =

    Maintaining Discrete Probability Distributions Optimally , author =. Proceedings of the 20th International Colloquium on Automata, Languages and Programming , series =

  104. [116]

    Reiser , year = 2014, title =

    Brian J. Reiser , year = 2014, title =

    2014

  105. [117]

    Baggett and M

    R. Baggett and M. Simecek and C. Chambellan and K. Tsui and M. Fraune , year = 2025, title =

    2025

  106. [118]

    NLP-OSS , year=

    Gptcache: An open-source semantic cache for llm applications enabling faster answers and cost savings , author=. NLP-OSS , year=

  107. [119]

    ICLR , year=

    Graphrouter: A Graph-Based Router for LLM Selections , author=. ICLR , year=

  108. [120]

    arXiv preprint arXiv:2409.06857 , year=

    What is the role of small models in the llm era: A survey , author=. arXiv preprint arXiv:2409.06857 , year=

  109. [121]

    arXiv preprint arXiv:2502.11021 , year=

    Leveraging uncertainty estimation for efficient llm routing , author=. arXiv preprint arXiv:2502.11021 , year=

  110. [122]

    Dujian Ding and Ankur Mallick and Chi Wang and Robert Sim and Subhabrata Mukherjee and Victor R. Hybrid. ICLR , year=

  111. [123]

    Gonzalez and M Waleed Kadous and Ion Stoica , booktitle =

    Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica , booktitle =. Route

  112. [124]

    EMNLP , year=

    Tensoropera router: A multi-model router for efficient llm inference , author=. EMNLP , year=

  113. [125]

    arXiv preprint arXiv:2308.11601 , year=

    Tryage: Real-time, intelligent routing of user prompts to large language models , author=. arXiv preprint arXiv:2308.11601 , year=

  114. [126]

    arXiv preprint arXiv:2502.18482 , year=

    Mixllm: Dynamic routing in mixed large language models , author=. arXiv preprint arXiv:2502.18482 , year=

  115. [127]

    First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models , year=

    Universal LLM Routing with Correctness-Based Representation , author=. First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models , year=

  116. [128]

    NeurIPS , year=

    Routerdc: Query-based router by dual contrastive learning for assembling large language models , author=. NeurIPS , year=

  117. [129]

    arXiv preprint arXiv:2401.13979 , year=

    Routoo: Learning to route to large language models effectively , author=. arXiv preprint arXiv:2401.13979 , year=

  118. [130]

    arXiv preprint arXiv:2409.15518 , year=

    Eagle: Efficient training-free router for multi-llm inference , author=. arXiv preprint arXiv:2409.15518 , year=

  119. [131]

    arXiv preprint arXiv:2310.03046 , year=

    Ecoassistant: Using llm assistant more affordably and accurately , author=. arXiv preprint arXiv:2310.03046 , year=

  120. [132]

    arXiv preprint arXiv:2305.05176 , year=

    Frugalgpt: How to use large language models while reducing cost and improving performance , author=. arXiv preprint arXiv:2305.05176 , year=

    arXiv

  121. [133]

    Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , year=

    Search Engines in the AI Era: A Qualitative Understanding to the False Promise of Factual and Verifiable Source-Cited Responses in LLM-based Search , author=. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , year=

    2025

  122. [134]

    Companion Proceedings of the ACM on Web Conference 2025 , year=

    CuSMer: Multimodal Intent Recognition in Customer Service via Data Augment and LLM Merge , author=. Companion Proceedings of the ACM on Web Conference 2025 , year=

    2025

  123. [135]

    ACM SIGKDD , year=

    Large language models meet collaborative filtering: An efficient all-round llm-based recommender system , author=. ACM SIGKDD , year=

  124. [136]

    arXiv preprint arXiv:2503.08704 , year=

    Life-Cycle Routing Vulnerabilities of LLM Router , author=. arXiv preprint arXiv:2503.08704 , year=

  125. [137]

    arXiv preprint arXiv:2504.07113 , year=

    How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities , author=. arXiv preprint arXiv:2504.07113 , year=

  126. [138]

    2012 , publisher=

    Bayes theory , author=. 2012 , publisher=

    2012

  127. [139]

    Introduction to transformers for NLP: With the hugging face library and models to solve problems , year=

    Hugging face , author=. Introduction to transformers for NLP: With the hugging face library and models to solve problems , year=

  128. [140]

    arXiv preprint arXiv:2403.12031 , year=

    Routerbench: A benchmark for multi-llm routing system , author=. arXiv preprint arXiv:2403.12031 , year=

    arXiv

  129. [141]

    Advances in Neural Information Processing Systems , year=

    Post-hoc estimators for learning to defer to an expert , author=. Advances in Neural Information Processing Systems , year=

  130. [142]

    arXiv preprint arXiv:2505.12601 , year=

    Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers , author=. arXiv preprint arXiv:2505.12601 , year=

    arXiv

  131. [143]

    arXiv preprint arXiv:1904.09675 , year=

    Bertscore: Evaluating text generation with bert , author=. arXiv preprint arXiv:1904.09675 , year=

    1904 arXiv

  132. [144]

    ICLR , year=

    EmbedLLM: Learning compact representations of large language models , author=. ICLR , year=

  133. [145]

    def , year=

    Conjugate Bayesian analysis of the Gaussian distribution , author=. def , year=

  134. [146]

    arXiv preprint arXiv:2502.00409 , year=

    Doing more with less--implementing routing strategies in large language model-based systems: An extended survey , author=. arXiv preprint arXiv:2502.00409 , year=

  135. [147]

    arXiv preprint arXiv:2506.06579 , year=

    Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques , author=. arXiv preprint arXiv:2506.06579 , year=

  136. [148]

    NAACLW , year=

    Harnessing the power of multiple minds: Lessons learned from LLM routing , author=. NAACLW , year=

  137. [149]

    ACM Transactions on Information Systems , year=

    When automated assessment meets automated content generation: Examining text quality in the era of gpts , author=. ACM Transactions on Information Systems , year=

  138. [150]

    arXiv preprint arXiv:2409.13757 , year=

    Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance , author=. arXiv preprint arXiv:2409.13757 , year=

  139. [151]

    arXiv preprint arXiv:2310.03094 , year=

    Large language model cascades with mixture of thoughts representations for cost-efficient reasoning , author=. arXiv preprint arXiv:2310.03094 , year=

  140. [152]

    DAI , year=

    Beyond gpt-5: Making llms cheaper and better via performance-efficiency optimized routing , author=. DAI , year=

  141. [153]

    NeurIPS , year=

    Causal LLM Routing: End-to-End Regret Minimization from Observational Data , author=. NeurIPS , year=

  142. [154]

    AAAI , year=

    Icl-router: In-context learned model representations for llm routing , author=. AAAI , year=

  143. [155]

    ICLR , year=

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , author=. ICLR , year=

  144. [156]

    S elf C heck GPT : Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

    Manakul, Potsawee and Liusie, Adian and Gales, Mark. S elf C heck GPT : Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. Proceedings of EMNLP , year=

  145. [157]

    Proceedings of PACLIC , year=

    Sensitivity and robustness of large language models to prompt template in Japanese text classification tasks , author=. Proceedings of PACLIC , year=

  146. [158]

    arXiv preprint arXiv:2306.04528 , year=

    PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts , author=. arXiv preprint arXiv:2306.04528 , year=

  147. [159]

    Nature , year=

    Detecting hallucinations in large language models using semantic entropy , author=. Nature , year=

  148. [160]

    EMNLP , year=

    Routereval: A comprehensive benchmark for routing llms to explore model-level scaling up in llms , author=. EMNLP , year=

  149. [161]

    Song, Wei and Huang, Zhenya and Cheng, Cheng and Gao, Weibo and Xu, Bihan and Zhao, GuanHao and Wang, Fei and Wu, Runze , booktitle =

  150. [162]

    arXiv preprint arXiv:2311.12022 , year=

    Gpqa: A graduate-level google-proof q&a benchmark , author=. arXiv preprint arXiv:2311.12022 , year=

    arXiv

  151. [163]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    arXiv

  152. [164]

    ICLR , year=

    Let's verify step by step , author=. ICLR , year=

  153. [165]

    Proceedings of NAACL , year=

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs , author=. Proceedings of NAACL , year=

  154. [166]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    arXiv

  155. [167]

    2025 , journal =

    Gemma 3 Technical Report , author=. 2025 , journal =

    2025

  156. [168]

    2025 , howpublished =

    Mistral-Small-3.2-24B-Instruct-2506 , author =. 2025 , howpublished =

    2025

  157. [169]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  158. [170]

    2024 , howpublished =

    Llama 3.3 70B Instruct , author =. 2024 , howpublished =

    2024

  159. [171]

    2025 , howpublished =

    Gemini 2.5 Flash-Lite , author =. 2025 , howpublished =

    2025

  160. [172]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.