Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 200-token preview beats full LLM ensembles at lower cost.

desk verdict LightRouter's preview-then-route-then-merge idea is new and worth a look, but the paper's central ablation is missing: it never shows the selector, rather than the extra aggregation layer, drives the gains. read the letter →

arxiv 2505.16221 v1 pith:YVGCSAD4 submitted 2025-05-22 cs.AI

Yifan Zhang , Xinkui Zhao , Zuxin Wang , Guanjie Cheng , Yueshen Xu , Shuiguang Deng , Jianwei Yin This is my paper · ORCID
classification cs.AI
keywords LightRouterLLMroutingmodelensemblingboottokenstop-kselectioninferencecostmulti-agentorchestrationopen-sourceLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LightRouter is an orchestration scheme that keeps the accuracy benefits of ensembling many large language models while avoiding the cost of running them all. It asks every candidate model to produce only a short prefix, uses a lightweight selector to rank those prefixes, and lets only the top two candidates finish their answers before an aggregator merges the results. The paper claims this two-stage preview-then-route-then-merge design matches or beats full ensemble baselines, improving accuracy by up to 25%, and approaches much more expensive models while cutting inference cost by up to 27%. If the claim holds, most of the value of multi-model collaboration can be captured with a small, cheap routing overhead.

What carries the argument

The central mechanism is a two-stage scheduler: a boot-token stage in which each candidate model emits only a short initial response (the paper uses 200 tokens), and a selection stage in which a Selector model ranks these partial responses by predicted semantic consistency and keeps only the top k candidates, with k=2 in the main experiments. The chosen outputs are then merged by an Aggregator, and the process can repeat over several layers. The theoretical backing is a variance-reduction argument: if candidate consistency scores are independent with mean mu and variance sigma squared, merging k outputs lowers variance to sigma squared over k, so selecting the top k is a budget-controlled way to capture ensemble gains while avoiding the degradation that low-quality outliers can cause.

What would settle it

Run LightRouter's selector on a sample of queries, but instead of letting only the top two candidates finish, let all five candidates finish and score their full responses against ground truth; if the selector's ranking of 200-token prefixes correlates no better than chance with full-response quality, the framework's savings come from the merging step alone rather than from routing.

Watch

Extended reading notes

Core claim

On the paper's own terms, LightRouter establishes that a small, fixed number of top-ranked candidates chosen from partial outputs is enough to preserve and often exceed the accuracy gains of full ensembling. The paper reports consistent wins over every candidate model in its pool, over a reranking baseline, and over the Mixture-of-Agents ensemble baseline across GSM8K, MATH, GPQA-Diamond, HumanEval, and MMLU, with the largest gains on math and code tasks. On MT-Bench, it reaches scores comparable to expensive proprietary models while spending less per query, reducing API costs by about 25% relative to MoA and more than 27% relative to one strong reasoning model. The method requires no prior knowledge of model strengths and no training, and all experiments use open-source models.

Load-bearing premise

The load-bearing premise is that a selector can judge the quality of a model's full answer from just the first 200 tokens, so that filtering to the top two of five candidates preserves the ensemble benefit.

Editorial extensions

If this is right

  • Practitioners can spend only 200 boot tokens per candidate model and still capture most of the accuracy benefit of running the full ensemble.
  • The gap between cheap open-source model pools and expensive proprietary models narrows on standard knowledge and reasoning benchmarks without task-specific tuning.
  • Cost savings grow with the size of the candidate pool, because full generation is limited to the chosen top k rather than every model.
  • Adding more aggregation layers yields diminishing returns: the paper observes a small gain at layer three and a decline at layer four, so two or three layers form the practical operating range.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the selector is just a prompted model, its ranking skill is task-dependent; a cheap probe comparing prefix rankings against full-response scores would tell users when LightRouter's savings are safe to rely on.
  • The variance-reduction argument assumes roughly independent candidate errors, but models from the same family often share training data, so the realized merging gain may be smaller than sigma squared over k, making the selector's job harder than the theory suggests.
  • A natural extension is to make the token budget adaptive: spend more boot tokens only when the selector's top choices disagree, while keeping the cheap path for queries where the top-ranked prefix is clearly better.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes LightRouter, a multi-LLM orchestration framework that queries all candidate models for a short boot-token prefix, uses a Selector LLM to rank the candidates and keep the top-k, and then iteratively aggregates the selected full outputs with an Aggregator LLM across multiple layers. The central claims are that this top-k routing preserves or improves on full-ensemble baselines (MoA, LLM-Blender) while reducing inference cost, and that with open-source candidates it can match proprietary frontier models. The evaluation covers GSM8K, MATH, GPQA-Diamond, HumanEval, MMLU, and MT-Bench, with a cost analysis in Section 4.4 and a Pareto-frontier plot in Figure 7.

Significance. If the central claims are supported, the contribution is significant: a cheap, training-free, preview-then-route-then-merge orchestration that outperforms full ensembles and expensive APIs would be practically valuable and relevant to the growing LLM-routing literature. The paper has real strengths: the evaluation uses independent public benchmarks, all components are open-source, the method is simple to implement, and no parameter fitting to the test targets appears to have occurred. The cost-vs-quality Pareto framing in Figure 7 is also useful. However, because the key attribution of gains to the selector is not isolated and the cost model is incomplete, the specific routing mechanism's contribution is not yet established.

major comments (4)
  1. [Section 3.2, Table 1] Table 1 reports a single point estimate per benchmark for each method, with no error bars, seeds, or confidence intervals, and Section 3.1 states that experiments use 'subsets of the MATH and MMLU benchmarks' without specifying the subset sizes or selection protocol. Several claimed advantages are small (e.g., MMLU 91.56 vs MoA 90.73; MATH 94.30 vs MoA 94.00), so without variance information the headline claim that LightRouter 'consistently surpasses' baselines is not statistically supported. The violin plot in Figure 4 addresses MT-Bench only and cannot substitute for benchmark-level uncertainty.
  2. [Sections 4.2 and 4.4] The paper never ablates the selector. The comparison in Table 1 pits LightRouter (top-2, two layers) against MoA (layer=1, all candidates) and LLM-Blender (K=1), so the gain could come from the extra aggregation layer or from the aggregator prompts rather than from prefix-based top-k routing. Section 4.4 shows that adding a third layer changes the MT-Bench score by 0.02 (9.37 to 9.39), indicating that the aggregator configuration alone is a non-negligible factor. I request a no-filter control (e.g., k=5 with the same two-layer aggregation) and a random-top-2 control with the same architecture, on at least GSM8K, MATH, and MMLU. Without such an ablation, the claimed 'up to 25% improvement' cannot be attributed to the routing mechanism.
  3. [Section 2.1.2, Total Cost] The cost equation T_total = Σ_{fi∈S} T(fi) + T(merge) omits the boot tokens generated by all N candidate models and the tokens produced by the Selector model. Since the abstract and Section 4.4 quantify cost savings (up to 27% per-query reduction; 7.46% vs LLM-Blender; 24.85% vs MoA), the comparison is unverifiable until a complete token formula—including N × boot_tokens and selector_tokens—is given and used in the cost calculations in Table 2 and Figure 7. The omission is material because the method's main selling point is cost efficiency.
  4. [Section 2.1.2, Merging and Top-k Selection] The theoretical model asserts that merging k independent outputs reduces variance to σ²/k and that S(Y_final, Y_true) ≤ (1/k) Σ S(f_i), but no derivation, assumptions, or citations support these statements. In particular, the merging step is performed by an LLM (f_merge), not by averaging, so the variance-reduction argument is not automatically valid; the inequality rules out the possibility that a good aggregator outperforms the average consistency of its inputs, which contradicts the empirical MoA results later in the paper. This makes the top-k objective k* = arg max E[S_k] − λ T_total(k) a restatement rather than a derived optimum. The NeurIPS Checklist (item 3) claims all theoretical results have complete proofs; this is inaccurate.
minor comments (7)
  1. [Section 3.1] The sentence 'Following prior works..., we conduct experiments on subsets of the MATH and MMLU benchmarks' should state the exact subset sizes and how they were sampled; otherwise the reported scores are not reproducible.
  2. [Section 4.4] The cost figures in the text (0.0031, 0.0037, 0.0047) appear to be per-query USD, but Table 2 reports per-1M-token prices; clarify how output lengths and candidate counts are converted into per-query costs.
  3. [Figure 4] The caption says 'performance distribution across repeated runs' but does not state the number of runs, sampling temperature, or whether the same prompts are used; without these details the variance comparison is not interpretable.
  4. [Table 1] The legend says the best individual model is highlighted in red and the best comparative method in green, but in the text-only version no highlighting is visible; consider using symbols or annotations that survive the submission format.
  5. [Section 5.1] The phrase 'PAIRRANKER' should be 'PairRanker' for consistency with the cited LLM-Blender paper.
  6. [Checklist item 8] 'All calculation is done with api call' is unclear; specify the API provider, request limits, and compute resources as claimed.
  7. [Section 7] The limitation that LightRouter 'only supports selection among non-reasoning models' is in tension with the reported gains on GSM8K and MATH, which are reasoning tasks; the authors should clarify how this limitation applies to the main results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: results are external benchmark measurements with no fitted parameter renamed as a prediction.

full rationale

LightRouter's derivation chain contains no step in which a claimed prediction is equivalent by construction to an input, and no load-bearing self-citation. Section 2.1.2 asserts variance reduction to sigma-squared/k and the bound S(Y_final, Y_true) <= (1/k) sum S(f_i) without proof, but these are unproven premises rather than conclusions that reduce to themselves; the top-k selection argument is conditional on those premises. The empirical headline claims rest on external benchmarks (MMLU, GSM8K, MATH, HumanEval, GPQA-Diamond, MT-Bench) with all baseline methods sharing the same candidate pool, so there is no fitting of a parameter to the target metric that is then re-reported as a prediction. The main genuine concerns are experimental, not circular: hyperparameters (200 boot tokens, k=2, two layers) are selected from MT-Bench and MT-Bench is also part of the headline evaluation, no no-filter/random-top-2 ablation isolates the selector's contribution, and the cost formula T_total = sum_{fi in S} T(fi) + T(merge) omits boot-token costs for all candidates and the selector's own generation. These weaken the attribution and the cost claims, but they are missing-control/missing-proof issues, not instances of a result being forced by its own definition.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or model-level entities are introduced; the 'boot token' is a design parameter, not an entity. The load-bearing assumptions are about the informativeness of partial outputs (empirically plausible but not isolated) and about variance reduction by LLM-based aggregation (asserted, not derived). The free parameters (k=2, token=200, layers=2, Deepseek-v3 as selector/aggregator) are chosen by hand or from sweeps and materially affect the reported numbers.

free parameters (4)
  • top-k selected models = 2
    Primary setting uses top-2 selection; sweep over k=1..4 in Section 4.4; k=2 used for all main results.
  • boot token budget = 200
    Selected as the 'Recommended Zone' from the MT-Bench trade-off in Figure 6; main results use 200 tokens.
  • number of layers = 2
    Primary two-layer configuration; layers 2-4 tested in Section 4.4; accuracy declines at 4.
  • selector and aggregator model = Deepseek-v3
    The same model serves as both Selector and Aggregator; Figure 5 shows aggregator choice affects results, but the selector choice is not ablated.
assumptions (5)
  • domain assumption Early errors in autoregressive generation propagate and degrade later outputs (Section 2.1.1).
    Motivates routing decisions on early tokens; plausible but not proven, and not tested directly.
  • domain assumption Candidate consistency scores S(fi) are independent random variables with mean mu and variance sigma^2 (Section 2.1.2).
    Assumed to justify variance reduction; no evidence of independence or identical distribution.
  • ad hoc to paper Merging k outputs with fmerge reduces variance to sigma^2/k (Section 2.1.2).
    Stated without proof; fmerge is an LLM, not an averaging operation, so the analogy to sample averaging is not established.
  • ad hoc to paper Merged consistency is bounded by the average of the individual consistencies, S(Yfinal,Ytrue) <= (1/k) sum S(fi) (Section 2.1.2).
    Used to justify filtering low-quality outputs, but the inequality is asserted and is not obviously true; a good merge can exceed the average.
  • domain assumption A 200-token prefix is sufficient for the Selector to estimate full-response quality (Section 2.2 and Figure 6).
    Core premise of the method; supported only by the aggregate MT-Bench trade-off, not by an ablation showing selector-vs-random selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead." pith.science (2026). https://pith.science/paper/YVGCSAD4

@misc{pith2026250516221,
  author       = {Pith},
  title        = {Pith review of: LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YVGCSAD4}},
  note         = {Machine review of arXiv:2505.16221}
}
read the original abstract

The rapid advancement of large language models has unlocked remarkable capabilities across a diverse array of natural language processing tasks. However, the considerable differences among available LLMs-in terms of cost, performance, and computational demands-pose significant challenges for users aiming to identify the most suitable model for specific tasks. In this work, we present LightRouter, a novel framework designed to systematically select and integrate a small subset of LLMs from a larger pool, with the objective of jointly optimizing both task performance and cost efficiency. LightRouter leverages an adaptive selection mechanism to identify models that require only a minimal number of boot tokens, thereby reducing costs, and further employs an effective integration strategy to combine their outputs. Extensive experiments across multiple benchmarks demonstrate that LightRouter matches or outperforms widely-used ensemble baselines, achieving up to a 25% improvement in accuracy. Compared with leading high-performing models, LightRouter achieves comparable performance while reducing inference costs by up to 27%. Importantly, our framework operates without any prior knowledge of individual models and relies exclusively on inexpensive, lightweight models. This work introduces a practical approach for efficient LLM selection and provides valuable insights into optimal strategies for model combination.

Figures

Figures reproduced from arXiv: 2505.16221 by the authors.

Figure 1
Figure 1. Illustration of the LightRouter architecture. The framework routes prompts through a selector agent and multiple layers of aggregator agents, progressively combining outputs from a pool of candidate models to generate the final response. Different agent roles are color-coded. An overview of the LightRouter architecture is depicted in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Grouped bar chart of MT-Bench scores for all evaluated models and methods. Each group [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Boxplot comparison of individual candidate scores and aggregated MoA score across six [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Performance distribution of different LLMs across repeated runs on MT-Bench. Our method [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance of different LLMs when used as aggregators. Moreover, to systematically evaluate the impact of the aggregator, we select a diverse set of open-source LLMs with varying capacities and instruction-following abili￾ties to serve as aggregators. Specifically, we…
Figure 6
Figure 6. Figure 6: Trade-off analysis between token bud￾get, accuracy, and cost. Accuracy increases marginally with more tokens, while cost rises steadily. The 200-token point provides a prac￾tical balance. 2 4 6 8 10 Cost per 1000 Queries (USD) 7.0 7.5 8.0 8.5 9.0 9.5 Performance Score …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Balancing Information Accuracy and Response Timeliness in Networked LLMs

    cs.LG 2025-08 conditional novelty 5.0 of 10

    For binary questions, combining m specialized LLMs with a Bayesian majority rule improves accuracy, and the paper derives the optimal m that trades accuracy against system delay.

Reference graph

Works this paper leans on

31 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [3]

    ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models

    Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, Ben He, Shanshan Jiang, and Bin Dong. Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models.arXiv preprint arXiv:2303.16421,

    work page Pith review arXiv

  3. [4]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  4. [6]

    What is the role of small models in the llm era: A survey.arXiv preprint arXiv:2409.06857,

    Lihu Chen and Gaël Varoquaux. What is the role of small models in the llm era: A survey.arXiv preprint arXiv:2409.06857,

    arXiv

  5. [8]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    arXiv

  6. [9]

    Graphrouter: A graph-based router for llm selections

    Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: A graph-based router for llm selections. arXiv preprint arXiv:2410.03834,

    arXiv

  7. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  8. [12]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

Show all 31 references
  1. [13]

    Routereval: A comprehensive benchmark for routing llms to explore model-level scaling up in llms.arXiv preprint arXiv:2503.10657,

    Zhongzhan Huang, Guoming Ling, Vincent S Liang, Yupei Lin, Yandong Chen, Shanshan Zhong, Hefeng Wu, and Liang Lin. Routereval: A comprehensive benchmark for routing llms to explore model-level scaling up in llms.arXiv preprint arXiv:2503.10657,

    arXiv

  2. [15]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561,

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561,

    arXiv

  3. [16]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  4. [17]

    Simcls: A simple framework for contrastive learning of abstractive summarization.arXiv preprint arXiv:2106.01890,

    Yixin Liu and Pengfei Liu. Simcls: A simple framework for contrastive learning of abstractive summarization.arXiv preprint arXiv:2106.01890,

    arXiv

  5. [19]

    Summareranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization.arXiv preprint arXiv:2203.06569,

    Mathieu Ravaut, Shafiq Joty, and Nancy F Chen. Summareranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization.arXiv preprint arXiv:2203.06569,

    arXiv

  6. [20]

    ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, et al

    URL https://openreview.net/forum? id=Ti67584b98. ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, et al. Deepseek-prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal ...

    arXiv

  7. [21]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    arXiv

  8. [23]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    11 Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    arXiv

  9. [24]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Qwen Team. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    arXiv

  10. [25]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    arXiv

  11. [26]

    Knowledge fusion of large language models.arXiv preprint arXiv:2401.10491,

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models.arXiv preprint arXiv:2401.10491,

    arXiv

  12. [27]

    Fusing models with complementary expertise.arXiv preprint arXiv:2310.01542,

    Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric Xing, and Mikhail Yurochkin. Fusing models with complementary expertise.arXiv preprint arXiv:2310.01542,

    arXiv

  13. [28]

    Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692,

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692,

    arXiv

  14. [29]

    Which llm to play? convergence-aware online model selection with time-increasing bandits

    Yu Xia, Fang Kong, Tong Yu, Liya Guo, Ryan A Rossi, Sungchul Kim, and Shuai Li. Which llm to play? convergence-aware online model selection with time-increasing bandits. InProceedings of the ACM Web Conference 2024, pages 4059–4070,

    2024

  15. [30]

    Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068,

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068,

    arXiv

  16. [31]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence.arXiv preprint arXiv:2406.11931,

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence.arXiv preprint arXiv:2406.11931,

    arXiv

  17. [1991]

    Mixtral of experts.arXiv preprint arXiv:2401.04088,

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    arXiv

  18. [2017]

    Large language model routing with benchmark datasets.arXiv preprint arXiv:2309.15789,

    Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets.arXiv preprint arXiv:2309.15789,

    arXiv

  19. [2020]

    Reconcile: Round-table conference improves reasoning via consensus among diverse llms.arXiv preprint arXiv:2309.13007,

    Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms.arXiv preprint arXiv:2309.13007,

    arXiv

  20. [2021]

    Routing to the expert: Efficient reward-guided ensemble of large language models.arXiv preprint arXiv:2311.08692,

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models.arXiv preprint arXiv:2311.08692,

    arXiv

  21. [2023]

    Oceangpt: A large language model for ocean science tasks.arXiv preprint arXiv:2310.02031,

    Zhen Bi, Ningyu Zhang, Yida Xue, Yixin Ou, Daxiong Ji, Guozhou Zheng, and Huajun Chen. Oceangpt: A large language model for ocean science tasks.arXiv preprint arXiv:2310.02031,

    arXiv

  22. [2024]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  23. [2025]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    10 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    2009 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.