Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Bench-CoE: a Framework for Collaboration of Experts from Benchmark

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Coarse per-subject accuracy from public benchmark leaderboards is enough label data to train an expert router that beats every constituent model in aggregate, at no extra inference cost.

desk verdict Subject-level routing from benchmark leaderboards is a new and cheap idea, but the paper's OOD transfer claim lacks the controls to support it. read the letter →

arxiv 2412.04167 v1 pith:QRDLASBS submitted 2024-12-05 cs.AI

Yuanshuai Wang , Xingjian Zhang , Jinkun Zhao , Siwei Wen , Peilin Feng , Shuhao Liao , Lei Huang , Wenjun Wu This is my paper · ORCID
classification cs.AI
keywords mixtureofexpertsLLMroutingbenchmark-drivensubject-levellabelsquery-levelexpertcollaborationout-of-distributiongeneralizationmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bench-CoE tries to establish that collaboration among off-the-shelf LLM experts does not require expensive per-query labeling: the coarse per-subject accuracy numbers that benchmark leaderboards already publish are sufficient training signal for a router that assigns each query to the best expert. The paper formalizes two variants — Query-Level, which labels every query with the expert that answers it best, and Subject-Level, which labels every query with the expert that tops that query's subject on the leaderboard. Across language (MMLU-Pro, Winogrande, Big-Bench-Hard) and multimodal (MMMU, MMStar) evaluations, both variants beat the best single constituent model in aggregate, and the subject-level router generalizes to out-of-distribution benchmarks better than the query-level one. If these results hold, expert collaboration becomes nearly free: no expert training, no extra inference to create routing labels, and only the selected expert runs at serve time.

What carries the argument

The load-bearing object is the subject-level label. For a subject $S^b$ of a benchmark $D^K$, every query in that subject is assigned the label $$$y^{{K,b}}$_i = \arg\max_l \frac{1}{|b|}\sum_{i=1}^{|b|} $p^{{K,b}}$_{l,i},$$ the expert $M_l$ with the highest average performance on the subject. The label is read directly off the benchmark leaderboard, replacing the fine-grained query-level label $$$y^{{K,b}}$_i = \arg\max_l $p^{{K,b}}$_{l,i},$$ which would require running all experts on every query. The router $R^\mathcal{L}_\theta(x)$, instantiated as a BERT classifier for text and a vision-language model for multimodal inputs, is trained with a classification loss on benchmark queries paired with either label type, and at inference routes each query to $\arg\max_l R^\mathcal{L}_\theta(x)$. The subject-level substitution carries the argument: it converts public benchmark evaluations into free training signal, and the paper attributes its out-of-distribution robustness to subject labels tracking stable subject-level strengths rather than instance-level input patterns.

What would settle it

Take the paper's out-of-distribution pair (router trained on MMLU-Pro, tested on Big-Bench-Hard), and compute the per-subject oracle there: route every query in each Big-Bench-Hard subject to the expert that actually scores highest on that subject, and compare that oracle accuracy with the subject-level router's accuracy and with the best single expert. If the router trails the oracle by a wide margin and no longer beats the best single expert, the claimed transfer of leaderboard rankings across datasets fails; if it matches the oracle, subject transfer is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that benchmark evaluations are a usable and inexpensive source of routing labels for collaboration of experts. In the Query-Level formulation, each query is labeled with the expert that scores best on that query, which requires running every expert on every query; this achieves large in-distribution gains (64.28% on MMLU-Pro versus 52.04% for the best single expert) but overfits across distributions. In the Subject-Level formulation, every query in a subject inherits the label of the expert with the highest average score on that subject, so labels come directly from published leaderboard results with no extra inference cost; this gives a smaller in-distribution gain (52.24% on MMLU-Pro, 51.78% on MMMU) but transfers better, beating the best single expert on Big-Bench-Hard (69.91% versus 66.35%) and on MMStar (60.09% versus 59.22%) after training on MMLU-Pro and MMMU respectively. The paper also reports that routing four models of at most 9B parameters beats much larger individual models (Llama-3-70B, Mixtral-8x7B-Instruct, Yi-1.5-34B-Chat) on MMLU-Pro, and concludes that a router trained on benchmark labels selects the right expert often enough to lift aggregate accuracy whenever the experts have complementary subject strengths.

Load-bearing premise

The expert that tops a subject on the training benchmark is assumed to remain the best expert for that subject on the test benchmark and at deployment: the per-subject leaderboard ranking must transfer across data distributions, or the subject-level router routes to a champion that no longer is one.

Editorial extensions

If this is right

  • A router trained only on public benchmark data (queries plus leaderboard subject accuracy) can beat the best constituent expert in aggregate, so assembling an expert panel does not require running every expert on every training query.
  • Subject-level routing transfers across benchmarks: a router trained on MMLU-Pro lifts aggregate accuracy on Big-Bench-Hard, and one trained on MMMU lifts accuracy on MMStar, so per-subject expert strengths appear stable across distributions.
  • Only the selected expert executes at inference, so serving cost stays at the level of a single model, and the routed panel of small models (at most 9B parameters) can outperform much larger single models on the same benchmark.
  • Query-level routing is preferable when training and test data share a distribution; subject-level routing is preferable when the test distribution is shifted or unknown.
  • Keeping the system current with new models requires only updating the per-subject leaderboard champion rather than relabeling the routing dataset from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the subject-level gain is bounded by the stability of per-subject rankings between the label benchmark and the deployment benchmark; on any test set whose per-subject winners differ from the training leaderboard, the router would systematically route to a stale champion and the gain should shrink or reverse.
  • My inference: the query-level router's only reported loss against the best single model (MMStar, −3.22%) points to the router's input modality as the bottleneck — with text-only inputs, queries that are distinguishable only by image content cannot be separated, and a multimodal router is the direct test of this diagnosis.
  • My inference: the framework implies a portfolio rule for choosing experts — adding a model that wins a niche subject but is weak overall should help more than adding a balanced all-rounder, because routing gains come from the spread of complementary strengths; the paper's Appendix D dominance caveat is the boundary case where that spread vanishes.
  • My inference: a stress test worth running is finer-grained subject splits — training the subject-level router on narrow subjects (e.g., separate science subfields rather than one 'science' label) should interpolate between subject-level and query-level performance on out-of-distribution data and reveal how much of the generalization comes from label coarseness itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Bench-CoE, a framework for routing queries among a set of LLM/LMM experts based on benchmark-derived labels. It formalizes two routing approaches: query-level, where each input is labeled with the best-performing expert on that exact query, and subject-level, where each input is labeled with the expert that performs best on the query's benchmark subject. The router is trained on these labels and selects one expert per query at inference. Experiments are run on language tasks (MMLU-Pro, Winogrande, Big-Bench-Hard) and multimodal tasks (MMMU, MMStar) under naive, in-distribution, and out-of-distribution scenarios. The paper reports that the subject-level router outperforms the best individual expert in the out-of-distribution cases (BBH +3.56%, MMStar +0.87%) and that the query-level router performs well in-distribution, concluding that Bench-CoE consistently beats all single models.

Significance. If the central claim is correct, the paper offers a simple and inexpensive way to construct routable multi-expert systems: subject-level leaderboard results from existing benchmarks become the training signal for the router, avoiding instance-level labeling and large-scale training. The formal distinction between query-level and subject-level routing is useful, and the observation that subject-level routing generalizes better out-of-distribution is plausible and worth testing. The authors also release code, which supports reproducibility. However, the significance is currently limited by the absence of control baselines in the out-of-distribution experiments, the in-sample nature of the headline naive results, and the lack of statistical significance testing. These issues bear directly on the paper's main generalization claim, so the significance as presented is not yet fully established.

major comments (4)
  1. [Section 4.3, Tables 6-7] The out-of-distribution experiments lack the controls needed to support the claim that subject-level benchmark labels transfer across datasets. There is no comparison to a router trained with shuffled subject-to-expert assignments, to a feature-only baseline (e.g., text length, token statistics), or to an oracle subject classifier on the target benchmark. Since Big-Bench-Hard and MMStar do not provide subject labels for the router to validate against (Table 1 lists Has Subject = No for BBH), the reported gains of +3.56% and +0.87% could result from the router exploiting surface text cues that happen to correlate with expert accuracy, rather than from the benchmark-derived subject-expert ordering. This is load-bearing for the central generalization claim and must be addressed with explicit control experiments.
  2. [Section 4.1 and Appendix B.1, Tables 2-3] The naive evaluation scenario trains and evaluates the router on the same MMLU-Pro validation set (and similarly on the same MMMU subset), as stated in B.1: "we conducted both training and testing on the validation set." The large gains in Table 2 (64.28% vs. 52.04%) and Table 3 (51.78% vs. 47.67%) are therefore in-sample results, not evidence of generalizable routing. The paper acknowledges this is a feasibility test, but the abstract and Section 4.1 phrase the conclusion more broadly, and the same-set results should not be used as headline evidence for the claim that Bench-CoE outperforms every single model.
  3. [Appendix B.1 vs. Table 1] There is an internal inconsistency about whether Big-Bench-Hard has subject labels. Table 1 lists "Has Subject = No" for Big-Bench-Hard, while B.1 states that "only the MMLU-Pro and Big-Bench-Hard datasets include manually annotated discipline-specific labels." This contradiction matters because the subject-level out-of-distribution experiment in Section 4.3 requires some notion of subject on the target dataset to verify that the router's subject predictions are meaningful. If BBH truly lacks subject labels, the subject-level router is being evaluated on a dataset where its routing rationale cannot be checked, further weakening the transfer claim.
  4. [Section 4, Tables 2-7] All reported results lack confidence intervals, error bars, or significance tests, and several gains are small (e.g., +0.2% in Table 2 and +0.87% in Table 7). Without repeated runs or a statistical test, it is impossible to tell whether these improvements exceed noise. In addition, the router hyperparameters (BERT/VLM architectures, training epochs, learning rates, or random seeds) are not reported, which limits reproducibility. The code is available, but the missing experimental details prevent an independent assessment of result stability and sensitivity.
minor comments (6)
  1. [Throughout (e.g., Section 3.3)] The term "subjective-level" is used in several places (e.g., the Section 3.3 heading) where "subject-level" is clearly intended; please correct these typos.
  2. [Figure 3 caption] The caption calls Gemma-2-9b-it "the top MoE model," but Gemma-2-9b-it is a dense transformer, not a mixture-of-experts model; the caption should be corrected.
  3. [Section 3.4] The evaluation scenarios refer to "D1" and "D2" before defining them; please introduce the notation explicitly or rephrase to avoid ambiguity.
  4. [Table 4] The heading "Performance on Winogrande" says "the route" in the text; it should be "the router."
  5. [Section 4.1] The heading "Comparision to Larger LLMs" contains a typo; it should be "Comparison to Larger LLMs."
  6. [Appendix D] The limitation described in Appendix D is useful and honest, but its implications for the paper's claims are not discussed in the main text; specifically, the Abstract's unconditional statement that Bench-CoE "outperforms any single model" should be qualified by the condition that experts must have complementary strengths, as conceded in Appendix D.

Circularity Check

1 steps flagged · score 3.0 of 10

One in-sample query-level result reduces to its own training labels; the central OOD claim is independent, though its interpretation is under-tested.

  1. fitted input called prediction [Section 3.2 Eq. (7); Section 3.4 'Naive Evaluation Scenario'; Section 4.1 Table 2]
    "The id of the model with the best performance is designated as the query-level label for that query, as defined below. yK,b i = arg max l pK,b l,i . (7) ... In the language task experiment, Bench-CoE was trained and evaluated on the same dataset MMLU-Pro[30]. This setup tests the router’s ability to select the most suitable model when both training and testing datasets are the same."

    The query-level label for each training query is literally the argmax over the experts' per-query accuracies on that same query (Eq. 7). The naive experiment then trains the router on those labels and evaluates on the same MMLU-Pro test split. Thus the router's 'prediction' for a query is scored against the very per-query winner that generated its training label; the reported +12.24% gain is an in-sample fit, not an independent prediction. The paper honestly labels this scenario 'Naive Evaluation' and does not use it as evidence of generalization, so the circularity is localized rather than load-bearing for the main OOD claim.

full rationale

The only concrete circular step is the naive query-level experiment, where the training label for a query is the best-performing expert on that same query and evaluation is performed on the same split. That result is self-referential by construction. The paper itself flags this as a basic-feasibility check rather than a generalization result. The in-distribution experiments (Winogrande; MMMU train/test split) and out-of-distribution experiments (MMLU-Pro to Big-Bench-Hard; MMMU to MMStar) use separate data splits, so their reported gains are not forced by the label-generation equations. No fitted constants are smuggled into a derivation, and no load-bearing self-citation chain appears: the cited benchmarks and models are external, not prior work by the same authors. Appendix D's discussion of unsuitable scenarios is a genuine limitation statement, not a circular maneuver. The skeptic's concern that the OOD gains may stem from surface cues rather than subject semantics is a correctness and control-condition issue, not a circularity issue: the paper's OOD claim would be unverified, but it is not equivalent to its inputs by construction. Overall, the central derivation is self-contained and independently testable, with one localized in-sample component, warranting a low-to-moderate score of 3.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new particles, forces, or physical constants are introduced. The method's load-bearing assumptions are empirical transfer assumptions about router learning and expert diversity, not fitted derivation constants.

free parameters (1)
  • Router hyperparameters (BERT/VLM training settings) = Not reported
    The router is trained with unspecified learning rate, epochs, batch size and architecture details; these choices affect routing accuracy and hence the reported gains.
assumptions (3)
  • domain assumption A text-trained router can infer which expert will answer a query correctly from the query text alone.
    Router training in Section 3 assumes text features predict expert performance; the MMStar query-level failure in Table 7 shows this fails when image cues are needed.
  • domain assumption Subject-level benchmark scores transfer across data distributions.
    Out-of-distribution experiments in Section 4.3 train on MMLU-Pro/MMMU and test on Big-Bench-Hard/MMStar, assuming relative subject strengths remain stable.
  • domain assumption Candidate experts have complementary strengths rather than one dominant model.
    Appendix D states that when an expert significantly outperforms or underperforms all others, CoE cannot improve performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bench-CoE: a Framework for Collaboration of Experts from Benchmark." pith.science (2026). https://pith.science/paper/QRDLASBS

@misc{pith2026241204167,
  author       = {Pith},
  title        = {Pith review of: Bench-CoE: a Framework for Collaboration of Experts from Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRDLASBS}},
  note         = {Machine review of arXiv:2412.04167}
}
read the original abstract

Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks. To meet the demands of various tasks, an increasing number of LLMs-driven experts with diverse capabilities have been developed, accompanied by corresponding benchmarks to evaluate their performance. This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks. Bench-CoE includes a set of expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Moreover, we formulate Query-Level and Subject-Level approaches based on our framework, and analyze the merits and drawbacks of these two approaches. Finally, we conduct a series of experiments with vary data distributions on both language and multimodal tasks to validate that our proposed Bench-CoE outperforms any single model in terms of overall performance. We hope this method serves as a baseline for further research in this area. The code is available at \url{https://github.com/ZhangXJ199/Bench-CoE}.

Figures

Figures reproduced from arXiv: 2412.04167 by the authors.

Figure 1
Figure 1. The framework of Bench-CoE. Our Bench-CoE Frame [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of routing methods in LLMs combination: (a) The MoE model utilizes multiple FFNs as expert modules during [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance Across Subjects on MMLU Pro. Bench [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: The performance of Llama-3-70B and Bench-CoE on [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems

    cs.AI 2025-02 conditional novelty 4.0 of 10

    A survey that classifies LLM routing strategies into pre-generation and post-generation approaches and four implementation families, framed as a performance-cost optimization problem.

Reference graph

Works this paper leans on

37 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learn- ing natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Pro- cessing, pages 632–642, Lisbon, Portugal, 2015. Association for Computational Linguistics. 1

    work page 2015

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

    work page 2020

  3. [3]

    Frugalgpt: How to use large language models while reducing cost and improving performance

    Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. ArXiv preprint , abs/2305.05176,

    arXiv

  4. [4]

    Are we on the right way for evaluating large vision-language models? ArXiv preprint, abs/2403.20330,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? ArXiv preprint, abs/2403.20330,

    arXiv

  5. [5]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. ArXiv preprint, abs/2312.14238, 2023. 6

    arXiv 2023

  6. [6]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. ArXiv preprint, abs/2404.16821, 2024. 6

    arXiv 2024

  7. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168, 2021. 1

    arXiv

  8. [8]

    BERT: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers) , pages 4171–4186, Minne...

    work page 2019

Show all 37 references
  1. [9]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Sub- habrata Mukherjee, Victor R ¨uhle, Laks V . S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: cost-efficient and quality-aware query routing. In ICLR. OpenReview.net,

  2. [10]

    Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity. Journal of Machine Learning Re- search, 23(120):1–39, 2022. 2

    2022

  3. [11]

    Graphrouter: A graph-based router for llm selections

    Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: A graph-based router for llm selections. ArXiv preprint , abs/2410.03834, 2024. 2

    2024 arXiv

  4. [12]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b. ArXiv preprint , abs/2310.06825, 2023. 6

    arXiv

  5. [13]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. ArXiv preprint, abs/2401.04088, 2024. 7

    2024 arXiv

  6. [14]

    Llm- blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm- blender: Ensembling large language models with pairwise ranking and generative fusion. In ACL (1) , pages 14165– 14178. Association for Computational Linguistics, 2023. 2

    2023

  7. [15]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv preprint, abs/2408.03326, 2024. 6

    2024 arXiv

  8. [16]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1

    2023

  9. [17]

    Routing to the expert: Efficient reward-guided ensemble of large language models

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. In NAACL-HLT, pages 1964–1974. Association for Compu- tational Linguistics, 2024. 2, 3, 5

    1964

  10. [18]

    Introducing meta llama 3: The most capable openly available llm to date

    AI Meta. Introducing meta llama 3: The most capable openly available llm to date. Meta AI, 2024. 7

    2024

  11. [19]

    Hdmole: Mixture of lora experts with hierarchical routing and dynamic thresholds for fine-tuning llm-based asr mod- els

    Bingshen Mu, Kun Wei, Qijie Shao, Yong Xu, and Lei Xie. Hdmole: Mixture of lora experts with hierarchical routing and dynamic thresholds for fine-tuning llm-based asr mod- els. ArXiv preprint, abs/2409.19878, 2024. 2

    2024 arXiv

  12. [20]

    Routellm: Learning to route llms with preference data

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route llms with preference data. ArXiv preprint, abs/2406.18665, 2024. 2, 3, 5

    2024 arXiv

  13. [21]

    Smaug: Fixing failure modes of preference optimisation with dpo-positive

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. ArXiv preprint, abs/2402.13228, 2024. 6

    2024 arXiv

  14. [22]

    Rankmean: Module-level importance score for merging fine-tuned llm models

    Gabriel Perin, Xuxi Chen, Shusen Liu, Bhavya Kailkhura, Zhangyang Wang, and Brian Gallagher. Rankmean: Module-level importance score for merging fine-tuned llm models. In Findings of the Association for Computational Linguistics ACL 2024, pages 1776–1782, 2024. 2

    2024

  15. [23]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 1

    2019

  16. [24]

    SQuAD: 100,000+ questions for machine com- prehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine com- prehension of text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2383–2392, Austin, Texas, 2016. Association for Co...

    2016

  17. [25]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64 (9):99–106, 2021. 5

    2021

  18. [26]

    Polyrouter: A multi-llm querying system

    Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He. Polyrouter: A multi-llm querying system. ArXiv preprint, abs/2408.12320, 2024. 2

    2024 arXiv

  19. [27]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Sch ¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowd- hery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In ACL (Findings), pages 13003–1305...

    2023

  20. [28]

    Gemma Team. Gemma. 2024. 6

    2024

  21. [29]

    Tjong Kim Sang

    Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-independent named entity recogni- tion. In COLING-02: The 6th Conference on Natural Lan- guage Learning 2002 (CoNLL-2002) , 2002. 1

    2002

  22. [30]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. ArXiv preprint , abs/2406.01574, 2024. 1, 2, 5, 6

    2024 arXiv

  23. [31]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. ArXiv preprint, abs/2407.10671, 2024. 6

    2024 arXiv

  24. [32]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. ArXiv preprint, abs/2408.01800, 2024. 6

    2024 arXiv

  25. [33]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. ArXiv preprint, abs/2403.04652, 2024. 7

    2024 arXiv

  26. [34]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  27. [35]

    Vcr: Visual caption restoration

    Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, and Yoshua Ben- gio. Vcr: Visual caption restoration. arXiv preprint arXiv: 2406.06462, 2024. 1

    2024 arXiv

  28. [36]

    Eagle: Ef- ficient training-free router for multi-llm inference

    Zesen Zhao, Shuowei Jin, and Z Morley Mao. Eagle: Ef- ficient training-free router for multi-llm inference. ArXiv preprint, abs/2409.15518, 2024. 3

    2024 arXiv

  29. [37]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR. OpenReview.net, 2024. 1 A. Models and Datasets A.1. Language Task Models Qwen2-7B-Instruct is an instruction-...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.