{"id":"0d266895-0c02-40ab-883c-56ea91900b9c","arxiv_id":"2412.04167","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Bench-CoE trains a router on benchmark subject scores to send each query to the best-performing expert model, beating individual models in most but not all tested settings.","lead":"This paper describes Bench-CoE, a way to combine several existing AI models into one system by training a small router to send each question to the model most likely to answer it correctly. The router is trained using scores from public benchmarks, and the authors report that the combined system often beats every single model, though not in all tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD gains in Tables 6-7 may stem from surface text cues rather than benchmark subject labels; without control routing baselines, the core transfer claim is unverified.","rationale":"The reader's weakest assumption is that subject-level leaderboard rankings transfer across datasets. My concern is a sharper version: even if rankings transfer, the OOD experiments provide no evidence that the router is using those rankings, because no control rules out the possibility that the router exploits non-subject textual correlates. This is the single most load-bearing issue because the paper's novelty is precisely that coarse benchmark labels are sufficient for routing; if the OOD gains are explained by an unmodeled confound, the method's central contribution is not established. The in-distribution improvements are small (0.87% on Winogrande, 3.11% on MMMU), and the naive results (Table 2) are circular, so the OOD subject-level results are the main support for the abstract's claim. A shuffled-label control is a direct, low-cost experiment that would determine whether the benchmark-derived subject ordering is causally responsible. I do not recommend changing the verdict because the paper's subject-level results are plausible and the requested control is a standard addition; the reader's CONDITIONAL verdict already anticipates the need for further verification.","tokens_in":14607,"tokens_out":10139,"duration_ms":107496,"concrete_test":"Train the identical BERT subject-level router on the MMLU-Pro validation set, but with a fixed random permutation applied to the expert labels, so the predicted 'subject' no longer maps to the benchmark-derived best expert. Evaluate this shuffled-label router on the BBH validation set. If its accuracy is close to the reported 69.91% rather than to the best single model's 66.35%, the benchmark subject ordering is not the source of the gain; if it drops to ~66.35%, the ordering is necessary. Additionally, report a univariate baseline that routes by question length (e.g., above/below median word count to Mathstral vs. Gemma) on BBH for comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that benchmark subject labels suffice for cross-dataset routing rests on the OOD experiments (Tables 6-7). Yet the paper never verifies that the router's subject predictions are meaningful on the target benchmarks. BBH and MMStar have no subject labels (Table 1), so the router trained on MMLU-Pro/MMMU can only extract surface textual cues. No control is provided: there is no comparison to a router trained with shuffled subject-to-expert assignments, a text-length/feature baseline, or an oracle subject classifier. Under these conditions, the +3.56% (Table 6) and +0.87% (Table 7) could arise simply because the router separates inputs by length, presence of numbers, or other low-level features that happen to correlate with expert accuracy on those datasets, irrespective of the benchmark-derived expertise ordering. Appendix D rules out only the degenerate case of a dominating expert; it does not establish that the label information is actually used. The internal inconsistency between Section B.1 (claims BBH has discipline labels) and Table 1 (BBH: Has Subject = No) further clouds what the OOD test actually measures. The naive scenario (Table 2) cannot settle this because it trains and evaluates on the same split, making the result an optimistic bound rather than a test of transfer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Bench-CoE, a framework for routing queries among a set of LLM/LMM experts based on benchmark-derived labels. It formalizes two routing approaches: query-level, where each input is labeled with the best-performing expert on that exact query, and subject-level, where each input is labeled with the expert that performs best on the query's benchmark subject. The router is trained on these labels and selects one expert per query at inference. Experiments are run on language tasks (MMLU-Pro, Winogrande, Big-Bench-Hard) and multimodal tasks (MMMU, MMStar) under naive, in-distribution, and out-of-distribution scenarios. The paper reports that the subject-level router outperforms the best individual expert in the out-of-distribution cases (BBH +3.56%, MMStar +0.87%) and that the query-level router performs well in-distribution, concluding that Bench-CoE consistently beats all single models.","tokens_in":14790,"tokens_out":3626,"duration_ms":37415,"significance":"If the central claim is correct, the paper offers a simple and inexpensive way to construct routable multi-expert systems: subject-level leaderboard results from existing benchmarks become the training signal for the router, avoiding instance-level labeling and large-scale training. The formal distinction between query-level and subject-level routing is useful, and the observation that subject-level routing generalizes better out-of-distribution is plausible and worth testing. The authors also release code, which supports reproducibility. However, the significance is currently limited by the absence of control baselines in the out-of-distribution experiments, the in-sample nature of the headline naive results, and the lack of statistical significance testing. These issues bear directly on the paper's main generalization claim, so the significance as presented is not yet fully established.","major_comments":[{"comment":"The out-of-distribution experiments lack the controls needed to support the claim that subject-level benchmark labels transfer across datasets. There is no comparison to a router trained with shuffled subject-to-expert assignments, to a feature-only baseline (e.g., text length, token statistics), or to an oracle subject classifier on the target benchmark. Since Big-Bench-Hard and MMStar do not provide subject labels for the router to validate against (Table 1 lists Has Subject = No for BBH), the reported gains of +3.56% and +0.87% could result from the router exploiting surface text cues that happen to correlate with expert accuracy, rather than from the benchmark-derived subject-expert ordering. This is load-bearing for the central generalization claim and must be addressed with explicit control experiments.","section":"Section 4.3, Tables 6-7"},{"comment":"The naive evaluation scenario trains and evaluates the router on the same MMLU-Pro validation set (and similarly on the same MMMU subset), as stated in B.1: \"we conducted both training and testing on the validation set.\" The large gains in Table 2 (64.28% vs. 52.04%) and Table 3 (51.78% vs. 47.67%) are therefore in-sample results, not evidence of generalizable routing. The paper acknowledges this is a feasibility test, but the abstract and Section 4.1 phrase the conclusion more broadly, and the same-set results should not be used as headline evidence for the claim that Bench-CoE outperforms every single model.","section":"Section 4.1 and Appendix B.1, Tables 2-3"},{"comment":"There is an internal inconsistency about whether Big-Bench-Hard has subject labels. Table 1 lists \"Has Subject = No\" for Big-Bench-Hard, while B.1 states that \"only the MMLU-Pro and Big-Bench-Hard datasets include manually annotated discipline-specific labels.\" This contradiction matters because the subject-level out-of-distribution experiment in Section 4.3 requires some notion of subject on the target dataset to verify that the router's subject predictions are meaningful. If BBH truly lacks subject labels, the subject-level router is being evaluated on a dataset where its routing rationale cannot be checked, further weakening the transfer claim.","section":"Appendix B.1 vs. Table 1"},{"comment":"All reported results lack confidence intervals, error bars, or significance tests, and several gains are small (e.g., +0.2% in Table 2 and +0.87% in Table 7). Without repeated runs or a statistical test, it is impossible to tell whether these improvements exceed noise. In addition, the router hyperparameters (BERT/VLM architectures, training epochs, learning rates, or random seeds) are not reported, which limits reproducibility. The code is available, but the missing experimental details prevent an independent assessment of result stability and sensitivity.","section":"Section 4, Tables 2-7"}],"minor_comments":[{"comment":"The term \"subjective-level\" is used in several places (e.g., the Section 3.3 heading) where \"subject-level\" is clearly intended; please correct these typos.","section":"Throughout (e.g., Section 3.3)"},{"comment":"The caption calls Gemma-2-9b-it \"the top MoE model,\" but Gemma-2-9b-it is a dense transformer, not a mixture-of-experts model; the caption should be corrected.","section":"Figure 3 caption"},{"comment":"The evaluation scenarios refer to \"D1\" and \"D2\" before defining them; please introduce the notation explicitly or rephrase to avoid ambiguity.","section":"Section 3.4"},{"comment":"The heading \"Performance on Winogrande\" says \"the route\" in the text; it should be \"the router.\"","section":"Table 4"},{"comment":"The heading \"Comparision to Larger LLMs\" contains a typo; it should be \"Comparison to Larger LLMs.\"","section":"Section 4.1"},{"comment":"The limitation described in Appendix D is useful and honest, but its implications for the paper's claims are not discussed in the main text; specifically, the Abstract's unconditional statement that Bench-CoE \"outperforms any single model\" should be qualified by the condition that experts must have complementary strengths, as conceded in Appendix D.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The central idea is attractive and the authors have been transparent about some limitations, but the current evidence is not yet sufficient for a rigorous journal publication. The most urgent fixes are: (1) adding control routing baselines in the out-of-distribution experiments to rule out surface-feature exploitation, (2) resolving the contradiction about Big-Bench-Hard subject labels, and (3) reporting variance or significance measures. If these are addressed, the paper could become a valuable contribution to LLM routing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The subject-level variant is a genuinely simple and useful idea: use coarse benchmark subject averages as router labels, so you never need per-query inference. That part is worth taking seriously. The query-level variant is, as the paper admits, an abstraction of prior work like RouteLLM and ZOOTER. The new element is subject-level routing, and it does what it promises on most of the reported pairs, including two OOD transfers (Big-Bench-Hard +3.56, MMStar +0.87). That is a real result, if it holds.\n\nThe paper is honest in places: it flags the naive experiments as feasibility checks, and it acknowledges the query-level router fails on MMStar out-of-distribution. That failure by itself contradicts the abstract's \"outperforms any single model,\" so the claim is too broad.\n\nThe soft spots are concentrated where the paper is most ambitious. First, no error bars or significance tests anywhere; some margins are +0.2 or +0.87 points, which could be noise. Second, the OOD transfer claim is the least controlled part. BBH and MMStar have no subject labels, so the router trained on MMLU-Pro/MMMU can only rely on surface textual cues. There is no shuffled-label baseline, no text-length/feature baseline, no oracle-subject comparison. So the mechanism — that the router learns subject-level expertise ordering — is not established. The positive OOD numbers could just as plausibly come from the router separating inputs on length or sentence style. Third, there's an internal inconsistency: Section B.1 says Big-Bench-Hard has discipline-specific labels, but Table 1 says it does not. That needs to be resolved.\n\nThe naive experiments train and test on the same split, which the paper itself says is only a feasibility demonstration. That is fine, but the abstract leans on those numbers for the headline claim.\n\nAll that said, the core idea is sound and cheap, and the OOD results, even if mechanistically unclear, are worth chasing down. This deserves a serious referee, but the authors should be asked for controls, variance estimates, and a more careful claim. I'd bring it to our reading group as a case study in how leaderboard priors can be repurposed, and I'd cite it if I worked on routing.","headline":"Subject-level routing from benchmark leaderboards is a new and cheap idea, but the paper's OOD transfer claim lacks the controls to support it.","tokens_in":15409,"tokens_out":3075,"would_cite":true,"duration_ms":30646,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Coarse per-subject accuracy from public benchmark leaderboards is enough label data to train an expert router that beats every constituent model in aggregate, at no extra inference cost.","keywords":["mixture of experts","LLM routing","benchmark-driven routing","subject-level labels","query-level routing","expert collaboration","out-of-distribution generalization","multimodal routing"],"falsifier":"Take the paper's out-of-distribution pair (router trained on MMLU-Pro, tested on Big-Bench-Hard), and compute the per-subject oracle there: route every query in each Big-Bench-Hard subject to the expert that actually scores highest on that subject, and compare that oracle accuracy with the subject-level router's accuracy and with the best single expert. If the router trails the oracle by a wide margin and no longer beats the best single expert, the claimed transfer of leaderboard rankings across datasets fails; if it matches the oracle, subject transfer is confirmed.","tokens_in":14358,"feed_emoji":"🧭","tokens_out":11739,"duration_ms":97282,"temperature":0.7,"pith_summary":"Bench-CoE tries to establish that collaboration among off-the-shelf LLM experts does not require expensive per-query labeling: the coarse per-subject accuracy numbers that benchmark leaderboards already publish are sufficient training signal for a router that assigns each query to the best expert. The paper formalizes two variants — Query-Level, which labels every query with the expert that answers it best, and Subject-Level, which labels every query with the expert that tops that query's subject on the leaderboard. Across language (MMLU-Pro, Winogrande, Big-Bench-Hard) and multimodal (MMMU, MMStar) evaluations, both variants beat the best single constituent model in aggregate, and the subject-level router generalizes to out-of-distribution benchmarks better than the query-level one. If these results hold, expert collaboration becomes nearly free: no expert training, no extra inference to create routing labels, and only the selected expert runs at serve time.","feed_headline":"Train a router on benchmark scores and it beats every expert","feed_subtitle":"Routing each query to the expert that tops its benchmark subject lifts accuracy above every single model, for free.","key_machinery":"The load-bearing object is the subject-level label. For a subject $S^b$ of a benchmark $D^K$, every query in that subject is assigned the label $$$y^{{K,b}}$_i = \\arg\\max_l \\frac{1}{|b|}\\sum_{i=1}^{|b|} $p^{{K,b}}$_{l,i},$$ the expert $M_l$ with the highest average performance on the subject. The label is read directly off the benchmark leaderboard, replacing the fine-grained query-level label $$$y^{{K,b}}$_i = \\arg\\max_l $p^{{K,b}}$_{l,i},$$ which would require running all experts on every query. The router $R^\\mathcal{L}_\\theta(x)$, instantiated as a BERT classifier for text and a vision-language model for multimodal inputs, is trained with a classification loss on benchmark queries paired with either label type, and at inference routes each query to $\\arg\\max_l R^\\mathcal{L}_\\theta(x)$. The subject-level substitution carries the argument: it converts public benchmark evaluations into free training signal, and the paper attributes its out-of-distribution robustness to subject labels tracking stable subject-level strengths rather than instance-level input patterns.","core_discovery":"The central claim is that benchmark evaluations are a usable and inexpensive source of routing labels for collaboration of experts. In the Query-Level formulation, each query is labeled with the expert that scores best on that query, which requires running every expert on every query; this achieves large in-distribution gains (64.28% on MMLU-Pro versus 52.04% for the best single expert) but overfits across distributions. In the Subject-Level formulation, every query in a subject inherits the label of the expert with the highest average score on that subject, so labels come directly from published leaderboard results with no extra inference cost; this gives a smaller in-distribution gain (52.24% on MMLU-Pro, 51.78% on MMMU) but transfers better, beating the best single expert on Big-Bench-Hard (69.91% versus 66.35%) and on MMStar (60.09% versus 59.22%) after training on MMLU-Pro and MMMU respectively. The paper also reports that routing four models of at most 9B parameters beats much larger individual models (Llama-3-70B, Mixtral-8x7B-Instruct, Yi-1.5-34B-Chat) on MMLU-Pro, and concludes that a router trained on benchmark labels selects the right expert often enough to lift aggregate accuracy whenever the experts have complementary subject strengths.","pith_inferences":["My inference: the subject-level gain is bounded by the stability of per-subject rankings between the label benchmark and the deployment benchmark; on any test set whose per-subject winners differ from the training leaderboard, the router would systematically route to a stale champion and the gain should shrink or reverse.","My inference: the query-level router's only reported loss against the best single model (MMStar, −3.22%) points to the router's input modality as the bottleneck — with text-only inputs, queries that are distinguishable only by image content cannot be separated, and a multimodal router is the direct test of this diagnosis.","My inference: the framework implies a portfolio rule for choosing experts — adding a model that wins a niche subject but is weak overall should help more than adding a balanced all-rounder, because routing gains come from the spread of complementary strengths; the paper's Appendix D dominance caveat is the boundary case where that spread vanishes.","My inference: a stress test worth running is finer-grained subject splits — training the subject-level router on narrow subjects (e.g., separate science subfields rather than one 'science' label) should interpolate between subject-level and query-level performance on out-of-distribution data and reveal how much of the generalization comes from label coarseness itself."],"forward_implications":["A router trained only on public benchmark data (queries plus leaderboard subject accuracy) can beat the best constituent expert in aggregate, so assembling an expert panel does not require running every expert on every training query.","Subject-level routing transfers across benchmarks: a router trained on MMLU-Pro lifts aggregate accuracy on Big-Bench-Hard, and one trained on MMMU lifts accuracy on MMStar, so per-subject expert strengths appear stable across distributions.","Only the selected expert executes at inference, so serving cost stays at the level of a single model, and the routed panel of small models (at most 9B parameters) can outperform much larger single models on the same benchmark.","Query-level routing is preferable when training and test data share a distribution; subject-level routing is preferable when the test distribution is shifted or unknown.","Keeping the system current with new models requires only updating the per-subject leaderboard champion rather than relabeling the routing dataset from scratch."],"supporting_citations":[{"why":"supplies MMLU-Pro, the language benchmark whose subject leaderboard scores become the subject-level routing labels and whose queries train the router.","marker":"[30]"},{"why":"supplies MMMU, the multimodal benchmark used the same way for the multimodal subject-level router.","marker":"[34]"},{"why":"RouteLLM, the preference-data query-level routing method that Query-Level Bench-CoE abstracts and contrasts as the expensive, data-hungry baseline.","marker":"[20]"},{"why":"ZOOTER, the reward-guided query-level routing method that Query-Level Bench-CoE likewise abstracts as a baseline requiring per-query expert testing.","marker":"[17]"},{"why":"BERT, the text classifier architecture used as the router for language inputs.","marker":"[8]"},{"why":"Big-Bench-Hard, the out-of-distribution language target where subject-level routing reports its largest gain (69.91% versus 66.35%).","marker":"[27]"},{"why":"MMStar, the out-of-distribution multimodal target where subject-level routing beats the best single expert and query-level routing does not.","marker":"[4]"}],"fun_headline_variants":["Benchmark scores alone can route tasks to beat any single expert","Benchmark leaderboards can train a router to beat every expert","A router trained on benchmark scores beats 70B models","No extra inference: benchmark labels pick the right expert","Benchmark scores as routing labels beat single experts for free"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The expert that tops a subject on the training benchmark is assumed to remain the best expert for that subject on the test benchmark and at deployment: the per-subject leaderboard ranking must transfer across data distributions, or the subject-level router routes to a champion that no longer is one.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark scores alone can route tasks to beat any single expert","Benchmark leaderboards can train a router to beat every expert","A router trained on benchmark scores beats 70B models","No extra inference: benchmark labels pick the right expert","Benchmark scores as routing labels beat single experts for free"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000768,"raw_usage":{"total_tokens":3436,"prompt_tokens":1012,"completion_tokens":2424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2342}},"tokens_in":628,"tokens_out":2424,"duration_ms":17184,"temperature":1.0,"reasoning_tokens":2342,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:40:58.722035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the paper's out-of-distribution pair (router trained on MMLU-Pro, tested on Big-Bench-Hard), and compute the per-subject oracle there: route every query in each Big-Bench-Hard subject to the expert that actually scores highest on that subject, and compare that oracle accuracy with the subject-level router's accuracy and with the best single expert. If the router trails the oracle by a wide margin and no longer beats the best single expert, the claimed transfer of leaderboard rankings across datasets fails; if it matches the oracle, subject transfer is confirmed.","supporting_citations":[{"cited_title":"Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi","cited_arxiv_id":null,"evidence_quote":"supplies MMMU, the multimodal benchmark used the same way for the multimodal subject-level router."},{"cited_title":"Routing to the expert: Efficient reward-guided ensemble of large language models","cited_arxiv_id":null,"evidence_quote":"ZOOTER, the reward-guided query-level routing method that Query-Level Bench-CoE likewise abstracts as a baseline requiring per-query expert testing."},{"cited_title":"BERT: Pre-training of deep bidirectional trans- formers for language understanding","cited_arxiv_id":null,"evidence_quote":"BERT, the text classifier architecture used as the router for language inputs."},{"cited_title":"Le, Ed H","cited_arxiv_id":null,"evidence_quote":"Big-Bench-Hard, the out-of-distribution language target where subject-level routing reports its largest gain (69.91% versus 66.35%)."}],"review_version":1}