{"id":"58d293a6-1780-4b1b-a0f6-299c0cfad2d8","arxiv_id":"2412.01868","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A router that first classifies a prompt into a category and then picks a single expert LLM for that category lets a composition of open-weight models reach Arena-Hard 59.4 with roughly 31 billion active parameters.","lead":"This paper presents a system that automatically routes each user prompt to one of several smaller open-source language models, using a two-step router that first classifies the topic and then picks a model for that topic. The reported result is benchmark performance close to much larger proprietary models while only running a fraction of the total parameters per query.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Arena-Hard headline may be inflated by train/eval leakage: router training data includes LMSYS Chatbot Arena prompts (lin Chiang et al., 2024), the same source as Arena-Hard, with no reported decontamination.","rationale":"I read the paper as making an empirical claim: a two-step router composed of a category classifier and a MILP-optimized category-to-expert mapping lets a pool of five open-weight LLMs outperform each individual expert on Arena-Hard and MT-Bench at lower average active parameter counts. The most load-bearing condition for this claim is that the benchmark numbers are clean, i.e., that the evaluation prompts were not used to fit the router or the category-to-expert mapping. The paper's data pipeline makes that condition insecure: the general chat portion of the router training data comes from the LMSYS Chatbot Arena training split and internal user trials, and Arena-Hard is also constructed from Chatbot Arena prompts. The paper reports no decontamination for this portion, only for the domain-specific seed datasets. Because the category-to-expert mapping is optimized on the training data, any overlap with Arena-Hard would directly inflate the headline score. That is a sharper threat to the central claim than the one-expert-per-category constraint, which is a valid concern about optimality but would not by itself overturn the headline if the measured score is trustworthy. I therefore disagree with the reader's choice of weakest assumption. I still keep the verdict at CONDITIONAL: the contamination is testable, the rest of the paper (the MILP reformulation, the modular system design, the SambaNova implementation) is not affected by this specific score, and I do not see evidence of deliberate manipulation. If the overlap test found substantial leakage, the Arena-Hard claim would need to be withdrawn or recomputed on a disjoint subset. A secondary issue is that the reported 'average active parameters' excludes the 7B e5-mistral embedding router, so the true per-query compute is at least 7B parameters higher than advertised; this does not break the qualitative claim (even 38B is below the 72B largest expert) but should be clarified.","tokens_in":29,"tokens_out":9382,"duration_ms":145116,"concrete_test":"Compute exact-match and near-duplicate overlap between the assembled router training set (Section 8.1, including the LMSYS 'training split' and internal trial prompts) and the Arena-Hard test prompts using normalized text matching plus embedding cosine similarity with intfloat/e5-mistral-7b-instruct at a threshold of 0.9. Remove any overlapping prompts from the training set, retrain or re-solve the category-to-expert mapping, and re-run the Arena-Hard evaluation for the CoE and Robust-CoE configurations; report the resulting scores and active-parameter averages. If the overlap is zero, the headline is not invalidated by this mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 8.1 extends the CoE router training data by applying the semi-supervised pipeline to 'general chat prompts from internal user trials and training split (lin Chiang et al., 2024)' (the LMSYS Chatbot Arena dataset). Arena-Hard (Li et al., 2024) is explicitly built from Chatbot Arena prompts. The category-to-expert mapping (Eqs. 6-7) is trained by optimizing per-category win-rates against GPT-4o on this data; if Arena-Hard test prompts or near-duplicates are present in the LMSYS 'training split' or the internal trial data, the mapping is directly fit to the evaluation set, and the reported 59.4 score is not a valid out-of-distribution measurement. The paper only notes that the domain seed datasets were restricted to training splits; it does not report any decontamination of the general chat prompts. This is the most load-bearing concern because the central claim is an empirical benchmark headline, and the data provenance creates a concrete, testable leakage path that the paper does not close.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Composition of Experts (CoE), a compound AI system that routes each prompt through a two-step mechanism: a category router (a k-NN classifier on embeddings) followed by a category-to-expert mapping obtained by solving a mixed-integer linear program (MILP) under a parameter budget. A Robust-CoE variant uses entropy-based uncertainty to divert uncertain prompts to a 'general' category. The authors report Arena-Hard, MT-Bench, and knowledge-benchmark results showing that CoE configurations with fewer average active parameters can match or exceed individual open-weight experts, and describe a serving implementation on SambaNova SN40L hardware.","tokens_in":16531,"tokens_out":4217,"duration_ms":31776,"significance":"The MILP formulation in Section 5.2 is clean and the modular two-step training pipeline is a practical contribution that could lower the cost of composing existing open-weight LLMs. The claim that a router-based mix of open-weight models can beat each individual expert while using fewer active parameters is plausible and, if correctly evaluated, would be useful to the community. However, the empirical validation currently has a potentially load-bearing data-leakage issue and lacks statistical rigor, so the quantitative headline should not be accepted at face value.","major_comments":[{"comment":"The router training data includes 'general chat prompts from internal user trials and training split (lin Chiang et al., 2024)'—the LMSYS Chatbot Arena dataset—while Section 9.1 evaluates on Arena-Hard, which is constructed from Chatbot Arena prompts (Li et al., 2024). The paper does not report any decontamination or overlap analysis between the router training set and the Arena-Hard test prompts. Because the category-to-expert mapping is optimized on these data (Section 8.3), the headline Arena-Hard scores in Figure 13 may reflect direct or near-duplicate leakage rather than generalization. Please report exact overlap statistics (e.g., n-gram or embedding similarity) and re-run the evaluation on a decontaminated subset, or retrain the router without the LMSYS data.","section":"Section 8.1"},{"comment":"All benchmark numbers in Section 9 and Table 2 appear to be single runs without error bars, confidence intervals, or significance tests. Given the stochasticity of LLM generation and judging, differences such as CoE 190B versus Robust-CoE 190B in Figure 13 (60.2 vs. 62.1) may be within noise. Please provide multiple seeds or bootstrap confidence intervals for the headline numbers.","section":"Section 9"},{"comment":"The abstract claims an Arena-Hard score of 59.4 with 31 billion average active parameters, but Section 9.1 does not explicitly identify which budget B and which variant (CoE or Robust-CoE) produces this point. To make the central claim verifiable, please add a table or explicit annotation mapping the abstract numbers to the exact configuration, including B, the chosen expert set, and the entropy threshold if Robust-CoE is used.","section":"Abstract / Section 9.1"},{"comment":"The two-step routing imposes a hard constraint C1K = 1M in (5), assigning one fixed expert to each category, yet the paper does not quantify the accuracy loss of this constraint relative to per-prompt best-expert routing. Figure 3 shows jittery per-prompt labels and Figure 4 shows category clusters, but no held-out comparison of the two routing schemes is provided. Given that the knowledge-intensive benchmarks in Section 9.3 show performance drops for CoE, this gap is material to the claim that category-level assignment recovers most of the per-prompt routing benefit. Please measure the oracle gap (per-prompt best expert vs. category-best expert) on the router test split.","section":"Section 4.1"}],"minor_comments":[{"comment":"The word 'Furthurmore' should be 'Furthermore'.","section":"Section 4"},{"comment":"The y-axis label reads 'Area-Hard'; this should be 'Arena-Hard'.","section":"Figure 13"},{"comment":"The phrase 'curated about58000' is missing a space; please correct to 'about 58,000'.","section":"Section 8.1"},{"comment":"The reference 'Benchmarking samba-1' lacks author names and a year; please provide a complete citation.","section":"References"},{"comment":"Please clarify how the average number of active parameters is computed for multi-turn MT-Bench conversations when different experts are selected at different turns.","section":"Section 9.2"},{"comment":"The judge model is GPT-4o-mini, while the desired completions are from GPT-4o-2024-02-15-preview; please state explicitly why this choice of judge and reference model does not bias the win-rate estimates.","section":"Section 8.3"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on self-cited work for key components (Raju et al., 2024 for the judge template; Prabhakar et al., 2024 for the hardware). While this is not a flaw by itself, the lack of an ablation testing an alternative judge prompt or an independent evaluation protocol weakens the evidence. In addition, no code or data is released, which hampers reproducibility of the decontamination checks we request. The fit with the journal's scope is acceptable, but the empirical claims need to be substantiated before I can recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the two-step category-to-expert formulation: first classify the prompt into a category, then solve a parameter-budgeted MILP to assign one expert per category. That is clean, modular, and genuinely new relative to Shnitzer, Lu, and LLM-Blender. The Robust-CoE entropy fallback is also a sensible addition. The SN40L tiered-memory discussion is system-level useful, though it stays at the level of architecture reasoning rather than measurement.\n\nThe soft spots are real and cluster around the empirical section. The most serious is the leakage path the stress-test note flags: the router training data extends onto general chat prompts from the LMSYS Chatbot Arena training split, and Arena-Hard is built from Chatbot Arena prompts. The paper reports no decontamination for that extension. That is a concrete, testable route to inflating the headline 59.4, and the authors only say the domain seed datasets were restricted to training splits. The stress-test note holds up; they need to close this.\n\nSecond, I could not map the abstract's headline 59.4 with 31B active parameters to any specific B in Section 9. The reported B=190B runs give 60.2 and 62.10 with 65B and 69B active. Maybe the 59.4 comes from a different budget, but the paper does not say, and that ambiguity is too easy to avoid.\n\nThird, no error bars, no significance tests, single runs everywhere. For a benchmark headline that is the whole claim, that is thin.\n\nFourth, the category-sufficiency assumption is load-bearing and unmeasured. Figure 3 shows per-prompt best-expert labels are jittery, but they never quantify the accuracy loss from forcing one expert per category, nor whether the clusters in Figure 4 transfer out-of-distribution to benchmark prompts. That is the weakest theoretical joint.\n\nTo give credit where it is due: the MILP derivation is internally consistent and reproducible from the text, the data pipeline is clearly described, and using GPT-4o-mini win-rates to fit the mapping while evaluating on Arena-Hard is not circular by itself. Self-citation to Raju et al. for the judge template is normal practice, not a flaw.\n\nThis paper deserves a serious referee. The engineering contribution is solid, and the empirical claims are testable and fixable. I would send it to review with a request for decontamination analysis, error bars, and a clear mapping of abstract numbers to configurations. If those come back clean, I'd cite it for the MILP routing formulation.","headline":"A genuinely useful two-step routing formulation with a clean MILP core, but the headline benchmark numbers need decontamination checks and error bars before I'd trust them.","tokens_in":17100,"tokens_out":2060,"would_cite":true,"duration_ms":21695,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T50","90C11"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that a two-step router, which classifies each prompt into a category and then maps the category to one of five open-weight expert LLMs, produces a system that outscores each individual expert on Arena-Hard and MT-Bench…","keywords":["composition of experts","LLM routing","mixture of experts","modular AI systems","mixed-integer linear programming","uncertainty quantification","LLM-as-a-judge","parameter budget"],"falsifier":"Take a large set of prompts all labeled as one category (e.g., coding) and ask two experts with different strengths within that category; if the per-prompt best-expert labels are highly split and a per-prompt router clearly beats the single-expert-per-category CoE on that category, while the category router cannot distinguish the sub-cases, the core simplification would fail. Concretely, measure the win rate of the CoE's chosen expert versus the oracle best expert on each category in a held-out set; a large gap would refute the sufficiency of one expert per category.","tokens_in":16119,"feed_emoji":"🧩","tokens_out":5556,"duration_ms":42274,"temperature":0.7,"pith_summary":"The paper proposes Composition of Experts (CoE), a system that combines several existing open-weight LLMs into one modular whole. A router first classifies each input prompt into a category and then maps that category to the best available expert, so each prompt is answered by only one model. The authors show that such a system can outperform each of its component experts on aggregate while using fewer average active parameters than the largest expert. This matters because it offers a cheap, modular path to improving LLM performance without pretraining or fine-tuning a monolithic model.","feed_headline":"A router mixes five open LLMs and beats each one alone","feed_subtitle":"Composition of Experts scores 59.4 on Arena-Hard and 9.06 on MT-Bench using far fewer active parameters than the largest model.","key_machinery":"The load-bearing object is the two-step routing function R(p) = CE(CR(p)), where CR is a category router and CE is a category-to-expert mapping represented as a binary matrix C ∈ {0,1}^{M×K} with exactly one expert per category, enforced by C1_K = 1_M. The training problem reduces to a mixed-integer linear program over C and an auxiliary binary vector y that encodes which experts are used, with the budget constraint s_E^T y ≤ B imposed through y ≤ C^T 1_M ≤ M y. This reformulation makes the system modular: adding or removing an expert or category changes only the costs and constraints, not the architecture.","core_discovery":"The central discovery is that per-prompt best-expert routing is too jittery to learn directly, but routing through categories restores learnable structure. The paper demonstrates that a two-step router, a k-NN category classifier on text embeddings followed by a category-to-expert mapping, yields a CoE that scores 59.4 on Arena-Hard with 31 billion average active parameters and 9.06 on MT-Bench with 54 billion, drawing on five open-weight models ranging from 7B to 72B. The category-to-expert mapping is trained by solving a mixed-integer linear program that minimizes total judged loss subject to a cumulative parameter budget. The authors also introduce Robust-CoE, which uses classifier entropy to send uncertain prompts to a general category, and show it maintains performance under distribution shift.","pith_inferences":["If the categories are truly the right latent variable, then the router's accuracy on categories becomes the main bottleneck; one can test this by comparing CoE to an oracle that knows the best expert per category and measuring the degradation.","The category-to-expert MILP assumes each category is served by exactly one expert; for heterogeneous categories, allowing a small mixture of experts per category might close much of the gap to per-prompt routing at modest extra parameter cost.","The same two-step formulation could be applied beyond LLMs to any collection of trained models with complementary strengths, as long as inputs can be classified into stable categories.","The paper's reliance on an LLM judge for quality labels means the routing decisions inherit the judge's biases; an extension would be to cross-check routing decisions with human preference data on a held-out set."],"forward_implications":["CoE provides a practical way to assemble existing open-weight models into a system that beats each individual expert on aggregate benchmarks while activating fewer parameters than the largest member.","Because the router is a simple classifier on embeddings, adding new capabilities reduces to adding a new expert module and re-solving the MILP, avoiding costly fine-tuning.","Robust-CoE's uncertainty-based fallback to a general category recovers performance on knowledge-intensive benchmarks where the router training data has distribution mismatch.","The system's per-query compute cost equals that of querying a single expert plus the router, so response throughput can stay constant as experts are added.","CoE's modularity enables interpretability: failures can be traced to either router misclassification or a specific expert module."],"supporting_citations":[{"why":"Supplies the semi-supervised prompt curation pipeline and the LLM-as-a-judge template used to compute expert win-rates and training costs.","marker":"(Raju et al., 2024)"},{"why":"Provides the MT-Bench benchmark used for multi-turn evaluation of CoE and individual experts.","marker":"(Zheng et al., 2023)"},{"why":"Provides the Arena-Hard benchmark used for single-turn evaluation of CoE and individual experts.","marker":"(Li et al., 2024)"},{"why":"Provides MMLU-Pro knowledge benchmarks used for fine-grained domain evaluation, where Robust-CoE's advantage is shown.","marker":"(Wang et al., 2024)"},{"why":"Describes the SambaNova SN40L architecture whose tiered memory enables efficient CoE serving with fast expert swapping.","marker":"(Prabhakar et al., 2024)"},{"why":"Source of crowd-sourced Chatbot Arena prompts used for benchmark construction and as general chat seed data for the router training pipeline.","marker":"(Chiang et al., 2024)"}],"fun_headline_variants":["Two-step router beats larger LLMs with fewer active parameters","Category-first routing restores learnability in expert LLM ensembles","Five open LLMs, one router, and a 59.4 Arena-Hard score","A modular compound AI system that cuts compute and boosts accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system assumes that assigning each prompt category a single fixed expert captures most of the benefit of choosing the best expert for every individual prompt, so that per-prompt variation within a category does not matter much.","fun_headline_variants_meta":{"raw":{"variants":["Two-step router beats larger LLMs with fewer active parameters","Category-first routing restores learnability in expert LLM ensembles","Five open LLMs, one router, and a 59.4 Arena-Hard score","A modular compound AI system that cuts compute and boosts accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000413,"raw_usage":{"total_tokens":2169,"prompt_tokens":1010,"completion_tokens":1159,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":1084}},"tokens_in":626,"tokens_out":1159,"duration_ms":10615,"temperature":1.0,"reasoning_tokens":1084,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:34:45.825902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a large set of prompts all labeled as one category (e.g., coding) and ask two experts with different strengths within that category; if the per-prompt best-expert labels are highly split and a per-prompt router clearly beats the single-expert-per-category CoE on that category, while the category router cannot distinguish the sub-cases, the core simplification would fail. Concretely, measure the win rate of the CoE's chosen expert versus the oracle best expert on each category in a held-out set; a large gap would refute the sufficiency of one expert per category.","supporting_citations":[{"cited_title":"Mmlu-pro: A more robust and challenging multi- task language understanding benchmark (published at neurips 2024 track datasets and benchmarks),","cited_arxiv_id":null,"evidence_quote":"Provides MMLU-Pro knowledge benchmarks used for fine-grained domain evaluation, where Robust-CoE's advantage is shown."},{"cited_title":"Sambanova sn40l rdu: Breaking the barrier of trillion+ parameter scale gen ai computing","cited_arxiv_id":null,"evidence_quote":"Describes the SambaNova SN40L architecture whose tiered memory enables efficient CoE serving with fast expert swapping."}],"review_version":1}