Composition of Experts: A Modular Compound AI System Leveraging Large Language Models

Anand Sampat; Bo Li; Guoyao Feng; Jonathan Li; Kaizhao Liang; Raghu Prabhakar; Ravi Raju; Sumati Jairath; Swayambhoo Jain; Urmish Thakkar

REVIEW 4 major objections 6 minor 2 cited by

Composition of Experts: A Modular Compound AI System Leveraging Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that a two-step router, which classifies each prompt into a category and then maps the category to one of five open-weight expert LLMs, produces a system that outscores each individual expert on Arena-Hard and MT-Bench…

desk verdict A genuinely useful two-step routing formulation with a clean MILP core, but the headline benchmark numbers need decontamination checks and error bars before I'd trust them. read the letter →

arxiv 2412.01868 v1 pith:PUSGLOOW submitted 2024-12-02 cs.LG cs.AIcs.CLstat.ML

Swayambhoo Jain , Ravi Raju , Bo Li , Zoltan Csaki , Jonathan Li , Kaizhao Liang , Guoyao Feng , Urmish Thakkar

show 3 more authors

Anand Sampat Raghu Prabhakar Sumati Jairath

This is my paper · ORCID

classification cs.LGcs.AIcs.CLstat.ML MSC 68T0768T5090C11

keywords compositionofexpertsLLMroutingmixturemodularAIsystemsmixed-integerlinearprogramminguncertaintyquantificationLLM-as-a-judgeparameterbudget

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Composition of Experts (CoE), a system that combines several existing open-weight LLMs into one modular whole. A router first classifies each input prompt into a category and then maps that category to the best available expert, so each prompt is answered by only one model. The authors show that such a system can outperform each of its component experts on aggregate while using fewer average active parameters than the largest expert. This matters because it offers a cheap, modular path to improving LLM performance without pretraining or fine-tuning a monolithic model.

What carries the argument

The load-bearing object is the two-step routing function R(p) = CE(CR(p)), where CR is a category router and CE is a category-to-expert mapping represented as a binary matrix C ∈ {0,1}^{M×K} with exactly one expert per category, enforced by C1_K = 1_M. The training problem reduces to a mixed-integer linear program over C and an auxiliary binary vector y that encodes which experts are used, with the budget constraint s_E^T y ≤ B imposed through y ≤ C^T 1_M ≤ M y. This reformulation makes the system modular: adding or removing an expert or category changes only the costs and constraints, not the architecture.

What would settle it

Take a large set of prompts all labeled as one category (e.g., coding) and ask two experts with different strengths within that category; if the per-prompt best-expert labels are highly split and a per-prompt router clearly beats the single-expert-per-category CoE on that category, while the category router cannot distinguish the sub-cases, the core simplification would fail. Concretely, measure the win rate of the CoE's chosen expert versus the oracle best expert on each category in a held-out set; a large gap would refute the sufficiency of one expert per category.

Watch

Extended reading notes

Core claim

The central discovery is that per-prompt best-expert routing is too jittery to learn directly, but routing through categories restores learnable structure. The paper demonstrates that a two-step router, a k-NN category classifier on text embeddings followed by a category-to-expert mapping, yields a CoE that scores 59.4 on Arena-Hard with 31 billion average active parameters and 9.06 on MT-Bench with 54 billion, drawing on five open-weight models ranging from 7B to 72B. The category-to-expert mapping is trained by solving a mixed-integer linear program that minimizes total judged loss subject to a cumulative parameter budget. The authors also introduce Robust-CoE, which uses classifier entropy to send uncertain prompts to a general category, and show it maintains performance under distribution shift.

Load-bearing premise

The system assumes that assigning each prompt category a single fixed expert captures most of the benefit of choosing the best expert for every individual prompt, so that per-prompt variation within a category does not matter much.

Editorial extensions

If this is right

CoE provides a practical way to assemble existing open-weight models into a system that beats each individual expert on aggregate benchmarks while activating fewer parameters than the largest member.
Because the router is a simple classifier on embeddings, adding new capabilities reduces to adding a new expert module and re-solving the MILP, avoiding costly fine-tuning.
Robust-CoE's uncertainty-based fallback to a general category recovers performance on knowledge-intensive benchmarks where the router training data has distribution mismatch.
The system's per-query compute cost equals that of querying a single expert plus the router, so response throughput can stay constant as experts are added.
CoE's modularity enables interpretability: failures can be traced to either router misclassification or a specific expert module.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the categories are truly the right latent variable, then the router's accuracy on categories becomes the main bottleneck; one can test this by comparing CoE to an oracle that knows the best expert per category and measuring the degradation.
The category-to-expert MILP assumes each category is served by exactly one expert; for heterogeneous categories, allowing a small mixture of experts per category might close much of the gap to per-prompt routing at modest extra parameter cost.
The same two-step formulation could be applied beyond LLMs to any collection of trained models with complementary strengths, as long as inputs can be classified into stable categories.
The paper's reliance on an LLM judge for quality labels means the routing decisions inherit the judge's biases; an extension would be to cross-check routing decisions with human preference data on a held-out set.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real contribution here is the two-step category-to-expert formulation: first classify the prompt into a category, then solve a parameter-budgeted MILP to assign one expert per category. That is clean, modular, and genuinely new relative to Shnitzer, Lu, and LLM-Blender. The Robust-CoE entropy fallback is also a sensible addition. The SN40L tiered-memory discussion is system-level useful, though it stays at the level of architecture reasoning rather than measurement.

The soft spots are real and cluster around the empirical section. The most serious is the leakage path the stress-test note flags: the router training data extends onto general chat prompts from the LMSYS Chatbot Arena training split, and Arena-Hard is built from Chatbot Arena prompts. The paper reports no decontamination for that extension. That is a concrete, testable route to inflating the headline 59.4, and the authors only say the domain seed datasets were restricted to training splits. The stress-test note holds up; they need to close this.

Second, I could not map the abstract's headline 59.4 with 31B active parameters to any specific B in Section 9. The reported B=190B runs give 60.2 and 62.10 with 65B and 69B active. Maybe the 59.4 comes from a different budget, but the paper does not say, and that ambiguity is too easy to avoid.

Third, no error bars, no significance tests, single runs everywhere. For a benchmark headline that is the whole claim, that is thin.

Fourth, the category-sufficiency assumption is load-bearing and unmeasured. Figure 3 shows per-prompt best-expert labels are jittery, but they never quantify the accuracy loss from forcing one expert per category, nor whether the clusters in Figure 4 transfer out-of-distribution to benchmark prompts. That is the weakest theoretical joint.

To give credit where it is due: the MILP derivation is internally consistent and reproducible from the text, the data pipeline is clearly described, and using GPT-4o-mini win-rates to fit the mapping while evaluating on Arena-Hard is not circular by itself. Self-citation to Raju et al. for the judge template is normal practice, not a flaw.

This paper deserves a serious referee. The engineering contribution is solid, and the empirical claims are testable and fixable. I would send it to review with a request for decontamination analysis, error bars, and a clear mapping of abstract numbers to configurations. If those come back clean, I'd cite it for the MILP routing formulation.

Referee Report

4 major / 6 minor

Summary. The paper introduces Composition of Experts (CoE), a compound AI system that routes each prompt through a two-step mechanism: a category router (a k-NN classifier on embeddings) followed by a category-to-expert mapping obtained by solving a mixed-integer linear program (MILP) under a parameter budget. A Robust-CoE variant uses entropy-based uncertainty to divert uncertain prompts to a 'general' category. The authors report Arena-Hard, MT-Bench, and knowledge-benchmark results showing that CoE configurations with fewer average active parameters can match or exceed individual open-weight experts, and describe a serving implementation on SambaNova SN40L hardware.

Significance. The MILP formulation in Section 5.2 is clean and the modular two-step training pipeline is a practical contribution that could lower the cost of composing existing open-weight LLMs. The claim that a router-based mix of open-weight models can beat each individual expert while using fewer active parameters is plausible and, if correctly evaluated, would be useful to the community. However, the empirical validation currently has a potentially load-bearing data-leakage issue and lacks statistical rigor, so the quantitative headline should not be accepted at face value.

major comments (4)

[Section 8.1] The router training data includes 'general chat prompts from internal user trials and training split (lin Chiang et al., 2024)'—the LMSYS Chatbot Arena dataset—while Section 9.1 evaluates on Arena-Hard, which is constructed from Chatbot Arena prompts (Li et al., 2024). The paper does not report any decontamination or overlap analysis between the router training set and the Arena-Hard test prompts. Because the category-to-expert mapping is optimized on these data (Section 8.3), the headline Arena-Hard scores in Figure 13 may reflect direct or near-duplicate leakage rather than generalization. Please report exact overlap statistics (e.g., n-gram or embedding similarity) and re-run the evaluation on a decontaminated subset, or retrain the router without the LMSYS data.
[Section 9] All benchmark numbers in Section 9 and Table 2 appear to be single runs without error bars, confidence intervals, or significance tests. Given the stochasticity of LLM generation and judging, differences such as CoE 190B versus Robust-CoE 190B in Figure 13 (60.2 vs. 62.1) may be within noise. Please provide multiple seeds or bootstrap confidence intervals for the headline numbers.
[Abstract / Section 9.1] The abstract claims an Arena-Hard score of 59.4 with 31 billion average active parameters, but Section 9.1 does not explicitly identify which budget B and which variant (CoE or Robust-CoE) produces this point. To make the central claim verifiable, please add a table or explicit annotation mapping the abstract numbers to the exact configuration, including B, the chosen expert set, and the entropy threshold if Robust-CoE is used.
[Section 4.1] The two-step routing imposes a hard constraint C1K = 1M in (5), assigning one fixed expert to each category, yet the paper does not quantify the accuracy loss of this constraint relative to per-prompt best-expert routing. Figure 3 shows jittery per-prompt labels and Figure 4 shows category clusters, but no held-out comparison of the two routing schemes is provided. Given that the knowledge-intensive benchmarks in Section 9.3 show performance drops for CoE, this gap is material to the claim that category-level assignment recovers most of the per-prompt routing benefit. Please measure the oracle gap (per-prompt best expert vs. category-best expert) on the router test split.

minor comments (6)

[Section 4] The word 'Furthurmore' should be 'Furthermore'.
[Figure 13] The y-axis label reads 'Area-Hard'; this should be 'Arena-Hard'.
[Section 8.1] The phrase 'curated about58000' is missing a space; please correct to 'about 58,000'.
[References] The reference 'Benchmarking samba-1' lacks author names and a year; please provide a complete citation.
[Section 9.2] Please clarify how the average number of active parameters is computed for multi-turn MT-Bench conversations when different experts are selected at different turns.
[Section 8.3] The judge model is GPT-4o-mini, while the desired completions are from GPT-4o-2024-02-15-preview; please state explicitly why this choice of judge and reference model does not bias the win-rate estimates.

Circularity Check

2 steps flagged · score 3.0 of 10

No by-construction circularity: headline scores use Arena-Hard/MT-Bench's own GPT-4 judges, not the GPT-4o-mini judge that fits the expert mapping. Flagged: a load-bearing same-author citation (Raju et al., 2024) for the judge template and data pipeline, plus an unresolved overlap between the LMSYS router-training pool and Arena-Hard's Chatbot-Arena source.

self citation load bearing [Section 7 (Semi-Supervised Training Data Pipeline for CoE) and Section 8.3 (Category-to-expert mapping training)]
"Recent work (Raju et al., 2024) proposed a semi-supervised approach to build benchmarks with high degree of separability in the LLM performance in a cost effective manner. We leverage similar semi-supervised approach and use a prompt curation pipeline shown in Figure 6. ... We use the judge prompt template in (Raju et al., 2024) given its emphasis on effectively addresses the nuances associated with using LLM as a judge for variety of domains."

All five authors of Raju et al. (arXiv:2408.08808) are co-authors of the present paper, so the authority cited for the two load-bearing machinery pieces is the authors' own prior work. The category-to-expert mapping (Eqs. 6-7), which fully determines which expert answers any prompt, is computed from LLM-as-a-judge win-rates obtained with the self-cited template, and the router training corpus (about 58,000 prompts) is produced with the self-cited semi-supervised pipeline. The central empirical claim does not reduce to this citation, however: the Arena-Hard and MT-Bench scores are fresh measurements under those benchmarks' own GPT-4-based protocols, so the citation is load-bearing for the mechanism but not for the reported numbers.
fitted input called prediction [Section 8.1 (CoE Training data) with Section 9 (Experimental Results, Arena-Hard)]
"We restrict only to the training splits to avoid contamination with benchmarks. ... Equipped with these high quality category labeled seed prompts extend prompt dataset by following the semi-supervised pipeline in Figure 6 on general chat prompts from internal user trials and training split (lin Chiang et al., 2024). ... These benchmarks contain challenging prompts collected in a crowd-sourced manner via Chatbot-Arena and represent a real-world usage of LLMs (Chiang et al., 2024)."

The category-to-expert mapping is fitted to win-rates over prompts that include the LMSYS Chatbot-Arena Kaggle training split (lin Chiang et al., 2024), while Arena-Hard (Li et al., 2024) is built from the same Chatbot-Arena crowd-sourced pool. The paper restricts its HuggingFace seed datasets to training splits to avoid benchmark contamination but reports no decontamination for the LMSYS-derived general chat prompts in the same training pool. If Arena-Hard test prompts or near-duplicates lie in that pool, the headline 59.4 is a re-measurement of the fitted win-rate objective, not an out-of-distribution prediction.

full rationale

Walking the derivation chain: (1) Training data (Section 8.1) is a mix of HuggingFace domain seeds and LMSYS-extended general chat prompts; (2) the category router is a k-NN classifier trained with 98% test accuracy (Section 8.2); (3) the category-to-expert mapping is the MILP solution of Eqs. 4-7 whose costs lij are GPT-4o-mini-judged win-rates against GPT-4o completions (Section 8.3); (4) headline numbers are computed on Arena-Hard, MT-Bench and MMLU-Pro (Section 9). No step is self-definitional: the MILP reformulation of the budgeted assignment problem is a faithful equivalence, not a hidden identity, and the Figure 11 evaluation on the held-out test split uses the same win-rate metric as the objective only on data not used in fitting, which is standard practice. The headline 59.4 and 9.06 are not the fitted objective by construction, because Arena-Hard's and MT-Bench's judges are the benchmarks' own GPT-4-based protocols, not the GPT-4o-mini judge used to fit lij, and the paper even reports degraded CoE performance on knowledge-intensive benchmarks (Section 9.3), evidence that the fit does not force the reported results. The two circular-adjacent items that keep the score above 1 are the same-author citation (Raju et al., 2024) standing behind the judge template and the semi-supervised data pipeline, and the unresolved LMSYS/Arena-Hard pool overlap flagged above. Neither reduces the central claim to its inputs by construction, so the appropriate finding is a mild score rather than a 6+ forced-derivation verdict.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; CoE and Robust-CoE are system architectures, and the MILP formulation uses existing variables (C, y) with no new theoretical objects. The free parameters are all hyperparameters of the training or data pipeline.

free parameters (4)

k-NN neighborhood size k = 30
Selected on validation split in Section 8.2; router accuracy is measured for k=10,...,40.
Robust-CoE entropy threshold epsilon_t = 0.1
Chosen in Section 8.2 as the value among 0.05 to 1.5 that minimally reduces the category-router recall.
Data curation entropy filter threshold = 0.5
Set in Section 8.1 to filter low-confidence k-NN labels during semi-supervised expansion.
Category set size M = 13 visible categories (plus 'general' for Robust-CoE)
Defined in Section 8 as five domains and eight languages; the choice of categories is a design decision, not tuned from data.

assumptions (6)

domain assumption Expert capabilities can be predicted from prompt category
Section 4.1 relies on t-SNE clusters (Figures 3-4) to argue that category labels separate prompts better than per-prompt best-expert labels.
domain assumption One fixed expert per category is sufficient for near-optimal routing
The MILP in Section 5.2 imposes C1K=1M, assigning a single expert to each category; the paper does not quantify the loss versus per-prompt routing.
domain assumption LLM-as-a-judge with GPT-4o-mini gives reliable win-rate estimates
Section 8.3 uses GPT-4o-mini judgments to compute the costs l_ij; if the judge is biased, the expert mapping inherits the bias.
domain assumption A k-NN classifier over e5-mistral embeddings can act as an accurate category router
Section 8.2 assumes the embedding space is separable into the chosen categories; 98% test accuracy is reported on the same distribution, but no out-of-distribution guarantee is given.
domain assumption Semi-supervised data expansion yields a representative training distribution
Section 7 expands seed prompts using k-NN labels on internal trial data and a training split of a Kaggle dataset; the target benchmarks may have a different distribution.
domain assumption The loss function decomposes as a sum of per-category costs
The MILP objective sum_{i,j} l_ij c_ij assumes router misclassification errors are negligible or their effect is captured by the win-rates.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Composition of Experts: A Modular Compound AI System Leveraging Large Language Models." pith.science (2026). https://pith.science/paper/PUSGLOOW

@misc{pith2026241201868,
  author       = {Pith},
  title        = {Pith review of: Composition of Experts: A Modular Compound AI System Leveraging Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUSGLOOW}},
  note         = {Machine review of arXiv:2412.01868}
}

abstract

Large Language Models (LLMs) have achieved remarkable advancements, but their monolithic nature presents challenges in terms of scalability, cost, and customization. This paper introduces the Composition of Experts (CoE), a modular compound AI system leveraging multiple expert LLMs. CoE leverages a router to dynamically select the most appropriate expert for a given input, enabling efficient utilization of resources and improved performance. We formulate the general problem of training a CoE and discuss inherent complexities associated with it. We propose a two-step routing approach to address these complexities that first uses a router to classify the input into distinct categories followed by a category-to-expert mapping to obtain desired experts. CoE offers a flexible and cost-effective solution to build compound AI systems. Our empirical evaluation demonstrates the effectiveness of CoE in achieving superior performance with reduced computational overhead. Given that CoE comprises of many expert LLMs it has unique system requirements for cost-effective serving. We present an efficient implementation of CoE leveraging SambaNova SN40L RDUs unique three-tiered memory architecture. CoEs obtained using open weight LLMs Qwen/Qwen2-7B-Instruct, google/gemma-2-9b-it, google/gemma-2-27b-it, meta-llama/Llama-3.1-70B-Instruct and Qwen/Qwen2-72B-Instruct achieve a score of $59.4$ with merely $31$ billion average active parameters on Arena-Hard and a score of $9.06$ with $54$ billion average active parameters on MT-Bench.

Figures

Figures reproduced from arXiv: 2412.01868 by the authors.

**Figure 1.** Abstract representation of Composition of Experts (CoE) for a given set of expert LLMs E = {E1, · · · , EK}. For a given subset of experts Es ⊆ E, the CoE(p; Es, R) routes the input prompt p to one of the expert in Es using the routing function R(p) and produces the output by loading and the running that expert. a fine-grained routing by employing a gating function in each transformer module. In contrast, the propos… view at source ↗

**Figure 2.** Abstract representation of Composition of Experts (CoE) with two step routing. For a given subset of experts Es ⊆ E, the CoE(p; Es, CE, CR) routes the input prompt p by first mapping it to one of the M categories using category-router CR(p) followed by category-to-expert mapping CE (CR(p)) to choose a designated expert for that category [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** 2D t-SNE plot for prompt-embeddings obtained from text-embedding model intfloat/e5-mistral-7b-instruct for prompts in the CoE training data. Labels based on best expert LLMs chosen from the expert set comprising of Qwen/Qwen2-7BInstruct, google/gemma-2-9b-it, google/gemma-2-27b-it, metallama/Llama-3.1-70B-Instruct and Qwen/Qwen2-72B-Instruct. Best expert is obtained by using LLM-as-a-judge with details provided in… view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: 2D t-SNE plot for prompt-embeddings obtained from text-embedding model intfloat/e5-mistral-7b-instruct for prompts in the CoE training data. Labels are based on various categories comprising of variety of domains and languages. the category-to-expert mapping. More conc…

**Figure 5.** Figure 5: A simplified sequence of operations for CoE serving via SN40L. Router weights are in HBM. Expert weights are in DDR, with a region pre-allocated in HBM for the “current” expert(s). Equipped with above facts, the problem (4) for a given category router CRˆ can be re-wr…

**Figure 6.** Figure 6: Semi-supervised pipeline to curate high quality prompts for CoE training data for various categories. Starting with high-quality labeled small seed dataset we leverage text-embeddings to train a k-NN classifier. The k-NN classifier is used to further expand the trainin…

**Figure 8.** Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Confusion matrix on the test split for k-NN (k = 30) based the category router CR using intfloat/e5-mistral-7b-instruct as the text embedding model. sive Multitask Embedding Benchmark (Muennighoff et al., 2023) and train k-NN classifier on seed prompt set. We measure c…

**Figure 10.** Figure 10: Winrates of expert LLMs Qwen/Qwen2-7B-Instruct, google/gemma-2-9b-it, google/gemma-2-27b-it, meta-llama/Llama-3.1- 70B-Instruct and Qwen/Qwen2-72B-Instruct on different domains. The win-rates are computed using LLM-as-a-judge approach as described in Section 8.3. We c…

**Figure 11.** Figure 11: Win-rates against large monolithic model GPT-4o-2024- 02-15-preview on on the test split vs. B for experts LLMs, CoE and Robust-CoE. Various CoEs use k-NN (k = 30) based category router using intfloat/e5-mistral-7b-instruct as the text embedding model and category-to-…

**Figure 12.** Figure 12: Winrates of expert LLMs Qwen/Qwen2-7B-Instruct, google/gemma-2-9b-it, google/gemma-2-27b-it, meta-llama/Llama-3.1- 70B-Instruct and Qwen/Qwen2-72B-Instruct on different languages. The win-rates are computed using LLM-as-a-judge approach as described in Section 8.3. st…

**Figure 13.** Figure 13: shows performance on the Arena-Hard benchmark for CoE for various values of B. On the y-axis is the the Arena-Hard score and on the x-axis is the average number of active parameters for the benchmark. As total size B of CoE increases, CoE achieves better performance …

**Figure 15.** Figure 15: Average scores on knowledge intensive benchmarks vs. average number of active parameters in Billions for expert LLMs, CoEs and Robust-CoEs. Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and S…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Arch-Router: Aligning LLM Routing with Human Preferences
cs.CL 2025-06 conditional novelty 6.0 of 10

Arch-Router, a 1.5B fine-tuned generative model, matches chat queries to user-defined domain-action policies and reports higher accuracy than several proprietary models on adapted routing benchmarks.
CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering
cs.CL 2025-07 reject novelty 4.0 of 10

CoE-Ops routes DevOps questions to specialized LLM experts using an LLM classifier plus retrieval, reporting gains on DevOps-Eval that are compromised by possible test-set leakage.

Reference graph

Works this paper leans on

23 extracted references · 7 canonical work pages · cited by 2 Pith papers

[1]

Could you convert {query} from {in-lang} to {out-lang}?

Source dataset for each domain. from English to the target language. An example template is “Could you convert {query} from {in-lang} to {out-lang}?”. For cross-lingual prompts, we followed a similar approach, creating five unique templates and using an LLM to gen- erate 100 more based on these examples. These templates were ten applied to the UltraChat d...

work page 2023
[3]

Jiang, D., Ren, X., and Lin, B. Y . Llm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306.02561, 2023b. Ju, C., Bibaut, A., and van der Laan, M. The relative per- formance of ensemble methods with deep convolutional neural networks for image classification. Journal of Ap- plied Statistics, 4...

arXiv
[6]

Gshard: Scaling giant models with conditional computation and automatic sharding

Lepikhin, D., Lee, H., Xu, Y ., Chen, D., Firat, O., Huang, Y ., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668,

arXiv 2006
[7]

lin Chiang, W., Zheng, L., Dunlap, L., Gonzalez, J

URL https://arxiv.org/abs/ 2406.11939. lin Chiang, W., Zheng, L., Dunlap, L., Gonzalez, J. E., Stoica, I., Mooney, P., Dane, S., Howard, A., and Keat- ing, N. Lmsys - chatbot arena human preference predic- tions. https://kaggle.com/competitions/ lmsys-chatbot-arena,

arXiv
[8]

Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., and Zhou, J

Kaggle. Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., and Zhou, J. Routing to the expert: Efficient reward- guided ensemble of large language models.arXiv preprint arXiv:2311.08692,

arXiv
[9]

Muennighoff, N., Tazi, N., Magne, L., and Reimers, N

URL https://arxiv.org/abs/2308.08747. Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. Mteb: Massive text embedding benchmark,

arXiv
[10]

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al

URL https://arxiv.org/abs/2210.07316. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744,

arXiv
[12]

doi: 10.1109/HCS61935.2024.10664717

IEEE Computer Society. doi: 10.1109/HCS61935.2024.10664717. URL https://doi.ieeecomputersociety.org/ 10.1109/HCS61935.2024.10664717. Prabhakar, R., Sivaramakrishnan, R., Gandhi, D., Du, Y ., Wang, M., Song, X., Zhang, K., Gao, T., Wang, A., Li, K., Sheng, Y ., Brot, J., Sokolov, D., Vivek, A., Leung, C., Sabnis, A., Bai, J., Zhao, T., Gottscho, M., Jackso...

arXiv 2024

Show all 23 references

[13]

Raju, R., Jain, S., Li, B., Li, J., and Thakkar, U

URL https://arxiv.org/abs/2405.07518. Raju, R., Jain, S., Li, B., Li, J., and Thakkar, U. Constructing domain-specific evaluation sets for llm-as-a-judge. arXiv preprint arXiv:2408.08808,

arXiv
[16]

Llm routing with benchmark datasets

Composition of Experts: A Modular Compound AI System Leveraging Large Language Models Shnitzer, T., Ou, A., Silva, M., Soule, K., Sun, Y ., Solomon, J., Thompson, N., and Yurochkin, M. Llm routing with benchmark datasets. In NeurIPS 2023 Workshop on Dis- tribution Shifts: New ...

2023
[17]

Sukhbaatar, S., Golovneva, O., Sharma, V ., Xu, H., Lin, X

URL https://arxiv.org/abs/2402.06619. Sukhbaatar, S., Golovneva, O., Sharma, V ., Xu, H., Lin, X. V ., Rozi`ere, B., Kahn, J., Li, D., tau Yih, W., Weston, J., and Li, X. Branch-train-mix: Mixing expert llms into a mixture-of-experts llm,

arXiv
[18]

co/datasets/sambanovasystems/ x-self-instruct-seed-32

URL https://huggingface. co/datasets/sambanovasystems/ x-self-instruct-seed-32 . Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ram ´e, A., et al. Gemma 2: Improving open language models at a practical ...

arXiv
[19]

Llama 2: Open foundation and fine- tuned chat models

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 ,

arXiv
[20]

Mmlu-pro: A more robust and challenging multi- task language understanding benchmark (published at neurips 2024 track datasets and benchmarks),

Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark (published at neurips...

2024
[21]

Zhao, J., Wang, T., Abid, W., Angus, G., Garg, A., Kinnison, J., Sherstinsky, A., Molino, P., Addair, T., and Rishi, D

URL https://arxiv.org/abs/2406.01574. Zhao, J., Wang, T., Abid, W., Angus, G., Garg, A., Kinnison, J., Sherstinsky, A., Molino, P., Addair, T., and Rishi, D. Lora land: 310 fine-tuned llms that rival gpt-4, a technical report, 2024a. Zhao, W., Ren, X., Hessel, J., Cardie, C., ...

arXiv
[22]

A M ETHODOLOGY TO CREATE MULTILINGUAL PROMPTS Multilingual prompts can be categorized into three over- arching formats: native, translation, and cross-lingual

URL https: //arxiv.org/abs/2306.05685. A M ETHODOLOGY TO CREATE MULTILINGUAL PROMPTS Multilingual prompts can be categorized into three over- arching formats: native, translation, and cross-lingual. In native formats, prompts are written in the target language, and the model i...

2024 arXiv
[2009]

Sheikholeslami, F., Jain, S., and Giannakis, G

URL http: //axon.cs.byu.edu/˜martinez/classes/ 778/Papers/settles.activelearning.pdf. Sheikholeslami, F., Jain, S., and Giannakis, G. B. Efficient randomized defense against adversarial attacks in deep convolutional neural networks. In ICASSP 2019-2019 IEEE International Confe...

2019
[2018]

B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

2001 arXiv
[2019]

Sheikholeslami, F., Jain, S., and Giannakis, G. B. Minimum uncertainty based detection of adversaries in deep neural networks. In 2020 Information Theory and Applications Workshop (ITA), pp. 1–16. IEEE,

2020
[2020]

and Byrne, B

Kumar, S. and Byrne, B. Minimum bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pp. 169–176,

2004
[2022]

Sambanova sn40l rdu: Breaking the barrier of trillion+ parameter scale gen ai computing

Prabhakar, R. Sambanova sn40l rdu: Breaking the barrier of trillion+ parameter scale gen ai computing. In 2024 IEEE Hot Chips 36 Symposium (HCS) , pp. 1–24, Los Alamitos, CA, USA, aug

2024
[2023]

Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, ...

arXiv
[2024]

Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y ., Xie, G., Liu, Z., and Sun, M

URL https://arxiv.org/abs/2403.04132. Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y ., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback,

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Could you convert {query} from {in-lang} to {out-lang}?

Source dataset for each domain. from English to the target language. An example template is “Could you convert {query} from {in-lang} to {out-lang}?”. For cross-lingual prompts, we followed a similar approach, creating five unique templates and using an LLM to gen- erate 100 more based on these examples. These templates were ten applied to the UltraChat d...

work page 2023

[2] [3]

Jiang, D., Ren, X., and Lin, B. Y . Llm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306.02561, 2023b. Ju, C., Bibaut, A., and van der Laan, M. The relative per- formance of ensemble methods with deep convolutional neural networks for image classification. Journal of Ap- plied Statistics, 4...

arXiv

[3] [6]

Gshard: Scaling giant models with conditional computation and automatic sharding

Lepikhin, D., Lee, H., Xu, Y ., Chen, D., Firat, O., Huang, Y ., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668,

arXiv 2006

[4] [7]

lin Chiang, W., Zheng, L., Dunlap, L., Gonzalez, J

URL https://arxiv.org/abs/ 2406.11939. lin Chiang, W., Zheng, L., Dunlap, L., Gonzalez, J. E., Stoica, I., Mooney, P., Dane, S., Howard, A., and Keat- ing, N. Lmsys - chatbot arena human preference predic- tions. https://kaggle.com/competitions/ lmsys-chatbot-arena,

arXiv

[5] [8]

Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., and Zhou, J

Kaggle. Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., and Zhou, J. Routing to the expert: Efficient reward- guided ensemble of large language models.arXiv preprint arXiv:2311.08692,

arXiv

[6] [9]

Muennighoff, N., Tazi, N., Magne, L., and Reimers, N

URL https://arxiv.org/abs/2308.08747. Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. Mteb: Massive text embedding benchmark,

arXiv

[7] [10]

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al

URL https://arxiv.org/abs/2210.07316. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744,

arXiv

[8] [12]

doi: 10.1109/HCS61935.2024.10664717

IEEE Computer Society. doi: 10.1109/HCS61935.2024.10664717. URL https://doi.ieeecomputersociety.org/ 10.1109/HCS61935.2024.10664717. Prabhakar, R., Sivaramakrishnan, R., Gandhi, D., Du, Y ., Wang, M., Song, X., Zhang, K., Gao, T., Wang, A., Li, K., Sheng, Y ., Brot, J., Sokolov, D., Vivek, A., Leung, C., Sabnis, A., Bai, J., Zhao, T., Gottscho, M., Jackso...

arXiv 2024