arxiv: 2601.21257 · v2 · submitted 2026-01-29 · 💻 cs.CL

Recognition: no theorem link

MoCo: A One-Stop Shop for Model Collaboration Research

Shangbin Feng , Yuyang Bai , Ziyuan Yang , Yike Wang , Zhaoxuan Tan , Jiajie Yan , Zhenyu Lei , Wenxuan Ding

show 12 more authors

Weijia Shi Haojin Wang Zhenting Qi Yuru Jiang Heng Wang Chengsong Huang Yu Fei Jihan Yao Yilun Du Luke Zettlemoyer Yejin Choi Yulia Tsvetkov

Authors on Pith no claims yet

Pith reviewed 2026-05-16 10:15 UTC · model grok-4.3

classification 💻 cs.CL

keywords model collaborationlanguage modelsbenchmarking librarymulti-model systemscollaboration algorithmsensemble methodsAI modularity

0 comments

The pith

MoCo is a Python library with 26 collaboration methods that shows they outperform single language models in 61 percent of tested settings on average.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces MoCo as a unified Python library to bring together previously scattered research on model collaboration, where multiple language models exchange information to complement each other. It packages 26 methods operating at routing, text, logit, and parameter levels, along with 25 datasets covering reasoning, question answering, code, safety, and other tasks. Experiments run through the library establish that collaboration beats non-collaborative baselines in 61.0 percent of model-data combinations, with the strongest approaches delivering gains as large as 25.8 percent. The toolkit also supports scaling studies, efficiency measurements, and identification of problems that isolated models cannot solve.

Core claim

MoCo consolidates 26 model collaboration algorithms into one executable and benchmarkable framework and demonstrates through extensive runs that these strategies improve performance over single models in 61.0 percent of settings on average while enabling analysis of when and how collaboration helps most.

What carries the argument

The MoCo Python library, which executes, benchmarks, and compares collaboration methods that let models exchange information at routing, text, logit, or parameter levels.

If this is right

Most collaboration strategies improve results over single models across the majority of tested combinations.
The strongest methods deliver performance lifts up to 25.8 percent.
Collaborative systems can solve problems that single language models fail on.
The library enables direct comparison of training and inference costs across different collaboration approaches.
Users can bring their own datasets to evaluate collaboration on custom tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A shared library could reduce repeated implementation work and make it easier to compare new collaboration ideas against established ones.
The observed gains suggest value in building systems where models dynamically choose when and with whom to collaborate rather than always operating alone.
Longer term, this direction supports modular AI designs in which many smaller specialized models replace one large monolithic model.

Load-bearing premise

The 26 implemented methods are faithful reproductions of the original algorithms and the chosen datasets and metrics represent real-world collaboration benefits.

What would settle it

A side-by-side run of any original collaboration method and its MoCo re-implementation on identical data and metrics that shows materially different performance numbers.

Figures

Figures reproduced from arXiv: 2601.21257 by Chengsong Huang, Haojin Wang, Heng Wang, Jiajie Yan, Jihan Yao, Luke Zettlemoyer, Shangbin Feng, Weijia Shi, Wenxuan Ding, Yejin Choi, Yike Wang, Yilun Du, Yu Fei, Yulia Tsvetkov, Yuru Jiang, Yuyang Bai, Zhaoxuan Tan, Zhenting Qi, Zhenyu Lei, Ziyuan Yang.

**Figure 1.** Figure 1: MOCO is a comprehensive library for model collaboration research. Download MOCO, write a config file specifying model collaboration setups (models, data, hardware, etc.), execute and compare diverse model collaboration algorithms with MOCO. and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.2 1. Introduction Language models (LMs) are increasingly not used in isolat… view at source ↗

**Figure 2.** Figure 2: Scaling the number of models in model collaboration systems and evaluating on reasoning, QA, and safety domains. We observe a consistent upward trend that further improves over the best single model, with text-level and weight-level methods being more scalable and benefiting from a larger pool of diverse models. This indicates that by scaling up model collaboration, we could build bottom-up compositional A… view at source ↗

**Figure 3.** Figure 3: Impact of model pool diversity on collaboration performance. The x-axis shows the configurations of model pool diversity: 1 × 8, 2 × 4, 4 × 2 and 8 × 1. Results demonstrate that model collaboration benefits from increased diversity among participating models, indicating the need for model specialization. vision future research on dynamic model selection in model collaboration systems uniquely supported by… view at source ↗

**Figure 4.** Figure 4: For problems where none of the LLMs could solve individually, what percentage of them are solvable with the model collaboration system, across diverse tasks and collaboration strategies. We observe consistent collaborative emergence across settings with an average of 18.5%, indicating that many model collaboration algorithms do not merely offer a union of existing capabilities: new skills emerge in the col… view at source ↗

**Figure 5.** Figure 5: Employing random, prompt-based, or description-based strategies to select 3 models out of 8 for collaboration. Both strategies outperform the random baseline and no collaboration, indicating the importance of model selection strategies and highlighting the need for future research. on four levels of information exchange across models. Raffel (2023) proposes to “build machine learning models like open sou… view at source ↗

**Figure 6.** Figure 6: Collaborative emergence on the General-purpose QA domain. 0 10 20 30 Previously "Impossible" Problems Solved (%) logit_contrastive expo structure model_swarms greedy_soup logit_fusion switch_generation knowledge_card lorahub sparta multiagent_feedback multiagent_refine heterogeneous_swarms llm_blender mentor_collab dare_ties nudging 27.0 24.2 18.9 18.7 17.8 15.5 15.4 14.1 13.0 12.9 12.6 10.4 9.9 8.2 7.5 7.… view at source ↗

**Figure 7.** Figure 7: Collaborative emergence on the Safety domain. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

**Figure 8.** Figure 8: Collaborative emergence on the Coding domain. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

read the original abstract

Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, and complement each other. Existing research on this topic has mostly been disparate and disconnected, from different research communities, and lacks rigorous comparison. To consolidate existing research and establish model collaboration as a school of thought, we present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale. MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange such as routing, text, logit, and model parameters. MoCo integrates 25 evaluation datasets spanning reasoning, QA, code, safety, and more, while users could flexibly bring their own data. Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average, with the most effective methods outperforming by up to 25.8%. We further analyze the scaling of model collaboration strategies, the training/inference efficiency of diverse methods, highlight that the collaborative system solves problems where single LMs struggle, and discuss future work in model collaboration, all made possible by MoCo. We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

MoCo bundles 26 existing collaboration methods and 25 datasets into one library and runs large-scale comparisons, but the 61% outperformance claim rests on unverified reimplementations.

read the letter

MoCo collects 26 collaboration methods spanning routing, logits, parameters, and text, plus 25 datasets, into a single Python library and runs them against single-model baselines at scale. The new element is the unified toolkit itself plus the accompanying efficiency and scaling analyses that were not available in one place before. The experiments show collaboration beats the no-collaboration baseline in 61% of model-data pairs on average, with the strongest methods gaining up to 25.8%, and they flag cases where single models fail but the combined system succeeds. That kind of consolidated benchmark data is genuinely useful for anyone trying to compare or extend these approaches. The soft spot is the 61% figure. It comes from the authors' reimplementations of methods that differ in temperature handling, top-k logic, and gradient flow. The paper gives no per-method output-matching tests against the originals, no implementation checklist, and no ablation that separates reimplementation choices from actual collaboration gains. Without those, the exact percentage is harder to trust. Statistical details on the results are also thin. This is for researchers who want a ready-made starting point for experiments on modular or multi-model systems rather than a theoretical advance. It deserves peer review because the library contribution is concrete and the comparison scope is broad, even if the empirical claims need tighter checks on fidelity.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces MoCo, a Python library implementing 26 model collaboration methods spanning routing, text-level composition, logit fusion, and parameter merging. It integrates 25 datasets across reasoning, QA, code, safety and other tasks, and reports extensive experiments showing that most collaboration strategies outperform single-model baselines in 61.0% of (model, data) settings on average, with the strongest methods achieving gains up to 25.8%. Additional analyses cover scaling behavior, training/inference efficiency, and cases where collaboration solves problems that defeat individual LMs.

Significance. If the reimplementations prove faithful, MoCo supplies a much-needed standardized benchmark suite that consolidates previously disconnected lines of work on model collaboration. The empirical observation that collaboration helps in a majority of settings provides concrete motivation for modular, decentralized AI architectures and could accelerate reproducible research in this area.

major comments (2)

[Abstract] Abstract and Experiments section: the central 61.0% outperformance statistic is reported without the total number of (model, data) pairs evaluated, per-setting variance, or any statistical significance tests, making it impossible to assess whether the result is robust or could be driven by a small number of high-variance settings.
[Methods] Methods and Experiments sections: the 61.0% and 25.8% figures rest entirely on the authors' reimplementations of the 26 methods, yet no implementation checklist, hyperparameter reproduction protocol, or output-matching verification against the original papers is supplied. Small differences in temperature scaling, gradient stopping, or top-k handling could reverse the sign of many reported deltas.

minor comments (1)

[Abstract] Abstract: the phrase 'users could flexibly bring their own data' should be expanded to specify the exact data-loading interface and required format.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments and the recommendation for minor revision. We address each point below and will update the manuscript to improve statistical reporting and reproducibility.

read point-by-point responses

Referee: [Abstract] Abstract and Experiments section: the central 61.0% outperformance statistic is reported without the total number of (model, data) pairs evaluated, per-setting variance, or any statistical significance tests, making it impossible to assess whether the result is robust or could be driven by a small number of high-variance settings.

Authors: We agree that these details are necessary for evaluating robustness. In the revised manuscript we will state the exact total number of (model, data) pairs evaluated, report per-setting variance (standard deviation of the outperformance indicator), and include statistical significance tests (binomial test against 50% and bootstrap confidence intervals) in both the abstract and Experiments section. revision: yes
Referee: [Methods] Methods and Experiments sections: the 61.0% and 25.8% figures rest entirely on the authors' reimplementations of the 26 methods, yet no implementation checklist, hyperparameter reproduction protocol, or output-matching verification against the original papers is supplied. Small differences in temperature scaling, gradient stopping, or top-k handling could reverse the sign of many reported deltas.

Authors: We acknowledge the risk of implementation differences. The revised Methods section will contain an explicit implementation checklist and hyperparameter reproduction protocol covering temperature, top-k, gradient stopping, and other relevant settings for each of the 26 methods. We will also document any verification performed against the original papers. revision: yes

Circularity Check

0 steps flagged

No circularity: results are empirical benchmarks, not derivations reducing to author inputs

full rationale

The paper introduces a library (MoCo) that reimplements 26 collaboration methods and evaluates them on 25 datasets. The headline statistic (61% of (model, data) settings show outperformance, up to +25.8%) is obtained by direct execution of the reimplementations against external benchmarks. No equations, uniqueness theorems, ansatzes, or fitted parameters are defined in terms of the target results; the central claims do not reduce by construction to self-citations or author-chosen inputs. Any self-citations present are incidental and non-load-bearing for the empirical findings.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical software library paper; no free parameters, mathematical axioms, or invented entities are introduced or required for the central claims.

pith-pipeline@v0.9.0 · 5618 in / 986 out tokens · 23525 ms · 2026-05-16T10:15:01.066208+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages · 10 internal anchors

[1]

Program Synthesis with Large Language Models

Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

work page internal anchor Pith review Pith/arXiv arXiv
[2]

Theoremqa: A theorem-driven question answering dataset

Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y ., Ma, X., Xu, J., Wang, X., and Xia, T. Theoremqa: A theorem-driven question answering dataset. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889–7901, 2023b. Chiu, Y . Y ., Jiang, L., Lin, B. Y ., Park, C. Y ., Li, S. S., Ravi, S., Bhatia, M., Antoniak, M., Tsvetk...

work page arXiv 2023
[3]

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

work page internal anchor Pith review Pith/arXiv arXiv
[4]

Training Verifiers to Solve Math Word Problems

Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

Nudging: Inference-time alignment of llms via guided decoding.arXiv preprint arXiv:2410.09300,

Fei, Y ., Razeghi, Y ., and Singh, S. Nudging: Inference-time alignment of llms via guided decoding.arXiv preprint arXiv:2410.09300,

work page arXiv
[6]

Knowledge card: Filling llms’ knowledge gaps with plug-in specialized language models

Feng, S., Shi, W., Bai, Y ., Balachandran, V ., He, T., and Tsvetkov, Y . Knowledge card: Filling llms’ knowledge gaps with plug-in specialized language models. InThe Twelfth International Conference on Learning Represen- tations, 2024a. Feng, S., Shi, W., Wang, Y ., Ding, W., Balachandran, V ., and Tsvetkov, Y . Don’t hallucinate, abstain: Identifying ll...

work page arXiv 2025
[7]

Arcee’s MergeKit: A toolkit for merging large lan- guage models

Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V ., Benedict, B., McQuade, M., and Solawetz, J. Arcee’s MergeKit: A toolkit for merging large lan- guage models. In Dernoncourt, F., Preo t ¸iuc-Pietro, D., and Shimorina, A. (eds.),Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Pro- cessing: Industry Track...

work page 2024
[8]

The Llama 3 Herd of Models

Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

work page internal anchor Pith review Pith/arXiv arXiv
[9]

Care: Aligning language models for regional cultural awareness.arXiv preprint arXiv:2504.05154,

Guo, G., Naous, T., Wakaki, H., Nishimura, Y ., Mitsu- fuji, Y ., Ritter, A., and Xu, W. Care: Aligning language models for regional cultural awareness.arXiv preprint arXiv:2504.05154,

work page arXiv
[10]

Relayllm: Efficient reasoning via collaborative decod- ing.arXiv preprint arXiv:2601.05167,

Huang, C., Zheng, T., Huang, L., Li, J., Liu, H., and Huang, J. Relayllm: Efficient reasoning via collaborative decod- ing.arXiv preprint arXiv:2601.05167,

work page arXiv
[11]

Artificial hivemind: The open-ended homogeneity of language models (and beyond)

Jiang, L., Chai, Y ., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y ., Sap, M., and Choi, Y . Artificial hivemind: The open-ended homogeneity of language models (and beyond). InThe Thirty-ninth Annual Conference on Neu- ral Information Processing Systems Datasets and Bench- marks Track, 2025a. Jiang, P., Lin, J., Cao, L., Tian, R., Kang, S., Wang, Z., ...

work page arXiv
[12]

Pub- medqa: A dataset for biomedical research question an- swering

Jin, Q., Dhingra, B., Liu, Z., Cohen, W., and Lu, X. Pub- medqa: A dataset for biomedical research question an- swering. InProceedings of the 2019 conference on em- pirical methods in natural language processing and the 10 MoCo: A One-Stop Shop for Model Collaboration Research 9th international joint conference on natural language processing (EMNLP-IJCNLP...

work page 2019
[13]

Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V ., Liu, A., Dziri, N., Lyu, S., et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

work page internal anchor Pith review Pith/arXiv arXiv
[14]

In-the-flow agentic system opti- mization for effective planning and tool use

Li, Z., Zhang, H., Han, S., Liu, S., Xie, J., Zhang, Y ., Choi, Y ., Zou, J., and Lu, P. In-the-flow agentic system opti- mization for effective planning and tool use. InNeurIPS 2025 Workshop on Efficient Reasoning,

work page 2025
[15]

Liu, A., Han, X., Wang, Y ., Tsvetkov, Y ., Choi, Y ., and Smith, N. A. Tuning language models by proxy. InFirst Conference on Language Modeling, 2024a. Liu, C. Y ., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y ., and Zhou, Y . Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024b. Liu, G. K.-M.,...

work page arXiv
[16]

General-reasoner: Advancing llm reasoning across all domains.arXiv preprint arXiv:2505.14652,

Ma, X., Liu, Q., Jiang, D., Zhang, G., Ma, Z., and Chen, W. General-reasoner: Advancing llm reasoning across all domains.arXiv preprint arXiv:2505.14652,

work page arXiv
[17]

Olmo 3

Olmo, T., Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Tim- bers, F., Ivison, H., et al. Olmo 3.arXiv preprint arXiv:2512.13961,

work page internal anchor Pith review Pith/arXiv arXiv
[18]

M., Chang, Y ., and Iyyer, M

Pham, C. M., Chang, Y ., and Iyyer, M. Clipper: Com- pression enables long-context synthetic data generation. arXiv preprint arXiv:2502.14854,

work page arXiv
[19]

ToolRL: Reward is All Tool Learning Needs

Qian, C., Acikgoz, E. C., He, Q., Wang, H., Chen, X., Hakkani-T¨ur, D., Tur, G., and Ji, H. Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958,

work page internal anchor Pith review Pith/arXiv arXiv
[20]

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

work page internal anchor Pith review Pith/arXiv arXiv
[21]

The hallucination tax of rein- forcement finetuning.arXiv preprint arXiv:2505.13988,

11 MoCo: A One-Stop Shop for Model Collaboration Research Song, L., Shi, T., and Zhao, J. The hallucination tax of rein- forcement finetuning.arXiv preprint arXiv:2505.13988,

work page arXiv
[22]

W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al

Suzgun, M., Scales, N., Sch¨arli, N., Gehrmann, S., Tay, Y ., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. Challenging big-bench tasks and whether chain-of- thought can solve them. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051,

work page 2023
[23]

Align- ing large language models with implicit preferences from user-generated content.arXiv preprint arXiv:2506.04463,

Tan, Z., Li, Z., Liu, T., Wang, H., Yun, H., Zeng, M., Chen, P., Zhang, Z., Gao, Y ., Wang, R., et al. Align- ing large language models with implicit preferences from user-generated content.arXiv preprint arXiv:2506.04463,

work page arXiv
[24]

Minicheck: Efficient fact-checking of llms on grounding documents

Tang, L., Laban, P., and Durrett, G. Minicheck: Efficient fact-checking of llms on grounding documents. InPro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8818–8847,

work page 2024
[25]

Scir- iff: A resource to enhance language model instruction- following over scientific literature

Wadden, D., Shi, K., Morrison, J., Li, A., Naik, A., Singh, S., Barzilay, N., Lo, K., Hope, T., Soldaini, L., et al. Scir- iff: A resource to enhance language model instruction- following over scientific literature. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 6083–6120,

work page 2025
[26]

Always Tell Me The Odds: Fine-grained Conditional Probability Estimation

Wang, L., Jiang, Z., Liu, A., and Van Durme, B. Always tell me the odds: Fine-grained conditional probability estimation.arXiv preprint arXiv:2505.01595, 2025a. Wang, X., Li, H., Chen, H., Zhang, Z., and Zhu, W. Mod- ular machine learning: An indispensable path towards new-generation large language models.arXiv preprint arXiv:2504.20020, 2025b. Wang, Y .,...

work page internal anchor Pith review Pith/arXiv arXiv
[27]

D., and Potts, C

Wu, Z., Yu, Q., Arora, A., Manning, C. D., and Potts, C. Improved representation steering for language models. arXiv preprint arXiv:2505.20809,

work page arXiv
[28]

Prompt-mii: Meta-learning instruction induction for llms.arXiv preprint arXiv:2510.16932,

Xiao, E., Zeng, Y ., Chen, A., Li, C.-J., Bertsch, A., and Neu- big, G. Prompt-mii: Meta-learning instruction induction for llms.arXiv preprint arXiv:2510.16932,

work page arXiv
[29]

Qwen2 Technical Report

Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai...

work page internal anchor Pith review Pith/arXiv arXiv
[30]

Netsafe: Exploring the topological safety of multi-agent system

Yu, M., Wang, S., Zhang, G., Mao, J., Yin, C., Liu, Q., Wang, K., Wen, Q., and Wang, Y . Netsafe: Exploring the topological safety of multi-agent system. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 2905–2938,

work page 2025
[31]

L., Athiwaratkun, B., and Dao, T

Zhang, M., Mishra, M., Zhou, Z., Brandon, W., WANG, J., Kim, Y ., Ragan-Kelley, J., Song, S. L., Athiwaratkun, B., and Dao, T. Ladder-residual: Parallelism-aware ar- chitecture for accelerating large model inference with communication overlapping. InForty-second Interna- tional Conference on Machine Learning, 2025a. 12 MoCo: A One-Stop Shop for Model Coll...

work page arXiv
[32]

The majority is not always right: Rl training for solution aggregation.arXiv preprint arXiv:2509.06870,

Zhao, W., Aggarwal, P., Saha, S., Celikyilmaz, A., We- ston, J., and Kulikov, I. The majority is not always right: Rl training for solution aggregation.arXiv preprint arXiv:2509.06870,

work page arXiv
[33]

Weak- to-strong extrapolation expedites alignment

Zheng, C., Wang, Z., Ji, H., Huang, M., and Peng, N. Weak- to-strong extrapolation expedites alignment. InICML 2024 Workshop on Models of Human Feedback for AI Alignment,

work page 2024
[34]

Agieval: A human- centric benchmark for evaluating foundation models

Zhong, W., Cui, R., Guo, Y ., Liang, Y ., Lu, S., Wang, Y ., Saied, A., Chen, W., and Duan, N. Agieval: A human- centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguis- tics: NAACL 2024, pp. 2299–2314,

work page 2024
[35]

We employ the default hyperparameters in MOCOfor different model collaboration algorithms

Implementation DetailsWe by default employ 512 max new tokens, with an exception of 1024 for coding tasks, and τ= 0.7 and p= 0.9 for temperature and top-p sampling in text generation. We employ the default hyperparameters in MOCOfor different model collaboration algorithms. Model pool #1 includes the following three models from (Jiang et al., 2025c):BUNSE...

work page 2025
[36]

2000 1000 ARC-Challenge (Clark et al.,

work page 2000
[37]

2091 2092 MedQA (Jin et al.,

work page 2091