Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Token-level probability blending lets three LLMs outreason any one of them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

MCSU-based vocabulary alignment plus distance-based dynamic selection (DDS) lets several LLMs vote token-by-token, beating single models and prior ensemble baselines on multiple reasoning benchmarks without training.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Clever MCSU alignment and consistent few-point gains, but the KL-based selector as written is mathematically undefined, so the experimental numbers don't yet pin down a working method. the 4 major comments →

arxiv 2508.18763 v1 pith:HKMS2EJD submitted 2025-08-26 cs.AI

Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units

Chao Hao , Zezheng Wang , Yanhua Huang , Ruiwen Xu , Wenzhe Niu , Xin Liu , Zitong Yu This is my paper
classification cs.AI
keywords large language modelstoken-level ensemblingminimal complete semantic unitsdistribution distancedynamic selectiontraining-free reasoning enhancementKL divergencemulti-model collaboration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models disagree not only in their final answers but in every next-token probability they assign. This paper argues that those per-step distributions are where each model's knowledge lives, and that combining them at token level can improve reasoning without any training. The obstacle is that different models tokenize text differently, so the paper introduces minimal complete semantic units (MCSU)—words, numbers, or punctuation marks—as a common currency for comparing distributions. On top of that, a distribution-distance-based dynamic selection strategy (DDS) keeps only the distributions that cluster together at each autoregressive step, on the principle that correct answers are usually unique while errors are scattered. A reader should care because if this works, three moderate models can be bolted together at inference time to outperform each of them alone, occasionally producing correct answers that none of them would have produced.

Core claim

The central claim is that dynamic, distance-based selection over token-level probability distributions—after aligning models on minimal complete semantic units—reliably improves multi-step reasoning accuracy on math, commonsense, and symbolic benchmarks, and does so without retraining or vocabulary projection. At each step, DDS takes the top-k MCSU probabilities from each model, computes pairwise KL divergences, discards outlier distributions beyond a tuned threshold, and averages the rest; the highest-probability MCSU wins. The paper reports that this beats every single member model, beats output-level voting, and beats token-level ensembling baselines that align vocabularies with anchors o

What carries the argument

The two working parts are MCSU and DDS. MCSU replaces tokens as the alignment unit: a token sequence is grown until it forms a complete word, number, or punctuation mark, and the MCSU's probability is the product of its tokens' conditional probabilities, so different tokenizers can be compared in one semantically aligned space. DDS is the selection mechanism: after top-k truncation and padding to a union vocabulary, pairwise KL divergence between MCSU distributions determines which models are kept; distributions within the threshold cluster near what is presumed to be the correct continuation, outliers are dropped, and the surviving distributions are averaged. The threshold is set globally t

Load-bearing premise

The load-bearing assumption is that at every step the correct continuation is essentially unique and similar across models, so distributions that lie close together in KL divergence are near the truth and outliers can be discarded; the paper's own failure analysis shows styles and reasoning paths can legitimately diverge, and then the selection rule keeps everything and the highest-confidence wrong answer can win.

What would settle it

Take a benchmark with multiple valid answer phrasings (or rewrite GSM8K solutions so two models reach the same answer through different wording) and record, at each step, the pairwise KL divergence among the three MCSU distributions. If DDS accuracy on steps where the pairwise KL exceeds the 0.1 threshold is no better than choosing the single most confident model, the clustering premise is the reason for both the gains and the failures; a second check is whether a single high-confidence wrong MCSU (e.g., one model assigning 0.8 to an incorrect option while others disagree) reliably flips the f

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On the paper's results, training-free token-level ensemble with dynamic selection outperforms the best single model on all three reasoning families tested, with gains around one point on math and two to four points on commonsense and symbolic reasoning.
  • More models do not automatically help: adding a third model to a two-model collaboration helped on some tasks and hurt on others, which motivates dynamic per-step selection rather than fixed ensembling.
  • Correct answers can be produced by token-level recombination even when every member model's final answer is wrong, so token-level collaboration offers a capability that majority voting or answer selection cannot recover.
  • DDS transfers to code generation and a Chinese knowledge benchmark, and improves truthfulness scores on TruthfulQA, suggesting the mechanism is not confined to one task or language.
  • Because the method is training-free and needs no referee model, it can be deployed by loading several open-weight models and averaging aligned distributions at inference time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The global epsilon of 0.1 is likely not universal; a testable extension would adapt epsilon per step or per model pair, since distributional spread varies with task and with where in the reasoning process a step sits.
  • MCSU alignment might be reusable beyond ensembling—for example, in speculative decoding, inter-model verification, or any setting where multiple tokenizers must be compared without learned projections.
  • The emergence cases suggest token-level idea collision could be studied deliberately: measuring how often the winning MCSU comes from a model whose distribution was in the minority earlier would reveal whether the effect is a union of correct prefixes or a genuine mixture.
  • If the clustering assumption fails on high-variance or open-ended tasks (synonyms, stylistic diversity), DDS may degrade to selecting the most confident wrong token; measuring accuracy conditioned on pairwise KL at each step would map where the method is safe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DDS (Distribution Distance-based dynamic Selection), a training-free token-level ensemble method for heterogeneous LLMs. To handle vocabulary misalignment, it introduces Minimal Complete Semantic Units (MCSUs), aligning models at the level of words/numbers rather than raw tokens. At each autoregressive step, each model's next-MCSU distribution is top-k truncated; pairwise KL divergences over the union support are used to discard outlier distributions; the retained distributions are averaged and decoded greedily. Experiments on 12 reasoning benchmarks (math, commonsense, symbolic), plus code, Chinese, and reliability evaluations, report consistent accuracy gains over single models and output-level/token-level baselines. The paper also presents anecdotal cases where DDS is correct though all member models are wrong, and discusses failure cases in Appendix F.

Significance. If the method works as claimed, token-level distribution blending is a valuable training-free direction for combining heterogeneous LLMs, and the MCSU concept is a sensible way to reduce vocabulary mismatch without learned projections. The paper ships a broad benchmark suite and several ablations, and explicitly discusses limitations, including the failure mode where divergent-but-correct reasoning breaks the clustering assumption. These are genuine strengths. However, the central algorithm as written is not computable, and the empirical support lacks statistical grounding; the headline gains are small and depend on a validation-tuned threshold. The significance therefore hinges on a few fixes plus a clearer statement of the implemented selection rule.

major comments (4)
  1. [§3.2, Eq. (2)] The KL divergence used for selection is undefined as written. Eq. (3) sets all but the top-k MCSUs to zero; Eq. (4) only replaces values outside the union U with 1e-9, but for x∈U not in model i's top-k, P*_i(x) remains 0. Since the sum in Eq. (5) runs over U, and with k=5 and vocabularies of tens of thousands, each pair almost always has x where P*_i(x)>0 and P*_j(x)=0 (or vice versa), giving DKL=+∞. Additionally, P*_i is not renormalized after truncation, so it is not a probability distribution and DKL is not a valid divergence. The experimental tables therefore cannot be traced to the specified algorithm. The authors should either redefine P*_i to assign small mass to every element of U and renormalize, or explicitly state that the implementation smooths all union elements; the current text is inconsistent with the reported results.
  2. [§3.2] The MCSU probability is defined as the product of conditional token probabilities over the tokens in the MCSU. This is a prefix probability, not the probability of a disjoint next-MCSU event; the sum over MCSUs is not 1, so the 'distribution' used in Eq. (5) is not normalized even before top-k truncation. This compounds the KL issue in Eq. (5). The paper should clarify how these quantities are normalized (if at all) before divergence computation, and justify why the resulting value is a meaningful distance.
  3. [Tables 1–3, Table 7] The central claim that DDS 'significantly enhances' reasoning is not supported by statistical evidence. Tables 1–3 report no error bars, confidence intervals, or significance tests; for several benchmarks the DDS gain over the best single model is under 1 point (e.g., Table 1: AddSub 65.5 vs. Qwen 65.5; Table 3: Penguin 83.0 vs. GLM 81.5, but with no variance estimate). Moreover, Table 7 shows the method is sensitive to ε, and ε=0.1 is selected via validation (Appendix A), so the test numbers are conditional on a tuned hyperparameter. The authors should report standard deviations over repeated runs or bootstrap intervals, and state whether ε was fixed before seeing test sets. Without this, the consistent but small gains may be within noise.
  4. [§3.2 / Appendix F] The selection rule 'if DKL<ε then retain' is pairwise and non-transitive; for three models, A-B and B-C may be close while A-C is not, and the paper does not specify how the retained set is formed in that case. More fundamentally, the core assumption that correct distributions cluster while incorrect ones are scattered is acknowledged in Appendix F to fail when models produce divergent but correct continuations. Since that failure occurs at 'critical stages' such as key numbers/options, the proposed filtering can discard useful distributions exactly when cross-model reasoning styles differ. The authors should state the exact selection procedure and quantify how often the fallback 'retain all' is triggered; otherwise the reported gains cannot be attributed to the filtering mechanism rather than to simple averaging.
minor comments (5)
  1. [§3.1] The definition of MCSU relies on whitespace for English and on the assumption that Chinese tokens are complete words. This is reasonable for the tested benchmarks but should be stated as a design assumption; the claim that these two cases cover 'the majority of language types' is broad and unsupported.
  2. [Table 5 / §4.4] The reliability evaluation reports ROUGE, BLEU, and BLEURT on TruthfulQA. These are reference-based generation metrics; the standard TruthfulQA evaluation uses multiple-choice truthfulness and informativeness. The authors should clarify the reference set and why higher BLEU/ROUGE/BLEURT is interpreted as greater truthfulness.
  3. [Appendix A] The determination of ε by 'data-driven statistical analysis' is described only abstractly (histogram/CDF, mean). Reporting the actual distribution of KL values and the exact validation split would make the procedure reproducible.
  4. [§4.6] The emergent-capability examples are illustrative but not quantified. A systematic analysis of how often DDS produces a correct final answer when all three members are wrong, and how often it flips a correct member answer to wrong, would strengthen the claim.
  5. [Throughout] Minor typos and inconsistent notation: 'emsembling' in section headings, 'distrition' in §3.2, 'DSS' used in place of 'DDS' in Appendix C, and 'CommonseQA' for CommonsenseQA. The acronym DSS at line 'using DSS ensemble method' should be corrected.

Circularity Check

0 steps flagged

No significant circularity: the DDS/MCSU pipeline is operational and externally benchmarked; the only fitted value (KL threshold eps) is standard hyperparameter selection, and Appendix C self-citations are background, not load-bearing.

full rationale

Walk-through of the derivation chain: MCSU (Eq. 2) is an alignment construct; DDS (Eqs. 3-5) is an operational filter comparing distribution distances; the output is the argmax of the averaged retained distributions. None of these steps defines its target in terms of itself. The central claims ('DDS significantly enhances reasoning', Tables 1-4; 'emergent ability', Table 8) are measured against external benchmarks (SVAMP, GSM8K, CSQA, StrategyQA, BigBench, HumanEval, C-Eval, TruthfulQA), so they do not reduce to any input. The only fitted quantity is eps=0.1, set from validation KL statistics (Appendix A); since accuracy is a measured outcome and not a quantity forced by eps, this is ordinary hyperparameter selection, not 'fitted input called prediction' (Table 7 even shows sensitivity). Appendix C invokes several self-citations overlapping the authors (Hao et al. 2025a,b; Lin et al. 2025; Ye et al. 2025; Cai et al. 2025) for the background premise that the next-token distribution manifests LLM knowledge; however, the same premise is cited to non-self works in Section 1 (Hinton 2015; Wan et al. 2024; Radford et al. 2019), and the appendix itself concludes effectiveness is 'demonstrated ... through a large number of experiments.' Thus the self-citations are minor and non-load-bearing, warranting score 2 rather than 0. No uniqueness theorem is imported, no ansatz arrives via citation, and MCSU is transparently defined as 'a word, punctuation mark, or number' - an engineering construct, not a renamed result. Two concerns are flagged for correctness review, not circularity: (i) as written, Eqs. 3-5 can produce infinite/unnormalized KL values under top-k truncation with union support, so the algorithm is under-specified without renormalization/smoothing; (ii) Appendix F candidly acknowledges the clustering assumption fails when correct continuations are stylistically divergent. Neither is a self-referential reduction.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central claim depends mainly on the clustering assumption for correctness, on epsilon as a fitted hyperparameter, and on MCSU as the alignment device. None of these are derived from external constraints.

free parameters (2)
  • epsilon (KL divergence threshold) = 0.1
    Chosen by 'global mean through statistical methods' on a validation set (Appendix A); ablation in Table 7 shows accuracy varies by ~2.5 points across 0.05-0.5, so the headline results depend on this fitted value.
  • K (number of top MCSUs per model) = 5
    Set to 5 in implementation details with no ablation or justification; affects the support of each distribution and hence the KL distances.
axioms (4)
  • domain assumption Correct answers cluster in next-distribution space, incorrect answers are diverse.
    Underlies the DDS selection rule (Section 3.2); acknowledged as sometimes false in Appendix F.
  • domain assumption Whitespace-delimited sequences are complete semantic units for English; Chinese characters are MCSUs by definition.
    Used to define MCSUs and align vocabularies (Section 3.1); simplifying assumption about tokenization and semantics.
  • domain assumption LLM next-token distributions are the external manifestation of model knowledge and can be meaningfully combined.
    Foundation of token-level ensembling (Section 3, Appendix C); standard view in the cited literature.
  • standard math Product of token probabilities gives the probability of an MCSU.
    Equation 2 follows from the chain rule of probability.
invented entities (1)
  • Minimal Complete Semantic Unit (MCSU) no independent evidence
    purpose: Provides a cross-model aligned unit for combining distributions without tokenizer projection
    A definitional construct; the paper measures that ~90% of common English words are single tokens (Table 10), but the entity itself has no falsifiable handle outside this paper.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units." pith.science (2026). https://pith.science/paper/HKMS2EJD

@misc{pith2026250818763,
  author       = {Pith},
  title        = {Pith review of: Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HKMS2EJD}},
  note         = {Machine review of arXiv:2508.18763}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This paper investigates the enhancement of reasoning capabilities in language models through token-level multi-model collaboration. Our approach selects the optimal tokens from the next token distributions provided by multiple models to perform autoregressive reasoning. Contrary to the assumption that more models yield better results, we introduce a distribution distance-based dynamic selection strategy (DDS) to optimize the multi-model collaboration process. To address the critical challenge of vocabulary misalignment in multi-model collaboration, we propose the concept of minimal complete semantic units (MCSU), which is simple yet enables multiple language models to achieve natural alignment within the linguistic space. Experimental results across various benchmarks demonstrate the superiority of our method. The code will be available at https://github.com/Fanye12/DDS.

Figures

Figures reproduced from arXiv: 2508.18763 by Chao Hao, Ruiwen Xu, Wenzhe Niu, Xin Liu, Yanhua Huang, Zezheng Wang, Zitong Yu.

Figure 1
Figure 1. Figure 1: Motivation of the proposed DDS. For the same mathematical problem, Single LLM Qwen gave the wrong answer, token-level collaboration between Qwen and Llama produced the correct answer, whereas collaboration among Qwen, Llama, and GLM resulted in an incorrect answer. This demonstrates that multi￾model collaboration can improve answer but having more models in collaboration does not necessarily im￾prove outco… view at source ↗
Figure 2
Figure 2. Figure 2: From the implementation point of view, the proposed DDS is an adjustment based on single model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 8 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  2. [2]

    01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and...

    Pith/arXiv arXiv 2024

  3. [3]

    Rizhao Cai, Yawen Cui, Zitong Yu, Xun Lin, Changsheng Chen, and Alex Kot. 2025. https://doi.org/10.1109/TPAMI.2025.3601053 Rehearsal-free and efficient continual learning for cross-domain face anti-spoofing . IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1--18

    work page arXiv 2025

  4. [4]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, S...

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.17297 2024

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  6. [6]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    Pith/arXiv arXiv 2018

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    Pith/arXiv arXiv 2021

  8. [8]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325

    Pith/arXiv arXiv 2023

  9. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  10. [10]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833

    Pith/arXiv arXiv 2018

  11. [11]

    Yuchen Fan, Kaiyan Zhang, Heng Zhou, Yuxin Zuo, Yanxu Chen, Yu Fu, Xinwei Long, Xuekai Zhu, Che Jiang, Yuchen Zhang, Li Kang, Gang Chen, Cheng Huang, Zhizhou He, Bingning Wang, Lei Bai, Ning Ding, and Bowen Zhou. 2025. https://arxiv.org/abs/2508.10874 Ssrl: Self-search reinforcement learning . Preprint, arXiv:2508.10874

    Pith/arXiv arXiv 2025

  12. [12]

    Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations

    work page 2022

  13. [13]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346--361

    2021

  14. [14]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, ...

    Pith/arXiv arXiv 2024

  15. [15]

    Chao Hao, Shuai Wang, and Kaiwen Zhou. 2025 a . https://arxiv.org/abs/2508.04025 Uncertainty-aware gui agent: Adaptive perception through component recommendation and human-in-the-loop refinement . Preprint, arXiv:2508.04025

    Pith/arXiv arXiv 2025

  16. [16]

    Chao Hao, Zitong Yu, Xin Liu, Jun Xu, Huanjing Yue, and Jingyu Yang. 2025 b . A simple yet effective network based on vision transformer for camouflaged object and salient object detection. IEEE Transactions on Image Processing

    work page 2025

  17. [17]

    Geoffrey Hinton. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    Pith/arXiv arXiv 2015

  18. [18]

    Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523--533

    2014

  19. [19]

    Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Bing Qin, and Ting Liu. 2024. Enabling ensemble learning for heterogeneous large language models with deep parallel collaboration. arXiv preprint arXiv:2404.12715

    Pith/arXiv arXiv 2024

  20. [20]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems

    work page 2023

  21. [21]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561

    Pith/arXiv arXiv 2023

  22. [22]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    Pith/arXiv arXiv 2020

  23. [23]

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rockt \"a schel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782

    Pith/arXiv arXiv 2024

  24. [24]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  25. [25]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118

    Pith/arXiv arXiv 2023

  26. [26]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958

    Pith/arXiv arXiv 2021

  27. [27]

    Xun Lin, Ajian Liu, Zitong Yu, Rizhao Cai, Shuai Wang, Yi Yu, Jun Wan, Zhen Lei, Xiaochun Cao, and Alex Kot. 2025. https://doi.org/10.1109/TPAMI.2025.3573785 Reliable and balanced transfer learning for generalized multimodal face anti-spoofing . IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(9):7608--7625

    work page arXiv 2025

  28. [28]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146

    Pith/arXiv arXiv 2017

  29. [29]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  30. [30]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789

    Pith/arXiv arXiv 2018

  31. [31]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191

    Pith/arXiv arXiv 2021

  32. [32]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  33. [33]

    Mohaimenul Azam Khan Raiaan, Md Saddam Hossain Mukta, Kaniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eunus Ali, and Sami Azam. 2024. A review on large language models: Architectures, applications, taxonomies, open issues and challenges. IEEE Access

    work page 2024

  34. [34]

    Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. 2024. Learning to decode collaboratively with multiple language models. arXiv preprint arXiv:2403.03870

    Pith/arXiv arXiv 2024

  35. [35]

    Yunxiao Shi, Wujiang Xu, Zhang Zeqi, Xing Zi, Qiang Wu, and Min Xu. 2025. https://doi.org/10.18653/v1/2025.findings-acl.300 P ersona X : A recommendation agent-oriented user modeling framework for long behavior sequence . In Findings of the Association for Computational Linguistics: ACL 2025, pages 5764--5787, Vienna, Austria. Association for Computationa...

    doi:10.18653/v1/2025.findings-acl.300 2025

  36. [36]

    Yunxiao Shi, Xing Zi, Zijing Shi, Haimin Zhang, Qiang Wu, and Min Xu. 2024. https://doi.org/10.3233/FAIA240748 Enhancing retrieval and managing retrieval: A four-module synergy for improved quality and efficiency in rag systems . In ECAI 2024, pages 2258--2265. IOS Press

    doi:10.3233/faia240748 2024

  37. [37]

    Tal Shnitzer, Anthony Ou, M \' rian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. 2023. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789

    Pith/arXiv arXiv 2023

  38. [38]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615

    Pith/arXiv arXiv 2022

  39. [39]

    Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. 2023. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. arXiv preprint arXiv:2310.00280

    Pith/arXiv arXiv 2023

  40. [40]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937

    Pith/arXiv arXiv 2018

  41. [41]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    Pith/arXiv arXiv 2023

  42. [42]

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491

    Pith/arXiv arXiv 2024

  43. [43]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  44. [44]

    Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. 2024. Bridging the gap between different vocabularies for llm ensemble. arXiv preprint arXiv:2404.09492

    Pith/arXiv arXiv 2024

  45. [45]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    Pith/arXiv arXiv 2024

  46. [46]

    Yuxuan Yao, Han Wu, Mingyang Liu, Sichun Luo, Xiongwei Han, Jie Liu, Zhijiang Guo, and Linqi Song. 2024. Determine-then-ensemble: Necessity of top-k union for large language model ensembling. arXiv preprint arXiv:2410.03777

    Pith/arXiv arXiv 2024

  47. [47]

    Qilang Ye, Zitong Yu, Rui Shao, Yawen Cui, Xiangui Kang, Xin Liu, Philip Torr, and Xiaochun Cao. 2025. https://doi.org/10.1109/TPAMI.2025.3582389 Cat+: Investigating and enhancing audio-visual understanding in large language models . IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1--18

    work page arXiv 2025

  48. [48]

    Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuan-Jing Huang, and Xipeng Qiu. 2023. Exchange-of-thought: Enhancing large language model capabilities through cross-model communication. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15135--15153

    work page 2023

  49. [49]

    Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Chang, and Yueh-Se Li. 2024. Breaking the ceiling of the llm community by treating token generation as a classification for ensembling. arXiv preprint arXiv:2406.12585

    Pith/arXiv arXiv 2024

  50. [50]

    Heng Zhou, Hejia Geng, Xiangyuan Xue, Li Kang, Yiran Qin, Zhiyong Wang, Zhenfei Yin, and Lei Bai. 2025. Reso: A reward-driven self-organizing llm-based multi-agent system for reasoning tasks. arXiv preprint arXiv:2503.02390

    Pith/arXiv arXiv 2025

  51. [51]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  52. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.