Pith. sign in

REVIEW 3 major objections 4 minor 141 references

A relay-style handoff between heterogeneous LLMs, selecting the next model based on the current draft's diagnosed bottleneck, outperforms fixed ensembles, routing, and single-model self-refinement across four benchmarks and matches a fronti

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:53 UTC pith:QEVBPZRP

load-bearing objection Interesting framework, but the unvalidated proxy reward is the main thing standing between this and a solid publication. the 3 major comments →

arxiv 2607.29087 v1 pith:QEVBPZRP submitted 2026-07-31 cs.AI

Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration

Yanbin Fang , Xuan Wei , Wei Chen This is my paper
classification cs.AI
keywords large language modelswisdom of crowdscapability complementarityiterative refinementcomputational design sciencecontextual multi-armed bandit
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that combining several different AI language models pays off only when the combination is state-dependent: instead of fixing a workflow in advance or averaging finished answers, a coordinator inspects each draft, names its specific weakness, and hands the evolving answer to whichever model is most likely to fix that weakness. The authors instantiate this idea in WILC, built on two principles—iterative reflection-and-refinement (the full state, including the diagnosis, flows forward) and complementarity-driven selection (a dual gate decides both who works next and whether the handoff actually improved things). In experiments on code generation, math, knowledge, and visualization benchmarks, WILC beats single-model self-reflection, ensembles, and query-routing baselines, with average gains over the strongest baseline of ~3.4 points (14B crowd) and ~2.8 points (30B crowd). Under standardized pricing assumptions, WILC's per-query cost is about 7x lower than GPT-5.2 while delivering comparable average performance, because all models are self-hosted open-source models. The paper's deeper point is that collective LLM intelligence is sequential: each model corrects the previous one's bottleneck, rather than many models voting in parallel.

Core claim

The central claim is that 'relay-style complementarity'—a sequential process in which the next model is chosen specifically to undo the previous model's diagnosed bottleneck—is a real, mechanizable source of collective performance in LLM crowds. WILC does this with a burn-in round (every worker answers the query, and the best proxy-reward performer becomes the starting worker), followed by rounds in which the query is the original question, the latest answer, and the coordinator's diagnostic reflection. Worker selection is an adapted LinUCB (PCF) scoring each worker's expected reward on the embedded context; the PCG gate then retains only non-negative proxy-reward improvements, and a one-ste

What carries the argument

The load-bearing mechanism is the dual-gate complementarity verification: (1) PCF (prospective complementarity fit) uses contextual multi-armed bandits (LinUCB adapted with a cold-start phase, a weighted update that blends cold-start capability estimates with query-specific proxy feedback, and a burn-in round) to score each worker against the current context vector—the embedding of the query that now includes the previous answer and the coordinator's diagnostic reflection—and proposes the worker with the highest UCB score. (2) PCG (posterior complementarity gain) compares the proposed worker's proxy reward (R = max{1 − sγ, 0}, where s is the number of issues the coordinator lists) against th

Load-bearing premise

The load-bearing assumption is that the coordinator's count of issues—converted to proxy reward R = max{1 − sγ, 0}—is a trustworthy measure of answer quality for both choosing the next worker and deciding to stop; if that self-report diverges from true quality, WILC can hand the problem to the wrong model or terminate too early.

What would settle it

Run WILC and its strongest baseline in a setting where the coordinator's self-reported issue count is replaced by a ground-truth-based quality signal (e.g., held-out labels or a separately trained verifier), and the proxy-reward gating is left otherwise identical. If the advantage over the strongest baseline collapses or reverses, the mechanism rests on the proxy assumption rather than on complementarity itself.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Self-hosted medium-scale open-source crowds can reach frontier-level average performance at much lower per-query cost, since WILC's pricing estimate puts it near GPT-5.2 at roughly a seventh of the cost.
  • The framework's adaptive depth means simple queries terminate after one round (functioning as an evaluate-then-select mechanism), so the added inference overhead is spent only where refinement pays off.
  • Capability heterogeneity is an asset: WILC improves even when the pool contains a weak model, since relay extracts gains from all workers.
  • Because successor selection is driven by bottleneck type (e.g., logic gaps, code bugs, format violations) rather than surface task category, the mechanism keeps working on task types absent from the cold-start data—unlike query routing, which collapses on such types.
  • The two design principles (state-preserving reflection, complementarity-gated handoffs) are offered as transferable guidelines for multi-AI coordination beyond LLMs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the coordinator's self-report is not a faithful quality signal, PCF and PCG both degrade; a natural stress test is to swap in a stronger verifier or learned reward model and see whether the margin over baselines holds.
  • Two design details are worth isolating empirically: the burn-in coordinator is selected on ten queries from the target benchmark, and the cost ratio assumes roughly 1k input/500 output tokens per call; both choices could be relaxed or varied to test how much of the reported gain and cost advantage is intrinsic.
  • The paper's notion of relay-style complementarity could be measured directly—e.g., by aborting 'unproductive' handoffs and observing how often a successor actually raises proxy reward relative to random switching, which would quantify how much of the gain is selection vs. mere iteration.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes WILC, a multi-agent framework for sequential LLM collaboration. A fixed coordinator reflects on the current answer and enumerates issues; a LinUCB-based gate (PCF) selects the next worker using a context embedding and a proxy reward R = max{1−sγ, 0} derived from the issue count s; a second gate (PCG) stops iteration if the proxy reward does not increase. Experiments on HumanEval, MATH-500, MMLU, and VisEval with 14B and 30B open-source model crowds are compared with single-model, self-refinement, ensemble, and routing baselines. The paper claims WILC outperforms all baselines and approaches GPT-5.2 average performance at about 7× lower estimated per-query cost.

Significance. The design principles — state-preserving iterative refinement and complementarity-verified model selection — are timely, and the empirical scope is broad. Strengths include systematic ablations, robustness checks on coordinator identity and the exploration parameter, a transparent cost model, and a dedicated routing-method comparison. However, the central mechanism depends on an unvalidated proxy reward, and the coordinator is selected using target-benchmark queries. If the proxy were validated and the evaluation protocol tightened, the work would be a useful contribution to multi-LLM coordination and to wisdom-of-crowds theory. As it stands, the evidence does not yet support the claim that the gains arise specifically from complementarity-driven selection.

major comments (3)
  1. [Section 3.3.2, Eq. (6)] The proxy reward R = max{1−sγ, 0} is the sole feedback signal for LinUCB updates (Eq. 7), PCF selection (Eq. 5), PCG gating (Sec. 3.3.3), and termination. The paper never validates that the number s of issues enumerated by the coordinator tracks ground-truth answer quality. The random-selection ablation (Sec. 4.3) shows only that proxy-driven selection beats random selection; it does not test whether the proxy measures correctness. Consequently, the reported gains over baselines could stem from multi-round context or test-set tuning rather than complementarity-driven selection. Please add a validation analysis (e.g., correlation between R and true correctness per worker/query) or an oracle-reward ablation.
  2. [Section 3.2.1] The coordinator is selected using 10 queries from each target benchmark (HumanEval, MATH-500, MMLU, VisEval). Even though the selection uses only proxy rewards without ground-truth labels, choosing a configuration on the test distribution is a form of test-set adaptation and can inflate reported accuracy. The footnote's denial of 'data leakage' is too strong. Table 4 shows the final results are insensitive to whether the strongest or weakest model coordinates, but the strong/weak labels are based on single-execution accuracy and do not address the 10-query selection procedure itself. Please move coordinator selection to a separate validation set or include a random-coordinator condition.
  3. [Section 3.3.3, PCG gate] Complementarity gain is defined as a non-decrease in the same proxy reward that drives PCF selection and LinUCB updates. Because the proxy is the system's own internal success signal, the PCG gate only verifies consistency with the proxy, not that the selected transition improves the solution. Final accuracy is measured externally, so the framework is not fully circular, but stopping and handover decisions may be optimized to the proxy rather than to answer quality. A calibration of proxy rewards against ground-truth improvements on a held-out subset is needed before PCG can be claimed to verify complementarity.
minor comments (4)
  1. [Appendix E, Table E3] The cost comparison uses empirical average rounds computed on the test benchmarks. This is optimistic for deployment and should be stated as an upper bound or re-estimated on a validation set.
  2. [Table 5] The header says 'Single Run' but the cells report mean ± standard deviation. Please reconcile the terminology.
  3. [General] No code or data availability statement is provided. Given the complexity of WILC, releasing the implementation and evaluation scripts would materially aid reproducibility.
  4. [Section 4.4.1] The K-Means cluster analysis uses 30 clusters chosen 'for illustrative purposes'; please report stability of the ranking patterns across a range of cluster counts, or justify the choice more formally.

Circularity Check

1 steps flagged

PCG defines 'complementarity gain' as non-decrease of the coordinator's proxy reward (Eq. 6), making the internal verification self-referential; final accuracy is external, so circularity is partial.

specific steps
  1. self definitional [Section 3.3.3 (Posterior Complementarity Gain) and Eq. (6) in Section 3.3.2]
    "PCG first records the proxy reward of the previous answer, then evaluates the proxy reward for the current worker’s submission. If the current worker’s proxy reward is no lower than that of the preceding answer (i.e., R_r ≥ R_{r−1}), the worker is deemed to have provided a non-negative improvement and the iteration continues."

    The formal condition for complementarity in Section 3.3.1 requires 'a non-negative improvement to the quality of the answer,' but the PCG gate operationalizes that condition as non-decrease of R = max{1−sγ,0}, where s is the number of issues the coordinator itself enumerates. Thus 'complementarity gain' is true by construction whenever the coordinator's self-report does not get worse. The same signal drives PCF selection, burn-in ranking, OSFS overwriting, and termination, so the mechanism's internal success criterion is self-referential. It provides no independent evidence that ground-truth answer quality improved; only the separately measured final benchmark accuracy does.

full rationale

The paper's external evaluation is not circular: WILC's reported accuracies on HumanEval, MATH-500, MMLU, and VisEval are computed against ground truth, not against the proxy reward, and the random-selection ablation provides an independent comparison. However, the central internal mechanism that is claimed to 'verify' complementarity — the PCG gate — defines complementarity gain as a non-decrease in the coordinator's proxy reward (Eq. 6). That makes the framework's self-assessment of complementarity self-definitional, though not its final benchmark claim. The paper discloses that this assessment is 'by the coordinator rather than against ground truth,' so the issue is transparent, and no heavily load-bearing self-citation chain is present. Given the external final metric is independent, the overall circularity is partial, not total.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim depends on the LinUCB linear-reward assumption, on the validity of the coordinator's proxy reward, on the representativeness of LiveBench cold-start data, and on the assumption that selecting the coordinator with ten target-task queries does not leak test information. The framework introduces no new physical or abstract entities; the 'capability vector' is a learned parameter, not a new object.

free parameters (6)
  • Capability vectors θ_wk = ridge regression on LiveBench cold-start data (Eq. 4)
    Learned parameters that define worker selection; fitted to the auxiliary dataset rather than derived.
  • Exploration parameter α = 0.1
    Hand-set hyperparameter controlling the LinUCB exploration bonus; robustness tested but still a choice.
  • Issue penalty γ = 0.1
    Hand-set penalty per coordinator-identified issue in the proxy reward (Eq. 6).
  • Maximum rounds r_max = 6
    Hand-set cap on iterative refinement rounds.
  • PCA dimension d = 32
    Reduction dimension for query embeddings, chosen by the authors.
  • Coordinator selection batch = 10
    Number of target-task queries used to pick the permanent coordinator (Section 3.2.1).
axioms (5)
  • domain assumption Expected reward for worker w_k is linear in query embedding x: E[R|x] = x^T θ_wk
    Standard LinUCB assumption (Eq. 1/10); the paper does not test linearity.
  • ad hoc to paper The number of issues enumerated by the coordinator's reflection is inversely related to answer quality sufficiently for selection
    Proxy reward Eq. (6); no evidence of monotonic relationship with ground truth.
  • domain assumption Coordinator reflection reliably identifies the true bottleneck in the predecessor's output
    Required for relay-style complementarity; Section 3.2.2.
  • domain assumption LiveBench cold-start data is representative of the four target benchmarks and bottleneck-contextualized queries generalize
    Section 4.1.4; the paper shows only indirect evidence in Fig. 8.
  • ad hoc to paper Using 10 target-task queries to choose the coordinator does not leak test information
    Footnote 1; contradicts standard ML practice requiring a validation set.

pith-pipeline@v1.3.0-daily-deepseek · 35659 in / 17121 out tokens · 169559 ms · 2026-08-03T13:53:48.800902+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly deployed in enterprise settings, yet individual models remain bounded by model-specific capability limitations. These heterogeneous boundaries pose a deployment challenge, but also create an opportunity: strategically coordinating multiple LLMs may unlock collective intelligence exceeding any single model. Existing approaches fix how models are combined in advance, overlooking the dynamic, state-dependent role of complementarity in complex problem solving. Drawing on the wisdom-of-crowds paradigm, we reconceptualize collective LLM intelligence as relay-style complementarity: a sequential process in which each successor model is selected to address the specific bottleneck identified in its predecessor's output. To operationalize this, we propose WILC (Wisdom Integration of LLM Crowds), a framework grounded in two design principles. First, iterative reflection-and-refinement establishes a state-preserving workflow through which models diagnose and refine prior outputs. Second, complementarity-driven model selection governs transitions via a dual-gate mechanism: prospective complementarity fit (PCF) identifies the worker most suited to the current bottleneck, while posterior complementarity gain (PCG) evaluates whether the selected transition improves the evolving solution. Experiments across four diverse benchmarks show that WILC outperforms existing approaches, including single-model self-refinement, ensemble methods, and query-routing methods. Under standardized pricing assumptions, WILC matches the average benchmark performance of GPT-5.2 at roughly 7 times lower estimated per-query cost, while facilitating data sovereignty through self-hosted deployment. This study extends wisdom-of-crowds theory from static aggregation to sequential AI complementarity and provides transferable design principles for multi-AI coordination.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

141 extracted references · 7 linked inside Pith

  1. [1]

    Integrating generative AI into enterprise platforms: Insights from salesforce

    Kazem Haki, Dorsa Safaei, Adolfo Magan, and Martin Griffiths. Integrating generative AI into enterprise platforms: Insights from salesforce. Information Systems Journal, 2025

    2025

  2. [2]

    Industrial applications of large language models

    Mubashar Raza, Zarmina Jahangir, Muhammad Bilal Riaz, Muhammad Jasim Saeed, and Muhammad Awais Sattar. Industrial applications of large language models. Scientific Reports, 15 0 (1): 0 13755, 2025

    2025

  3. [3]

    Large language model routing with benchmark datasets

    Tal Shnitzer, Anthony Ou, Mirian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets. In Annual Conference on Neural Information Processing Systems, 2023

    2023

  4. [4]

    Routing to the expert: Efficient reward-guided ensemble of large language models

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1964--1974, 2024

    2024

  5. [5]

    Enterprise generative AI : 10+ use cases & best practices, 2024

    AIMultiple. Enterprise generative AI : 10+ use cases & best practices, 2024. URL https://research.aimultiple.com/enterprise-generative-ai/. Accessed: 2025-11-29

    2024

  6. [6]

    The Wisdom of Crowds

    James Surowiecki. The Wisdom of Crowds. Anchor, 2005

    2005

  7. [7]

    Combining crowd and machine intelligence to detect false news on social media

    Xuan Wei, Zhu Zhang, Mingyue Zhang, Weiyun Chen, and Daniel Dajun Zeng. Combining crowd and machine intelligence to detect false news on social media. MIS Quarterly, 46 0 (2): 0 977--1008, 2022 a

    2022

  8. [8]

    Distilling the wisdom of crowds: Prediction markets vs

    Pavel Atanasov, Phillip Rescober, Eric Stone, Samuel A Swift, Emile Servan-Schreiber, Philip Tetlock, Lyle Ungar, and Barbara Mellers. Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Science, 63 0 (3): 0 691--706, 2017

    2017

  9. [9]

    Human-algorithm collaborative truth inference in crowdsourcing

    Xuan Wei, Mingyue Zhang, Qingpeng Zhang, Zhi Li, and Daniel Dajun Zeng. Human-algorithm collaborative truth inference in crowdsourcing. INFORMS Journal on Computing, 2025

    2025

  10. [10]

    The crowd classification problem: Social dynamics of binary-choice accuracy

    Joshua Aaron Becker, Douglas Guilbeault, and Edward Bishop Smith. The crowd classification problem: Social dynamics of binary-choice accuracy. Management Science, 68 0 (5): 0 3949--3965, 2022

    2022

  11. [11]

    Model-based wisdom of the crowd for sequential decision-making tasks

    Bobby Thomas, Jeff Coon, Holly A Westfall, and Michael D Lee. Model-based wisdom of the crowd for sequential decision-making tasks. Cognitive Science, 45 0 (7): 0 e13011, 2021

    2021

  12. [12]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 0 24824--24837, 2022 b

    2022

  13. [13]

    Self-Refine : Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine : Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36: 0 46534--46594, 2023

    2023

  14. [14]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 8634--8652, 2023

    2023

  15. [15]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022

    2022

  16. [16]

    LLM-Blender : Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender : Ensembling large language models with pairwise ranking and generative fusion. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023

    2023

  17. [17]

    A survey on LLM -based multi-agent systems: Workflow , infrastructure, and challenges

    Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey on LLM -based multi-agent systems: Workflow , infrastructure, and challenges. Vicinagearth, 1 0 (1): 0 9, 2024

    2024

  18. [18]

    Large language model based multi-agents: A survey of progress and challenges

    T Guo, X Chen, Y Wang, R Chang, S Pei, NV Chawla, O Wiest, and X Zhang. Large language model based multi-agents: A survey of progress and challenges. In 33rd International Joint Conference on Artificial Intelligence (IJCAI 2024). IJCAI; Cornell arxiv, 2024

    2024

  19. [19]

    English dictionary, 2025

    Cambridge Dictionary . English dictionary, 2025. URL https://dictionary.cambridge.org/dictionary/english/complementarity. Retrieved November 24, 2025

    2025

  20. [20]

    Learning from crowdsourced multi-labeling: A variational bayesian approach

    Junming Yin, Jerry Luo, and Susan A Brown. Learning from crowdsourced multi-labeling: A variational bayesian approach. Information Systems Research, 32 0 (3): 0 752--773, 2021

    2021

  21. [21]

    u hl, Michael V \

    Patrick Hemmer, Max Schemmer, Niklas K \"u hl, Michael V \"o ssing, and Gerhard Satzger. Complementarity in human- AI collaboration: Concept , sources, and evidence. European Journal of Information Systems, pages 1--24, 2025

    2025

  22. [22]

    Does the whole exceed its parts? The effect of AI explanations on complementary team performance

    Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1--16, 2021

    2021

  23. [23]

    Design science in information systems research

    Alan R Hevner, Salvatore T March, Jinsoo Park, and Sudha Ram. Design science in information systems research. MIS quarterly, pages 75--105, 2004

    2004

  24. [24]

    Pathways for design research on artificial intelligence

    Ahmed Abbasi, Jeffrey Parsons, Gautam Pant, Olivia R Liu Sheng, and Suprateek Sarker. Pathways for design research on artificial intelligence. Information Systems Research, 35 0 (2): 0 441--459, 2024

    2024

  25. [25]

    Computational design science: A critical information systems research area contributing to artificial intelligence and data science

    Xiao Fang, Paul J Hu, Michael Chau, and Hsinchun Chen. Computational design science: A critical information systems research area contributing to artificial intelligence and data science. Available at SSRN 5455094, 2025

    2025

  26. [26]

    When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs . Transactions of the Association for Computational Linguistics, 12: 0 1417--1440, 2024

    2024

  27. [27]

    What makes reasoning invalid: Echo reflection mitigation for large language models

    Chen He, Xun Jiang, Lei Wang, Hao Yang, Chong Peng, Peng Yan, Fumin Shen, and Xing Xu. What makes reasoning invalid: Echo reflection mitigation for large language models. arXiv preprint arXiv:2511.06380, 2025

    arXiv 2025

  28. [28]

    A contextual-bandit approach to personalized news article recommendation

    Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, pages 661--670, 2010

    2010

  29. [29]

    u gener, J \

    Andreas F \"u gener, J \"o rn Grahl, Alok Gupta, and Wolfgang Ketter. Cognitive challenges in human--artificial intelligence collaboration: Investigating the path toward productive delegation. Information Systems Research, 33 0 (2): 0 678--696, 2022

    2022

  30. [30]

    Human-algorithm collaboration: Achieving complementarity and avoiding unfairness

    Kate Donahue, Alexandra Chouldechova, and Krishnaram Kenthapadi. Human-algorithm collaboration: Achieving complementarity and avoiding unfairness. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1639--1656, 2022

    2022

  31. [31]

    Bayesian modeling of human-- AI complementarity

    Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human-- AI complementarity. Proceedings of the National Academy of Sciences, 119 0 (11): 0 e2111547119, 2022

    2022

  32. [32]

    Ensemble methods in machine learning

    Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1--15. Springer, 2000

    2000

  33. [33]

    Ensemble learning: A survey

    Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery, 8 0 (4): 0 e1249, 2018

    2018

  34. [34]

    Complexity-based prompting for multi-step reasoning

    Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In 11th International Conference on Learning Representations, ICLR 2023, 2023

    2023

  35. [35]

    Making language models better reasoners with step-aware verifier

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5315--5333, 2023

    2023

  36. [36]

    Just ask one more time! Self -agreement improves reasoning of language models in (almost) all scenarios

    Lei Lin, Jiayi Fu, Pengli Liu, Qingyang Li, Yan Gong, Junchen Wan, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, and Kun Gai. Just ask one more time! Self -agreement improves reasoning of language models in (almost) all scenarios. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3829--3852, 2024 a

    2024

  37. [37]

    Examining inter-consistency of large language models collaboration: An in-depth analysis via debate

    Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7572--7590, 2023

    2023

  38. [38]

    MetaGPT : Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. MetaGPT : Meta programming for a multi-agent collaborative framework. In 12th International Conference on Learning Representations, ICLR 2024, 2024

    2024

  39. [39]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174--15186, 2024

    2024

  40. [40]

    ChatGPT research group for optimizing the crystallinity of MOFs and COFs

    Zhiling Zheng, Oufan Zhang, Ha L Nguyen, Nakul Rampal, Ali H Alawadhi, Zichao Rong, Teresa Head-Gordon, Christian Borgs, Jennifer T Chayes, and Omar M Yaghi. ChatGPT research group for optimizing the crystallinity of MOFs and COFs . ACS Central Science, 9 0 (11): 0 2161--2170, 2023

    2023

  41. [41]

    MedAgents : Large language models as collaborators for zero-shot medical reasoning

    Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. MedAgents : Large language models as collaborators for zero-shot medical reasoning. In ICLR 2024 Workshop on Large Language Model ( LLM ) Agents , 2024

    2024

  42. [42]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023

    2023

  43. [43]

    Social simulacra: Creating populated prototypes for social computing systems

    Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1--18, 2022

    2022

  44. [45]

    Some aspects of the sequential design of experiments

    Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58 0 (5): 0 527--535, 1952

    1952

  45. [46]

    Reinforcement learning: An introduction

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction . MIT press Cambridge, 1998

    1998

  46. [47]

    Finite-time analysis of the multiarmed bandit problem

    Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47 0 (2): 0 235--256, 2002

    2002

  47. [48]

    Spearman’s rank correlation coefficient

    Philip Sedgwick. Spearman’s rank correlation coefficient. BMJ, 349, 2014

    2014

  48. [51]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021 b

    2021

  49. [52]

    VisEval : A benchmark for data visualization in the era of large language models

    Nan Chen, Yuge Zhang, Jiahang Xu, Kan Ren, and Yuqing Yang. VisEval : A benchmark for data visualization in the era of large language models. IEEE Transactions on Visualization and Computer Graphics, 2024 a

    2024

  50. [53]

    ReAct : Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct : Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022

    2022

  51. [54]

    Universal self-consistency for large language models

    Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language models. In ICML Workshop on In-Context Learning, 2024 b

    2024

  52. [55]

    AWQ : Activation-aware weight quantization for on-device LLM compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ : Activation-aware weight quantization for on-device LLM compression and acceleration. Proceedings of machine learning and systems, 6: 0 87--100, 2024 b

    2024

  53. [56]

    LiveBench : A challenging, contamination-free LLM benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. LiveBench : A challenging, contamination-free LLM benchmark. arXiv preprint arXiv:2406.19314, 2024

    Pith/arXiv arXiv 2024

  54. [57]

    Nomic embed: Training a reproducible long context text embedder

    Zach Nussbaum, John Xavier Morris, Andriy Mulyar, and Brandon Duderstadt. Nomic embed: Training a reproducible long context text embedder. Transactions on Machine Learning Research, 2025

    2025

  55. [58]

    LLMRouter : An open-source library for LLM routing

    Tao Feng, Haozhen Zhang, and Jiaxuan You. LLMRouter : An open-source library for LLM routing. https://github.com/ulab-uiuc/LLMRouter, 2024

    2024

  56. [59]

    Router-R1 : Teaching LLMs multi-round routing and aggregation via reinforcement learning

    Haozhen Zhang, Tao Feng, and Jiaxuan You. Router-R1 : Teaching LLMs multi-round routing and aggregation via reinforcement learning. arXiv preprint arXiv:2506.09033, 2025

    arXiv 2025

  57. [60]

    RouteLLM : Learning to route LLMs from preference data

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM : Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  58. [61]

    GraphRouter : A graph-based router for LLM selections

    Tao Feng, Yanzhen Shen, and Jiaxuan You. GraphRouter : A graph-based router for LLM selections. In International Conference on Learning Representations, volume 2025, pages 26186--26203, 2025

    2025

  59. [62]

    ICML Workshop on In-Context Learning , year=

    Universal Self-Consistency for Large Language Models , author=. ICML Workshop on In-Context Learning , year=

  60. [63]

    Transactions on Machine Learning Research , year=

    Nomic Embed: Training a Reproducible Long Context Text Embedder , author=. Transactions on Machine Learning Research , year=

  61. [64]

    Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song , journal=

  62. [65]

    Journal of Operations Research , volume =

    Smith, John , title =. Journal of Operations Research , volume =

  63. [66]

    INFORMS Mathematics of Operations Research , volume =

    Jones, Sarah , title =. INFORMS Mathematics of Operations Research , volume =

  64. [67]

    Brown, David , title =

  65. [68]

    Surowiecki, James , title =

  66. [69]

    MIS Quarterly , volume=

    Combining crowd and machine intelligence to detect false news on social media , author=. MIS Quarterly , volume=. 2022 , publisher=

    2022

  67. [70]

    Integrating Generative

    Haki, Kazem and Safaei, Dorsa and Magan, Adolfo and Griffiths, Martin , journal=. Integrating Generative. 2025 , publisher=

    2025

  68. [71]

    Scientific Reports , volume=

    Industrial applications of large language models , author=. Scientific Reports , volume=. 2025 , publisher=

    2025

  69. [72]

    Annual Conference on Neural Information Processing Systems , year=

    Large Language Model Routing with Benchmark Datasets , author=. Annual Conference on Neural Information Processing Systems , year=

  70. [73]

    Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen , booktitle=

  71. [74]

    Routing to the Expert:

    Lu, Keming and Yuan, Hongyi and Lin, Runji and Lin, Junyang and Yuan, Zheng and Zhou, Chang and Zhou, Jingren , booktitle=. Routing to the Expert:

  72. [75]

    arXiv preprint arXiv:2310.01798 , year=

    Large language models cannot self-correct reasoning yet , author=. arXiv preprint arXiv:2310.01798 , year=

    Pith/arXiv arXiv

  73. [76]

    When can

    Kamoi, Ryo and Zhang, Yusen and Zhang, Nan and Han, Jiawei and Zhang, Rui , journal=. When can. 2024 , publisher=

    2024

  74. [77]

    Distilling the wisdom of crowds:

    Atanasov, Pavel and Rescober, Phillip and Stone, Eric and Swift, Samuel A and Servan-Schreiber, Emile and Tetlock, Philip and Ungar, Lyle and Mellers, Barbara , journal=. Distilling the wisdom of crowds:. 2017 , publisher=

    2017

  75. [78]

    wisdom of crowds

    Studying the “wisdom of crowds” at scale , author=. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume=

  76. [79]

    The crowd classification problem:

    Becker, Joshua Aaron and Guilbeault, Douglas and Smith, Edward Bishop , journal=. The crowd classification problem:. 2022 , publisher=

    2022

  77. [80]

    Cognitive Science , volume=

    Model-based wisdom of the crowd for sequential decision-making tasks , author=. Cognitive Science , volume=. 2021 , publisher=

    2021

  78. [81]

    Some aspects of the sequential design of experiments , journal=

    Robbins, Herbert , year=. Some aspects of the sequential design of experiments , journal=

  79. [82]

    Reinforcement learning:

    Sutton, Richard S and Barto, Andrew G and others , year=. Reinforcement learning:

  80. [83]

    Machine Learning , volume=

    Finite-time analysis of the multiarmed bandit problem , author=. Machine Learning , volume=. 2002 , publisher=

    2002

Showing first 80 references.