Pith. sign in

REVIEW 4 major objections 4 minor 50 references

A lightweight learned weighting of black-box model outputs lifts accuracy, fairness, and robustness across independent foundation models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:25 UTC pith:DQVKR3MB

load-bearing objection A credible empirical aggregation study with broad data, but the theoretical framing leans on independence assumptions that the paper itself concedes are rarely satisfied, and the 'consistently' claims are stronger than the tables show. the 4 major comments →

arxiv 2602.13792 v3 pith:DQVKR3MB submitted 2026-02-14 cs.AI cs.CL

StackingNet: Collective Inference Across Independent AI Foundation Models

Siyang Li , Chenhao Liu , Dongrui Wu , Zhigang Zeng , Lieyun Ding This is my paper
classification cs.AI cs.CL
keywords ensemble learningfoundation modelsblack-box aggregationstackingcollective intelligencereliability rankingadversarial pruningbias mitigation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

StackingNet is a meta-ensemble framework that aggregates only the output predictions of independently built foundation models at inference time. The paper argues that a simple weighted sum of these black-box outputs, trained on as little as 1% of labeled data or even none, improves accuracy over individual models and classic ensembles, reduces demographic bias in visual attribute ratings, ranks model reliability, and detects or prunes compromised models. The theoretical engine is ensemble variance reduction: under conditional independence of base-model errors, uniform combination divides squared error by the number of models, and optimal weights are inversely proportional to per-model error variance. If correct, model diversity becomes a usable resource rather than a coordination problem, with direct consequences for benchmarking and for platforms that route queries across API models.

Core claim

On its own terms, the paper establishes that collective inference among black-box foundation models can be reduced to constrained weighted aggregation of their predictions. For regression, the meta-model is a weighted sum of base predictions plus a global bias term, with nonnegative weights; for classification, it is a weighted sum of one-hot votes whose weights are learned through a cross-entropy objective plus an unsupervised consensus term. The same learned weights serve as reliability scores, enabling unsupervised ranking and adversarial pruning. The paper reports that this method outperforms individual models and classical ensemble/crowdsourcing combination methods across research-paper

What carries the argument

The central object is a learnable convex-combination meta-learner over base-model outputs: per-model weights w_j ≥ 0 (with sum-to-one regularization, plus a single bias term in regression) applied to the vector of black-box predictions. Classical ensemble theory motivates the design: the error-covariance matrix of base predictions yields inverse-variance optimal weights, the spectral decomposition of classifier output covariance gives unsupervised reliability estimates, and these estimates initialize or regularize the learned weights. The non-negativity constraints and the bias term are what turn this one-line architecture into an accuracy, fairness, ranking, and pruning instrument.

Load-bearing premise

The load-bearing premise is that different foundation models' prediction errors are independent (or at least uncorrelated) given the true answer; the paper itself concedes this rarely holds exactly, and if real models share training data or architecture families, their correlated errors weaken the variance-reduction and reliability-ranking guarantees.

What would settle it

On a labeled held-out set, measure the pairwise correlations of signed prediction errors among a typical pool of foundation models. If the average off-diagonal correlation is substantial and the learned weighted combination's squared error is close to the mean individual squared error rather than well below it, the variance-reduction mechanism claimed for StackingNet is falsified. A second test: if injecting a model that outputs pure random noise ever yields an above-median learned weight on a non-trivial fraction of runs, the ranking and pruning utility fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • StackingNet beats the best single base model in the reported regression tasks and most classification tasks, even when all base models are already strong.
  • A few tens of labeled examples, roughly 1% of the pool, suffice to align the ensemble with human consensus; additional labels give only marginal gains.
  • The learned weights can be read as reliability estimates: they correlate with true test accuracy, support ranking, and identify low-weight models for pruning, cutting query costs with little accuracy loss.
  • The same mechanism detects randomly predicting and label-flipping base models, assigning them near-zero weights in most settings.
  • Under the paper's assumptions, adding competent base models monotonically improves voting accuracy, whereas any single model's performance is bounded by its own bias and variance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because real foundation models share training data and design choices, the practical gain will likely fall short of the theoretical 1/M error reduction; the missing quantity to measure is the average off-diagonal correlation of model errors, which the paper does not report.
  • The reliability-ranking result points toward a benchmark protocol in which test labels are never public: the aggregation weights themselves become the evaluation, blunting dataset contamination; the paper gestures at this but does not formalize it.
  • A natural extension is online adaptation: re-estimating weights as the model pool changes, or adding pairwise interaction terms for correlated models, since the current architecture assumes a fixed and conditionally independent pool.
  • The pruning results suggest a concrete cost-performance tradeoff: a deployment could drop low-weight models until a specified query budget is reached, and the paper's data would allow a direct test of whether budgeted pruning preserves accuracy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes StackingNet, a lightweight meta-ensemble that learns convex weights over black-box model outputs—plus a bias term for regression—and claims that this simple aggregation improves accuracy, reduces bias and group-wise disparities, ranks model reliability, and enables pruning of compromised models, all without access to internal parameters or training data. The method is evaluated on four research-paper rating datasets (regression), thirteen facial-attribute rating tasks (regression), and eight HELM classification datasets, with comparisons to classic and recent crowdsourcing/ensemble baselines. Theoretical support is drawn from classical ensemble-learning results, with the architectural choices (non-negativity, bias, self-consensus loss) motivated by those results.

Significance. If the claims hold, StackingNet would be a practical, deployable middleware for coordinating API-accessed foundation models, requiring only a handful of labeled examples or none at all. The empirical breadth is substantial: 4 rating datasets with 6 LLMs, 13 attributes with 10 VLMs, and 8 classification datasets with 10 LLMs, with code and prediction data publicly promised. The paper also honestly attempts to discuss the independence assumptions underlying the theory. However, the central theoretical guarantees rely on assumptions that the paper itself concedes rarely hold, and the empirical record contains counterexamples to the 'consistent improvement' claim. The human-reviewer comparison also appears to use a consensus baseline that may include the very reviewers being evaluated. These issues are significant enough to require revision.

major comments (4)
  1. [Methods, Assumptions 2-3 (Eqs. 2-4); Lemma 1 (Eq. 9), Lemma 2 (Eqs. 14-15), Theorem 5 (Eq. 24)] The conclusion that combination reduces error by factor M, that optimal weights are inverse-variance, and that the principal eigenvector tracks balanced accuracy all rest on conditional independence (Assumption 2) and zero-mean uncorrelated errors (Assumption 3). The paper states in Methods that Assumption 3 'rarely holds exactly in practice,' and the Discussion concedes that overlapping training data limit aggregation benefits. Yet no empirical covariance diagnostics are reported: the off-diagonal entries of C in Eq. (12) are never measured, and no conditional-dependence statistics are given. I recommend reporting the empirical error covariance (or mean pairwise correlation) on held-out data, comparing the learned weights with oracle inverse-variance weights computed from full labels, and showing how the Lemma 1/2 error-reduction factor degrades as correlations increase. Without this, t
  2. [Abstract; Table 1] The abstract claims that StackingNet 'consistently improves accuracy' and 'outperforms individual models and classic ensembles.' This is not supported by Table 1 for the unsupervised U-StackingNet: it is not the best on IMDB (96.97 vs. 97.59 for Logistic Regression), LSAT (21.38 vs. 22.62), MMLU (54.42 vs. 57.73/58.80), or RAFT (88.33 vs. 89.18), and on LSAT all methods are near chance. The claim should be qualified to the supervised variant, to 'in most cases,' or to comparisons against individual base models, and the LSAT failure should be acknowledged. The current wording overstates the consistency of the results.
  3. [Results and Discussion, Figure 2A-D] The comparison of individual human reviewers with LLMs uses consensus ground-truth scores obtained by aggregating multiple human ratings, and then individual human MAE is computed relative to that consensus. If each human reviewer's own rating is included in the consensus, human error is systematically underestimated relative to LLM error, because the LLM ratings are compared against a consensus to which they did not contribute. This is a load-bearing issue for the claim that 'StackingNet reached accuracy comparable to or higher than the human average.' Please clarify whether a leave-one-out consensus was used; if not, re-analyze with each human excluded from the consensus used as the target, or report both versions. Otherwise the human-baseline comparison is biased.
  4. [StackingNet for classification combination, Eq. (28); Figure 4D-F] The same learned weight vector w is used for aggregation, reliability ranking, and pruning. This is partly circular: the unsupervised loss in Eq. (28) minimizes weighted disagreement with the ensemble's own consensus, so a low weight can indicate redundancy or consensus-divergence rather than low ground-truth accuracy. The paper's own Figure 4F shows that pruning the lowest-weighted models does not always improve performance, which undercuts the claim that StackingNet 'identifies or prunes models that degrade performance.' I recommend validating the learned weights against per-model accuracy on held-out labels, and reporting pruning results against a random-pruning baseline or an oracle-based pruning baseline. This would sharpen the distinction between utility for aggregation and utility for reliability assessment.
minor comments (4)
  1. [StackingNet for regression combination, value range constraints] The text refers to 'Ablation studies for the bias term and the constraints are shown in Supporting Information Table ??' with an unresolved placeholder. Extended Data Table 1 appears to contain this ablation and should be cited explicitly.
  2. [Methods, classification architecture] The paper says StackingNet 'unifies regression and classification within a single framework,' but the regression model has a bias term while the classification model does not, and the loss functions differ. Please clarify the sense in which the framework is unified beyond sharing the general weighted-combination idea.
  3. [Table 1 and throughout] The dataset name 'LSA T' appears with a spurious space; this should be 'LSAT.' Also, in Table 1, the 'Rank' column uses '1' twice for U-StackingNet and S-StackingNet; clarify whether this is an average rank and how ties are resolved.
  4. [Related work and reference [49]] Reference [49] is prior work by the same authors on 'Black-box test-time ensemble,' and in the method comparison the authors state that the spectral meta-learner SML uses a one-vs-rest extension from [49]. The manuscript should explicitly state the incremental contribution of StackingNet relative to [49], especially for the unsupervised classification component, so that novelty is transparent.

Circularity Check

0 steps flagged

No substantive circularity: theoretical results are cited from external literature, and empirical claims are tested against held-out labels.

full rationale

The paper's derivation chain does not reduce to its own inputs. The main theoretical statements (Theorem 1, Lemma 1, Theorem 2, Lemma 2, Theorems 3–5) are credited to external prior work (Breiman 1996; Perrone & Cooper 1995; Lam & Suen 1997; Parisi et al. 2014), and the paper does not claim to prove them. StackingNet's supervised weights are fit by empirical risk minimization on labeled examples, and its unsupervised variant is justified by an external spectral theorem; the ranking and pruning results are evaluated against held-out ground-truth performance, so they are empirical validations rather than definitional equivalences. The paper explicitly flags that Assumption 3 'rarely holds exactly in practice' and that overlapping training data limit aggregation benefits; this is a candid limitation, not a circular step. The only self-citation (ref. 49) appears in the description of the SML baseline's multi-class extension and is not load-bearing for StackingNet's central claims. Consistent with the instruction not to manufacture circularity, no circular step is exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

No new physical or ontological entities are posited; StackingNet is a methodological aggregation architecture. The main extra assumptions are the statistical independence/uncorrelation conditions and the use of human-consensus ground truth, both of which the paper itself flags as imperfect.

free parameters (4)
  • StackingNet regression weights w = learned per dataset; not tabulated
    Per-model weights fitted by the MSE objective; central to aggregation and ranking.
  • StackingNet regression bias b = learned non-negative scalar; not tabulated
    Calibrates the aggregate prediction range; part of the architecture.
  • StackingNet classification weights w = learned per dataset; not tabulated
    Reliability weights fitted by cross-entropy plus unsupervised and regularization losses.
  • Loss coefficients λ1, λ2 = not reported; sensitivity shown in Extended Data Fig. 5
    Hand-chosen to balance supervised, unsupervised, and normalization losses.
axioms (6)
  • domain assumption Assumption 1: test samples are i.i.d. from distribution D (Eq. 1)
    Standard statistical assumption; likely approximately true for curated benchmarks.
  • domain assumption Assumption 2: base learner predictions are conditionally independent given the true label (Eq. 2)
    Load-bearing for the error-reduction theorems; the paper admits it is strong and 'rarely holds exactly in practice.'
  • domain assumption Assumption 3: regression errors are zero-mean and uncorrelated (Eqs. 3-4)
    Used for the factor-1/M error reduction (Lemma 1) and closed-form weights (Lemma 2); acknowledged as rarely holding.
  • domain assumption Assumption 4: a majority of classifiers outperform random guessing (Eq. 5)
    Mild; used for sign disambiguation in the spectral reliability estimation.
  • standard math Prior ensemble theorems (Breiman 1996; Perrone & Cooper 1995; Lam & Suen 1997; Shapley & Grofman 1984; Parisi et al. 2014)
    Restated and used as the theoretical foundation; not derived in this paper.
  • domain assumption The consensus of multiple human ratings is the ground truth for paper quality
    Used to compute MAE for both human and LLM reviewers; human error is measured against an aggregate that includes the same human raters.

pith-pipeline@v1.3.0-alltime-deepseek · 21357 in / 17253 out tokens · 160752 ms · 2026-08-02T23:25:48.710975+00:00 · methodology

0 comments
read the original abstract

Artificial intelligence built on large foundation models has transformed language understanding, computer vision, and reasoning, yet these systems remain isolated and cannot readily share their capabilities. Coordinating the complementary strengths of independently developed, black-box foundation models is essential for trustworthy intelligent systems, yet no established method exists. Here we show that such coordination can be achieved through a meta-ensemble framework termed StackingNet, which aggregates the output predictions of independent models at inference. StackingNet improves accuracy, reduces individual-model error and group-wise disparities, ranks model reliability, and identifies or prunes models that degrade performance, all without access to internal parameters or training data. Across language comprehension, visual attribute estimation, and academic paper rating, it consistently outperforms individual models and classic ensembles, with gains that persist when the base models are uniformly strong. These gains stem from variance reduction and consensus alignment among independent models rather than from any emergent group cognition, and they widen as the model pool grows more diverse. By turning model diversity from a source of inconsistency into a resource for cooperation, StackingNet offers a practical path toward coordinated artificial intelligence, where progress emerges not only from larger single models but from principled cooperation among many specialized ones.

Figures

Figures reproduced from arXiv: 2602.13792 by Chenhao Liu, Dongrui Wu, Lieyun Ding, Siyang Li, Zhigang Zeng.

Figure 1
Figure 1. Figure 1: Collective inference of independent intelligent systems through StackingNet. a, Relationship between collective complexity and cognitive complexity across biological and artificial systems. b, Aggregated inference from multiple independent foundation models across diverse task types. c, StackingNet architecture and its learnable parameters. Each base model is treated as an independent black-box system whos… view at source ↗
Figure 2
Figure 2. Figure 2: Research paper rating error by individual human reviewers, individual LLMs, and collective inference of multiple LLMs. a-d, Mean absolute error (MAE; lower values indicate better performance) across four datasets: ICLR2025, ICLR2024, NeurIPS2024 and NeurIPS2023. Errors for individual humans are computed relative to consensus scores obtained by aggregating multiple human reviewers. StackingNet was trained i… view at source ↗
Figure 3
Figure 3. Figure 3: Facial attribute ratings by VLMs and StackingNet combination on the Chicago Face Database. a, Heatmaps of MAE of base models across thirteen attributes, stratified by two gender groups and six racial groups. b, Distribution of signed prediction errors for the base models and StackingNet across thirteen attributes in the normalized label space. Predictions were clipped, and the label space was min-max norma… view at source ↗
Figure 4
Figure 4. Figure 4: Performance of StackingNet on classification tasks for ranking and pruning. a, Ground-truth balanced classification accuracy (BCA) of base models. b, Estimated BCA by StackingNet using few-shot annotations; learned combination weights are shown. c, Kendall’s τ rank correlation with ground-truth rankings under few-shot estimation and StackingNet in supervised (S-StackingNet) or unsupervised (U-StackingNet) … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 7 linked inside Pith

  1. [1]

    Zhao, W . X.et al. A survey of large language models. Preprint at https://arxiv.org/abs/2303.18223v16 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Zhang, J., Huang, J., Jin, S. & Lu, S. Vision-language models for vision tasks: A survey. IEEE Trans. Pattern Analysis Mach. Intell. 46, 5625–5644 (2024)

    2024

  3. [3]

    Hendrycks, D. et al. Measuring massive multitask language understanding. In Int’l Conf. Learning Representations (2021)

    2021

  4. [4]

    Antol, S. et al. VQA: Visual question answering. In Proc. IEEE Int’l Conf. Computer Vision (2015)

    2015

  5. [5]

    Chen, M. et al. Evaluating large language models trained on code. Preprint at https://arxiv.org/abs/2107.03374 (2021)

    Pith/arXiv arXiv 2021

  6. [6]

    S., Wang, Y

    Liu, J., Xia, C. S., Wang, Y . & Zhang, L. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Proc. Advances in Neural Information Processing Systems (2023)

    2023

  7. [7]

    Milano, S., McGrane, J. A. & Leonelli, S. Large language mode ls challenge the future of higher education. Nat. Mach. Intell. 5, 333–334 (2023)

    2023

  8. [8]

    Tan, Z. et al. Large language models for data annotation and synthesis: A s urvey. Preprint at https://arxiv.org/abs/2402.13446 (2024)

    Pith/arXiv arXiv 2024

  9. [9]

    & Lin, J

    Zhuang, Z., Chen, J., Xu, H., Jiang, Y . & Lin, J. Large languag e models for automated scholarly paper review: A survey. Inf. Fusion 103332 (2025)

    2025

  10. [10]

    Can large language models provide useful feedback on resear ch papers? a large-scale empirical analysis

    Liang, W .et al. Can large language models provide useful feedback on resear ch papers? a large-scale empirical analysis. New Engl. J. Medicine AI 1, AIoa2400196 (2024)

    2024

  11. [11]

    Zhao, H. et al. Explainability for large language models: A survey. ACM Trans. Intell. Syst. T echnol.15, 1–38 (2024)

    2024

  12. [12]

    Advances, challenges and opportunities in creating data fo r trustworthy AI

    Liang, W .et al. Advances, challenges and opportunities in creating data fo r trustworthy AI. Nat. Mach. Intell. 4, 669–677 (2022)

    2022

  13. [13]

    & V ayena, E

    Jobin, A., Ienca, M. & V ayena, E. The global landscape of AI et hics guidelines. Nat. Mach. Intell. 1, 389–399 (2019)

    2019

  14. [14]

    Gallegos, I. O. et al. Bias and fairness in large language models: A survey. Comput. Linguist. 50, 1097–1179 (2024)

    2024

  15. [15]

    P ., Wu, C., Morency, L.-P

    Liang, P . P ., Wu, C., Morency, L.-P . & Salakhutdinov, R. Towards understanding and mitigating social biases in language models. In Proc. Int’l Conf. Machine Learning (2021)

    2021

  16. [16]

    Ouyang, L. et al. Training language models to follow instructions with human feedback. In Proc. Advances in Neural Information Processing Systems (2022)

    2022

  17. [17]

    Ji, J. et al. AI alignment: A comprehensive survey. Preprint at https://arxiv.org/abs/2310.19852 (2023)

    Pith/arXiv arXiv 2023

  18. [18]

    & Fan, Y

    Duan, H., Huo, M. & Fan, Y . From animal collective behaviors to swarm robotic cooperation. Natl. Sci. Rev. 10, nwad040 (2023)

    2023

  19. [19]

    Ren, X. et al. Pangu: Towards trillion parameter language model with spar se heterogeneous computing. Preprint at https://arxiv.org/abs/2303.10845 (2023)

    Pith/arXiv arXiv 2023

  20. [20]

    A., Y ang, J

    Cao, Y ., Geddes, T. A., Y ang, J. Y . H. & Y ang, P . Ensemble deep learning in bioinformatics. Nat. Mach. Intell. 2, 500–508 (2020)

    2020

  21. [21]

    & Franklin, M

    Li, G., Wang, J., Zheng, Y . & Franklin, M. J. Crowdsourced dat a management: A survey. IEEE Trans. Knowl. Data Eng. 28, 2296–2319 (2016)

    2016

  22. [22]

    Dawid, A. P . & Skene, A. M. Maximum likelihood estimation of o bserver error-rates using the EM algorithm. J. Royal Stat. Soc. Ser. C (Applied Statistics) 28, 20–28 (1979). 17/30

    1979

  23. [23]

    Wolpert, D. H. Stacked generalization. Neural Networks 5, 241–259 (1992)

    1992

  24. [24]

    Li, D. et al. From generation to judgment: Opportunities and challenges of LLM-as-a-judge. Preprint at https://arxiv.org/abs/2411.16594 (2025)

    arXiv 2025

  25. [25]

    Caliskan, A., Bryson, J. J. & Narayanan, A. Semantics derive d automatically from language corpora contain human-like biases. Science 356, 183–186 (2017)

    2017

  26. [26]

    S., Correll, J

    Ma, D. S., Correll, J. & Wittenbrink, B. The Chicago face data base: A free stimulus set of faces and norming data. Behav. Res. Methods 47, 1122–1135 (2015)

    2015

  27. [27]

    Holistic evaluation of language models

    Liang, P .et al. Holistic evaluation of language models. Trans. Mach. Learn. Res. (2023)

    2023

  28. [28]

    A survey on evaluation of large language models

    Chang, Y .et al. A survey on evaluation of large language models. ACM Trans. Intell. Syst. T echnol.15, 1–45 (2024)

    2024

  29. [29]

    Touvron, H. et al. Llama 2: Open foundation and fine-tuned chat models. Preprint at https://arxiv.org/abs/2307.09288 (2023)

    Pith/arXiv arXiv 2023

  30. [30]

    Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023)

    Pith/arXiv arXiv 2023

  31. [31]

    Attacks on third-party APIs of large language models

    Zhao, W .et al. Attacks on third-party APIs of large language models. In Int’l Conf. Learning Representations W orkshop on Secure and Trustworthy Large Language Models (Vienna, Austria, 2024)

    2024

  32. [32]

    Tang, R., Chuang, Y .-N. & Hu, X. The science of detecting LLM- generated text. Commun. ACM 67, 50–59 (2024)

    2024

  33. [33]

    Ensemble Methods: F oundations and Algorithms (CRC Press, 2012)

    Zhou, Z.-H. Ensemble Methods: F oundations and Algorithms (CRC Press, 2012)

    2012

  34. [34]

    Bagging predictors

    Breiman, L. Bagging predictors. Mach. Learn. 24, 123–140 (1996)

    1996

  35. [35]

    Perrone, M. P . & Cooper, L. N. When networks disagree: Ensemble methods for hybrid neural networks, 342–358 (1995)

    1995

  36. [36]

    & Suen, S

    Lam, L. & Suen, S. Application of majority voting to pattern r ecognition: an analysis of its behavior and performance. IEEE Trans. Syst. Man, Cybern. - Part A: Syst. Humans 27, 553–568 (1997)

    1997

  37. [37]

    & Grofman, B

    Shapley, L. & Grofman, B. Optimizing group judgmental accur acy in the presence of interdependencies. Public Choice 43, 329–343 (1984)

    1984

  38. [38]

    & Kluger, Y

    Parisi, F., Strino, F., Nadler, B. & Kluger, Y . Ranking and combining multiple predictors without labeled data. Proc. Natl. Acad. Sci. 111, 1253–1258 (2014)

    2014

  39. [39]

    & Ruvolo , P

    Whitehill, J., Wu, T.-F., Bergsma, J., Movellan, J. & Ruvolo , P . Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proc. Advances in Neural Information Processing Systems (2009)

    2009

  40. [40]

    & Hovy, E

    Hovy, D., Berg-Kirkpatrick, T., V aswani, A. & Hovy, E. Learn ing whom to trust with MACE. In Proc. Conf. North American Chapter of the Association for Computational Ling uistics: Human Language T echnologies, 1120–1130 (2013)

    2013

  41. [41]

    & Olshevsky, A

    Ma, Q. & Olshevsky, A. Adversarial crowdsourcing through ro bust rank-one matrix completion. In Proc. Advances in Neural Information Processing Systems (2020)

    2020

  42. [42]

    & Shah, D

    Karger, D., Oh, S. & Shah, D. Iterative learning for reliable crowdsourcing systems (2011)

    2011

  43. [43]

    Demartini, G., Difallah, D. E. & Cudré-Mauroux, P . ZenCrowd : leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In Proc. Int’l Conf. W orld Wide W eb, 469–478 (2012)

    2012

  44. [44]

    A ydin, B. et al. Crowdsourcing for multiple-choice question answering. In Proc. AAAI Conf. Artificial Intelligence , 2946–2953 (2014)

    2014

  45. [45]

    Li, Q. et al. Resolving conflicts in heterogeneous data by truth discover y and source reliability estimation. In Proc. ACM SIGMOD Int’l Conf. Management of Data , 1187–1198 (2014)

    2014

  46. [46]

    A lightweight, effective, and efficient model for label aggr egation in crowdsourcing

    Y ang, Y .et al. A lightweight, effective, and efficient model for label aggr egation in crowdsourcing. ACM Trans. Knowl. Discov. from Data 18, 1–27 (2024)

    2024

  47. [47]

    & Y u, Y

    Yin, L., Han, J., Zhang, W . & Y u, Y . Aggregating crowd wisdoms with label-aware autoencoders. In Proc. Int’l Joint Conf. Artificial Intelligence , 1325–1331 (2017)

    2017

  48. [48]

    & Cohn, T

    Li, Y ., Rubinstein, B. & Cohn, T. Exploiting worker correlat ion for label aggregation in crowdsourcing. In Proc. Int’l Conf. Machine Learning, 3886–3895 (2019)

    2019

  49. [49]

    Li, S., Wang, Z., Liu, C. & Wu, D. Black-box test-time ensembl e. IEEE Comput. Intell. Mag. 21, 57–68 (2026)

    2026

  50. [50]

    summary_of_the_paper

    Y u, S., Luo, M., Madusu, A., Lal, V . & Howard, P . Is your paper b eing reviewed by an LLM? benchmarking AI text detection in peer review. Preprint at https://arxiv.org/abs/2502.19614 (2025). 18/30 Acknowledgements This research was supported by National Natural Science Fou ndation of China (62525305). Author contributions statement S.L. conceived the s...

    arXiv 2025