{"id":"756aed8e-d847-4fbc-be20-d368573fc353","arxiv_id":"2505.12349","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Hybrid human-LLM crowds with locally weighted aggregation outperform human-only and LLM-only groups on both accuracy and bias reduction for headline verification.","lead":"Researchers tested 18 large language models on bias-eliciting fake news headlines and found they mirror human ethnic biases. Mixing humans and LLMs in hybrid crowds and weighting their answers by headline type cut bias and boosted accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's headline hybrid advantage (0.813 vs 0.752) lacks paired uncertainty estimates; the 'significantly enhance' claim may be sampling noise.","rationale":"The reader's formal weakest_assumption is external validity of the Abels et al. dataset and 40-person pool. I agree that is a limitation. However, I think the more immediate load-bearing issue is internal: the paper's own data do not currently demonstrate that the hybrid advantage is statistically distinguishable from noise, because Table 1 lacks paired intervals and the only reported CIs (Figure 2) are not clearly tied to the specific Table 1 rows. This is a correctness/evidence-strength issue rather than a generalization issue. If the paired bootstrap shows the 0.813 vs 0.752 gap is robust, then conditional acceptance on external validation remains appropriate; if it does not, the central claim needs to be weakened to a qualitative observation. I therefore keep the reader's CONDITIONAL verdict but shift the emphasis from dataset representativeness to missing uncertainty quantification. This is not a fraud or soundness objection; the method and code appear credible, and the concern is testable.","tokens_in":17296,"tokens_out":6341,"duration_ms":69661,"concrete_test":"Recompute the Table 1 group-size-8 comparison as a paired bootstrap over the same random draws used for Figure 2: for B=1,000 iterations, resample the human subset (and LLM subset for non-+ groups), refit ExpertiseTrees on identical cross-validation splits, and record D = Accuracy(ET(hybrid+)) - Accuracy(ET(LLM+)) and the corresponding difference in mean |counterfactual bias| across the six category-status cells. Report the 95% bootstrap CI for D and for the bias reduction. If the CI for D includes 0, or the bias-reduction CI overlaps the pure-crowd value, the headline significance claim is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is not external validity but whether the headline in-sample result is distinguishable from noise. The central claim—'hybrid crowds containing both significantly enhance performance and further reduce biases'—is anchored on Table 1's point estimates: ExpertiseTree(hybrid+) = 0.813 vs 0.752 (LLM+) and 0.749 (human), with 'near zero' counterfactual bias deltas. The paper reports no paired confidence interval or significance test for these numbers. Figure 2 shows 95% CI for accuracy by group size, but it is not shown that the specific group-size-8 ExpertiseTree comparison has non-overlapping intervals or that the paired difference excludes zero. This matters because the numbers carry two sources of sampling variability: the random draw of 8 humans from a pool of 40 (and, for hybrid, the random LLM subset when not using MMLU selection) and the cross-validation folds used to fit ExpertiseTree weights, while the 4-shot examples also vary. With roughly 40 human raters and few headlines per category-sentiment cell, an accuracy gap of 0.061 can plausibly arise from this variability. Similarly, the bias deltas for hybrid+ (e.g., +0.04 for ethnicity genuine and altered) are reported without intervals; a non-significant Mann-Whitney U test on a small cell is not evidence that the bias is 'negligible.' Thus the paper's quantitative support for the central claim is not yet established, independent of the acknowledged single-dataset limitation. The public code and prior ExpertiseTrees work are helpful, but they do not supply the missing uncertainty analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies bias mitigation in a headline authenticity task by aggregating responses from human and LLM crowds. It first replicates a human headline experiment with 18 LLMs, showing that LLMs mirror human counterfactual biases, especially for ethnicity-related headlines. It then evaluates simple averaging, weighted averaging, and locally weighted ExpertiseTree aggregation for LLM-only, human-only, and hybrid crowds. The central claim is that hybrid crowds containing both humans and LLMs, aggregated with ExpertiseTrees, achieve the highest accuracy (0.813 for ExpertiseTree(hybrid+) in Table 1) and reduce counterfactual bias to near-zero levels, outperforming both human-only and LLM-only groups. The paper also analyzes crowd diversity via the Q-statistic and reports that simple LLM averaging can amplify biases.","tokens_in":17588,"tokens_out":4406,"duration_ms":46696,"significance":"If the central result holds, the paper offers a practical, empirically grounded recipe for combining human and LLM responses to improve accuracy and reduce social biases in content moderation and similar tasks. It contributes to the collective intelligence literature by providing evidence that humans and LLMs have complementary strengths (diversity vs. accuracy) and that locally weighted aggregation can exploit this complementarity. The study is strengthened by its open-source code, the systematic evaluation of 18 diverse LLMs, and the use of appropriate significance tests for the per-cell counterfactual bias comparisons. However, the main quantitative claim—that hybrid crowds significantly outperform LLM-only and human-only groups—is currently under-supported because the headline accuracies in Table 1 are reported without paired confidence intervals or paired significance tests, and the accuracy threshold for the Likert-to-binary mapping is unspecified.","major_comments":[{"comment":"The central claim that ExpertiseTree(hybrid+) 'significantly enhance[s] performance' rests on point estimates (0.813 vs 0.752 for LLM+ and 0.749 for human) that are reported without paired uncertainty estimates. Group construction introduces multiple sources of sampling variability: random draws of 8 humans from a pool of 40, random LLM subsets for the non-plus groups, and cross-validation folds for weight fitting. Figure 2 shows 95% confidence intervals by group size, but the paper never states whether the specific group-size-8 comparisons have non-overlapping intervals or whether the paired difference excludes zero. I request bootstrap or paired permutation confidence intervals for the accuracy and bias-delta comparisons in Table 1, particularly for ExpertiseTree(hybrid+) versus ExpertiseTree(LLM+) and ExpertiseTree(human).","section":"§4.2, Table 1, Figure 2"},{"comment":"Accuracy is defined as 'the proportion of correctly identified headlines, whether genuine or altered,' but the paper does not specify how a 5-point Likert likelihood response is converted to a binary correct/incorrect decision. A threshold (e.g., p_h >= 0.5) is necessary to compute every accuracy number in Table 1 and Figure 2. Without this threshold, the headline results are not reproducible. Please state the threshold explicitly and provide a sensitivity analysis over plausible thresholds to confirm that the hybrid advantage is not an artifact of the chosen cutoff.","section":"§3.2, Accuracy definition"},{"comment":"The ExpertiseTree aggregator is central to the paper's positive results, but the supplementary description lacks algorithmic detail: no pseudocode, no specification of tree depth, minimum leaf size, split criterion, pruning, or regularization, and no description of how cross-validation is nested with the random group sampling. Because the method originates in the authors' prior work, these details may exist elsewhere, but the paper should state the exact configuration used here for independent verification. Additionally, since the method and the human comparison data come from the authors' own prior papers and the weights are fitted on the same headline task via cross-validation, the paper should address the risk that the method's advantage reflects tuning on this specific dataset; a nested model-selection or an evaluation on an independent task would materially strengthen the generality claim.","section":"§3.3 and §S1.2 (ExpertiseTrees)"},{"comment":"The paper is commendable for including a Limitations section that acknowledges the single-dataset scope, but the abstract and conclusion make broad claims: 'hybrid crowds containing both significantly enhance performance and further reduce biases across ethnic and gender-related contexts.' Given one dataset and one pool of 40 human participants, these statements should be explicitly scoped to the studied task and participant pool, or supported by at least one additional dataset or an independent human sample. This is not a fundamental flaw because the limitation is acknowledged, but the strength of the wording exceeds what the evidence currently supports.","section":"Limitations and Conclusion"}],"minor_comments":[{"comment":"The abstract uses 'significantly enhance performance' without reporting any significance test for the hybrid advantage; this wording should be aligned with the statistical evidence (or revised after adding paired tests).","section":"Abstract"},{"comment":"The paper should report the number of headlines in each category-sentiment demographic cell, since the Mann-Whitney U and Wilcoxon tests have low power with very small cell sizes; this context is essential for interpreting the cell shading.","section":"§3.2 and Table 1"},{"comment":"The terms 'Black' and 'African American' are used interchangeably; please choose one and use it consistently.","section":"Table 1 and §3.1"},{"comment":"The reference to 'Hendrycks et al.' lacks a complete citation (year and venue); please provide the full bibliographic entry.","section":"References"},{"comment":"Please clarify in the caption whether the shaded areas are 95% confidence intervals for the mean accuracy or for the distribution of group draws, and state how many random group draws were used to compute them.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and addresses an important problem, but the missing uncertainty quantification on the headline result is a standard yet essential requirement. The issue is fixable with additional analysis (paired bootstrap CIs, significance tests), so I recommend major revision rather than rejection. The reliance on the authors' own prior method and dataset is a further risk that should be addressed explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first head-to-head comparison I've seen of human-only, LLM-only, and hybrid crowds on a bias-eliciting task, and the headline result—hybrid crowds with locally weighted aggregation beat both pure crowds—is a real new data point. The secondary finding that averaging LLM outputs amplifies shared biases is also concrete and likely correct.\n\nWhat the paper does well: it diagnoses why LLM crowds fail to self-correct—average pairwise Q-statistic among LLMs is 0.855 vs 0.387 for humans, so errors are correlated. It then shows that ExpertiseTrees, which learn context-specific weights, recover a good deal of the crowd benefit. The experimental setup is clean, the code is public, and the limitations section is honest about the single dataset and model versions.\n\nWhere I'd push back: the central claim of 'significantly enhance' is anchored on Table 1, where ExpertiseTree(hybrid+) scores 0.813 vs 0.752 for LLM+ and 0.749 for human. Those numbers are reported without paired confidence intervals or a significance test. Given the pool is ~40 humans and 18 LLMs, and the ExpertiseTree weights are fit with cross-validation on the same task, a 0.061 gap could plausibly be noise. The near-zero bias deltas for hybrid+ also lack intervals. Figure 2 shows CIs for group size trends, but not for the specific group-8 comparisons in Table 1. So the 'significantly' is not yet established. That's the main soft spot.\n\nSecond, the aggregation weights are fitted to the same headline dataset. Cross-validation prevents gross overfitting, but this remains an in-sample demonstration. A careful referee should ask for a held-out task or at least a discussion of when this recipe transfers.\n\nThese are fixable, not fatal. The qualitative pattern—LLM diversity is low, hybrid diversity helps, simple averaging amplifies bias—is well supported by the Q-statistic analysis and the group-size curves. I'd trust that direction.\n\nWho's this for? Researchers in collective intelligence, AI fairness, and LLM evaluation. It's a solid empirical paper that deserves a serious referee; I'd send it out and ask for paired uncertainty quantification and an out-of-sample check before accepting. I'd also want the authors to soften 'significantly' until the intervals are there.","headline":"First head-to-head test of hybrid human-LLM crowds for bias mitigation; the qualitative story is solid, but the headline 'significant' claim lacks paired uncertainty.","tokens_in":18115,"tokens_out":3078,"would_cite":true,"duration_ms":30567,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hybrid crowds that mix humans and LLMs, combined through locally weighted aggregation, can judge bias-eliciting headlines more accurately than either group alone while reducing demographic bias to near zero.","keywords":["bias mitigation","hybrid human-LLM crowds","wisdom of crowds","ExpertiseTrees","locally weighted aggregation","counterfactual bias","LLM ensembles","collective intelligence"],"falsifier":"Run the same ExpertiseTree hybrid aggregation on an independent bias-eliciting headline dataset built around a different cultural context (e.g., caste or regional ethnicity) with a human pool of several hundred; if the hybrid crowd does not beat both human-only and LLM-only ExpertiseTrees while keeping counterfactual bias near zero, the claim is refuted. A cheaper check: measure the average Q-statistic of a newly released set of LLMs; if it has fallen to human levels (~0.4), the diversity argument for hybrids loses its empirical basis.","tokens_in":17091,"feed_emoji":"⚖️","tokens_out":5405,"duration_ms":48430,"temperature":0.7,"pith_summary":"This paper claims that the most reliable way to reduce demographic bias in language-model judgments is not to add more language models, but to add a few humans. On a set of genuine and altered news headlines, individual LLMs reproduce the same counterfactual biases humans show, yet LLMs are far more correlated with one another than humans are, so averaging LLM responses can lock in shared mistakes. The paper shows that locally weighted aggregation, which gives each responder context-dependent weight, recovers much of the wisdom of the crowd. The strongest result is that hybrid crowds of humans and LLMs, aggregated this way, reach higher accuracy than either group alone while leaving counterfactual bias near zero.","feed_headline":"Mixing humans with LLM crowds cuts headline bias to near zero","feed_subtitle":"Locally weighted aggregation over hybrid groups reaches 81.3% accuracy, beating human-only and LLM-only crowds.","key_machinery":"The load-bearing object is the ExpertiseTree, a decision-tree-like aggregation method whose leaves contain linear combinations of individual predictions rather than constants. Each leaf learns weights for a specific context region—here, headline category (age, ethnicity, gender)—so a responder who is reliable for gender headlines but biased on ethnicity headlines can be weighted up in one region and down in the other. Combined with a hybrid crowd, this lets the aggregate exploit LLM accuracy where LLMs are strong and human diversity where LLMs share the same blind spots. A second load-bearing component is the Q-statistic, which quantifies pairwise agreement and motivates why simple averaging fails when responders are highly correlated.","core_discovery":"The central discovery is a complementarity: LLMs bring higher individual accuracy and humans bring diversity, and the two together, when combined with context-sensitive weighting, outperform either alone. The paper measures diversity with the Q-statistic and finds LLM pairs far more similar (Q=0.855 ± 0.08) than human pairs (Q=0.387 ± 0.33), so simple averaging of LLM outputs barely improves accuracy and can even strengthen biases, as seen in the gender-altered aggregate. ExpertiseTrees, which learn separate linear combination weights for each headline category, route around biased or inaccurate responders. On the headline task, ExpertiseTree(hybrid+) reaches 0.813 accuracy with counterfactual bias deltas within ±0.04 for all reported categories, versus 0.752 for the LLM-only equivalent and 0.749 for the human-only equivalent. The paper concludes that modest hybrid ensembles can combine the accuracy of LLMs with the diversity of human perspectives to achieve more accurate and less biased collective judgments.","pith_inferences":["Because the human pool in the underlying dataset is small (around 40 participants), the quantitative superiority of hybrids is likely tied to that pool's particular mix of views; a much larger or differently selected pool could shift the balance.","The same design likely transfers to other judgment tasks where LLMs share systematic blind spots—content moderation, medical or legal triage—but only if the task has a comparable counterfactual structure that lets locally weighted aggregation learn where humans add diversity.","As LLM families converge on shared training data and alignment procedures, the Q-statistic between models should be monitored; if it rises toward 1, the hybrid advantage will depend even more on recruiting humans with genuinely distinct perspectives.","A testable extension is deliberately engineering LLM diversity through personas or fine-tuning and asking whether that substitutes for, or only augments, human diversity in the aggregate."],"forward_implications":["Simple averaging of multiple LLM responses is not a reliable bias-mitigation strategy; because LLMs are highly correlated with each other, averaging can amplify a bias that only a minority of models show.","Locally weighted aggregation (ExpertiseTrees) improves both accuracy and bias metrics over simple and static weighted averaging for LLM, human, and hybrid crowds.","Hybrid crowds of humans and LLMs matched or beat single-type crowds in accuracy while reducing counterfactual bias to levels no longer significant in the tested categories.","Group-size gains differ by composition: LLM-only groups plateau or decline as models are added, while hybrid groups continue to benefit from added humans because each human adds diversity, not redundancy.","Selecting LLMs by MMLU score (LLM+, hybrid+) improves aggregate performance further, but even with such selection, adding more LLMs without humans eventually degrades performance."],"supporting_citations":[{"why":"supplies the headline dataset, the human response pool, and the counterfactual/framing bias definitions the whole comparison builds on","marker":"[Abels et al., 2024]"},{"why":"introduces ExpertiseTrees, the locally weighted aggregation method used to combine crowds","marker":"[Abels et al., 2023]"},{"why":"provides the MMLU benchmark scores used to select the LLM+ and hybrid+ groups","marker":"[Hendrycks et al.]"},{"why":"supplies the theoretical premise that diverse groups can outperform high-ability groups, motivating the hybrid design","marker":"[Hong and Page, 2004]"},{"why":"prior evidence of a wisdom-of-LLM-crowds effect that this paper extends to bias mitigation and hybrid crowds","marker":"[Schoenegger et al., 2024]"},{"why":"defines stacking, the static weighted-average baseline that locally weighted aggregation must beat","marker":"[Breiman, 1996]"}],"fun_headline_variants":["LLM averaging worsens bias; human-LLM hybrid solves it","Hybrid crowds of humans and LLMs cut bias, boost accuracy","Locally weighted hybrid crowds beat solo LLM or human groups","Human diversity complements LLM accuracy in crowd bias","Simple LLM crowds fail; hybrid crowds reduce bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the headline dataset and the roughly 40 human participants from the earlier study being representative of real-world demographic bias; if those headlines or that participant pool do not generalize to other cultural contexts, the measured hybrid advantage may not hold.","fun_headline_variants_meta":{"raw":{"variants":["LLM averaging worsens bias; human-LLM hybrid solves it","Hybrid crowds of humans and LLMs cut bias, boost accuracy","Locally weighted hybrid crowds beat solo LLM or human groups","Human diversity complements LLM accuracy in crowd bias","Simple LLM crowds fail; hybrid crowds reduce bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000385,"raw_usage":{"total_tokens":2009,"prompt_tokens":895,"completion_tokens":1114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1030}},"tokens_in":511,"tokens_out":1114,"duration_ms":11373,"temperature":1.0,"reasoning_tokens":1030,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:35:15.913892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same ExpertiseTree hybrid aggregation on an independent bias-eliciting headline dataset built around a different cultural context (e.g., caste or regional ethnicity) with a human pool of several hundred; if the hybrid crowd does not beat both human-only and LLM-only ExpertiseTrees while keeping counterfactual bias near zero, the claim is refuted. A cheaper check: measure the average Q-statistic of a newly released set of LLMs; if it has fallen to human levels (~0.4), the diversity argument for hybrids loses its empirical basis.","supporting_citations":[],"review_version":1}