{"id":"d791f9fd-afb3-4e8a-8f64-a3b735a91cd8","arxiv_id":"2502.06207","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs become less accurate and more overconfident as human annotator agreement drops, and training on disagreement samples improves in-domain accuracy and confidence alignment.","lead":"This paper tests how well large language models judge offensive tweets when human annotators disagree, and whether training on disputed examples helps. It matters because content moderation systems need models that know when they are unsure instead of confidently making wrong calls.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Overconfidence claim rests on an improper scoring rule: temperature-vote averages vs. human label means (App. A.2 Eqs. 1-2) measure distribution mismatch, not calibration, and Table 5's GPT-4o w/ A0 row even shows MSE worsening on A0.","rationale":"The paper does solid empirical work: transparent prompts, multiple open and closed models, replication with Qwen, and a high-quality unaggregated dataset. The accuracy decline on A0 (e.g., GPT-4o F1 from 85.24% on A++ to 57.06% on A0) is robust and independent of the metric concern. However, the headline 'overconfidence' and the 'improved alignment' claim both depend on the MSE defined in Appendix A.2 between a 5-temperature hard-vote average and the mean of 5 human labels. This measure is not a calibration score; the proper Brier score against individual annotator labels would be substantially larger for confident predictions on split samples, so the proposed metric conflates cross-rater disagreement with model miscalibration. Moreover, the improvement claim is internally inconsistent: Table 5's GPT-4o w/ A0 row shows A0 MSE worsening from 0.1923 to 0.1979, so 'disagreement samples improve alignment' is not true for the flagship model on the key subset. The reader's weakest_assumption identified the metric as the critical assumption; I agree and sharpen it with the improper-scoring-rule argument and the table-level contradiction. The authors do acknowledge the approximate nature of the confidence estimate in their Limitations section, but they continue to frame the result as overconfidence in the abstract and conclusion. A recalibration analysis would settle whether the finding is genuinely about calibration or merely about divergence from the human vote distribution. This concern does not change the CONDITIONAL verdict: the paper's empirical evidence is worth conditional acceptance, but the abstract's phrasing goes beyond what the metric supports, and the flagged contradictions should be fixed or qualified.","tokens_in":22378,"tokens_out":13148,"duration_ms":117525,"concrete_test":"Run a recalibration analysis on the A0 test subset: (i) estimate each sample's model probability p_hat of 'offensive' from 100 temperature-1 samples (or from logits if available) instead of the 5-sample hard-vote average; (ii) compute the proper multi-label Brier score B = (1/5) * sum_j (p_hat_i - Y_ij)^2 over the five individual human annotations; (iii) bin p_hat into [0,0.2], (0.2,0.4], (0.4,0.6], (0.6,0.8], (0.8,1] and, within each bin on A0, compare the mean p_hat to the observed accuracy of the hard majority label. If accuracy in the high-confidence bins is close to p_hat, the overconfidence claim fails; if accuracy is substantially below p_hat, it survives. Also independently reproduce Table 5's w/ A0 row for GPT-4o; if the A0 MSE increase from 0.1923 to 0.1979 replicates, the improvement claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs are 'overconfident' and that disagreement-aware training improves 'alignment' is operationalized in Section 3.2 / Appendix A.2 (Eqs. 1-2) by two averages: model confidence c_i is the mean of five hard predictions under temperatures {0, 0.25, 0.5, 0.75, 1}, and the target y_i is the mean of five binary human labels. This metric is not a calibration measure. First, it is an improper scoring rule relative to individual annotator labels: for a sample with three 'offensive' and two 'non-offensive' votes (y=0.6), a model outputting c=1 is scored (1-0.6)^2=0.16, whereas the proper Brier score against the five individual labels is 0.4. The paper's MSE therefore rewards confident predictions on split samples, so it cannot measure whether confidence tracks accuracy. Second, high Cohen's kappa across temperatures (Figure 2, >0.75 on A0) is presented as evidence of overconfidence, but kappa measures self-consistency, not calibration against outcomes. Third, the ideal model is defined as one whose confidence equals the annotator disagreement rate, a normative target; the observed gap on A0 (MSE about 0.19-0.23) shows the model's sampling distribution is more extreme than the human vote distribution, which is distributional mismatch, not necessarily overconfidence in the model's own beliefs. The accuracy drop on A0 is real, but the 'overconfidence' finding, and the 'improved alignment' claim built on the same MSE, would not survive a redefinition of the metric. Additionally, the abstract's improvement claim is internally contradicted by Table 5: GPT-4o few-shot with A0 examples raises A0 MSE from 0.1923 to 0.1979 and improves A0 accuracy by only 0.47 points, so the flagship model does not show improved alignment on the key subset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates 12 large language models on the MD-Agreement offensive language dataset, which provides five binary annotations per tweet and three agreement levels (A++, A+, A0). In zero-shot classification, accuracy and F1 decline sharply as agreement decreases, with all models scoring below 65% accuracy on A0 samples. The authors define model confidence as the mean of five hard predictions sampled at temperatures 0, 0.25, 0.5, 0.75, and 1, and compute the MSE and Spearman correlation between this confidence and the soft label (the mean of the five human binary labels). They report that confidence alignment weakens on low-agreement items while models remain highly self-consistent under temperature resampling, which they interpret as overconfidence. In few-shot and instruction fine-tuning experiments, they find that including disagreement samples generally improves overall accuracy and, on some subsets, reduces the MSE, with Qwen replications and released code. The paper's central interpretive claim is that LLMs are overconfident on ambiguous inputs and that disagreement-aware training improves both accuracy and alignment with human judgment.","tokens_in":22673,"tokens_out":5948,"duration_ms":48766,"significance":"If the overconfidence finding holds, it has direct practical relevance for content moderation, because models would be systematically unreliable precisely where human judgment is split. The empirical sweep is a strength: twelve models from closed and open families, a well-established disagreement-annotated benchmark, transparent prompts, and replication experiments with Qwen models. The accuracy decline across agreement levels is consistent and robust. However, the paper's two headline claims—overconfidence in ambiguous cases and improved alignment from disagreement-aware training—both rest on a metric whose statistical interpretation is problematic. The current analysis does not yet establish that the observed behavior constitutes overconfidence in a calibrated sense, rather than a distributional mismatch between averaged binary model outputs and averaged binary human labels. The study is a useful contribution to the literature on LLM evaluation under annotation disagreement, but the central claim needs reanalysis with proper scoring rules or a clear reframing.","major_comments":[{"comment":"The MSE in Eq. (2) compares the mean of five binary human labels to the mean of five binary model predictions. This is not a proper scoring rule for the individual-label prediction problem. For an item with three offensive and two non-offensive labels (y=0.6), a model outputting c=1 receives MSE (1-0.6)^2=0.16, whereas the Brier score computed against the five individual labels is 0.4. The metric therefore rewards extreme predictions on split samples, and the reported A0 MSE of approximately 0.19-0.23 reflects distributional mismatch between the model's sampling distribution and the human vote distribution, not necessarily overconfidence in the model's own beliefs. Because the abstract's 'overconfident' claim and the 'improved alignment' claims in Section 4 both rely on this metric, the authors should reanalyze the data with a proper scoring rule (e.g., per-label Brier score, expected calibration error, or reliability diagrams) or explicitly reframe the finding as a distributional comparison.","section":"Section 3.2, Appendix A.2 (Eqs. 1-2)"},{"comment":"The use of Cohen's kappa across temperature resamples as evidence of overconfidence is not valid for that purpose. Kappa measures self-consistency: a model that always returns the same hard label at every temperature has kappa equal to 1 regardless of whether that label is correct or whether the item is genuinely ambiguous. The reported kappa above 0.75 on A0 shows only that temperature sampling does not change the binary decision; it does not show that the model's confidence exceeds its accuracy. To support the overconfidence conclusion, the authors need to compare confidence against accuracy within A0 (e.g., reliability curves or confidence-accuracy histograms) or demonstrate that the observed self-consistency exceeds what a calibrated model would exhibit on the same items.","section":"Section 3.2, Figure 2"},{"comment":"The abstract and Section 4 claim that training on disagreement samples improves alignment with human judgment, but on the A0 subset the few-shot results in Table 5 show the MSE increasing from 0.1923 (zero-shot) to 0.1979 with w/ A0 and to 0.2022 with w/ A++/0. The overall MSE reduction is driven primarily by the A++ and A+ subsets. The alignment-improvement claim should be qualified, or additional evidence such as a proper-score calibration analysis on A0 should be provided to show that disagreement-aware training actually improves calibration where agreement is weakest.","section":"Section 4.1, Table 5"}],"minor_comments":[{"comment":"The caption contains a typo, 'Eamples', which should be 'Examples'.","section":"Section 2.1, Table 1"},{"comment":"The column header 'Ao' appears in several tables and should be 'A0' for consistency with the notation defined in Section 2.1.","section":"Tables 3-6 and B1-B2"},{"comment":"The text states 'using LLaMa3-7B as the backbone', but all tables list LLaMa3-8B; the exact model variant should be clarified.","section":"Section 4.2"},{"comment":"The three rows labeled 'w/ A++' with different numerical values are likely intended to be 'w/ A++', 'w/ A+', and 'w/ A0'; the table labels should be corrected.","section":"Appendix B.5, Table B3"},{"comment":"The text cites 'as shown in Figure B3' for accuracy on offensive and non-offensive subsets, but Appendix B.6's Figure B3 shows linguistic-feature performance; the figure reference should be corrected.","section":"Section 3.1, finding (4)"},{"comment":"The cross-references to Table 5 and Table 6 appear swapped: Section 4.1 describes few-shot GPT-4o results yet refers to Table 6, while Section 4.2 describes instruction fine-tuning yet refers to Table 5; the references should be fixed.","section":"Sections 4.1-4.2"}],"recommendation":"major_revision","confidential_remarks":"The empirical sweep is solid and the accuracy trend is robust, but the title, abstract, and conclusion hinge on the 'overconfident' interpretation, which currently rests on an improper aggregation of binary outputs. I would advise the editor that the paper should not be accepted until the authors either reanalyze the confidence data with proper scoring rules or explicitly reframe the contribution as measuring distributional mismatch rather than calibration. The table cross-reference errors and inconsistent model names should also be corrected in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The paper systematically evaluates 12 LLMs on MD-Agreement, stratifying by annotator agreement, and the headline accuracy result is real: every model drops sharply on low-agreement (A0) samples, and scaling doesn't fix it. That is a clean, reproducible finding, and the appendix replication with Qwen supports it. The linguistic-feature analysis (sarcasm and rhetorical questions being hardest) is a genuinely useful addition, and the authors are upfront about the single-dataset limitation.\n\nThe soft spot is the confidence story. They measure confidence as the mean of hard predictions across five temperatures and compare it to the mean of five binary human labels (Eqs. 1-2). That gives a distributional-mismatch score, not a calibration error. A model that says 'offensive' 5 times out of 5 on a 3-2 human split gets penalized 0.16 by their MSE, while the proper Brier against the individual labels would be 0.4. High kappa across temperatures is self-consistency, not overconfidence in the Bayesian sense. So 'overconfident' is too strong a word for what they measured. The abstract's claim that disagreement samples in training improve both accuracy and alignment is also not uniformly true: GPT-4o few-shot with A0 actually worsens A0 MSE (0.1923 to 0.1979 in Table 5). The Qwen replication goes the other way, which is interesting but undercuts the general claim.\n\nThe authors acknowledge the metric limitation and the single dataset. That helps, but the title and abstract still overstate. A revision that reframes the finding as 'LLMs do not reproduce the human label distribution on ambiguous items' rather than 'LLMs are overconfident' would be more honest, and adding a token-probability calibration measure would strengthen it. The OOD analysis in B.5 is thin, covering one dataset with accuracy only.\n\nBottom line: the empirical backbone is solid and worth refereeing. The central accuracy finding will be useful to anyone building moderation pipelines with LLMs. I'd send it to review with a clear request to revise the confidence interpretation.","headline":"Solid agreement-stratified evaluation of LLMs on offensive language; the 'overconfident' conclusion is a stretch given the confidence proxy.","tokens_in":23312,"tokens_out":2878,"would_cite":true,"duration_ms":28192,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs give their most confident offensive-language verdicts on exactly the tweets human annotators disagree about.","keywords":["annotation disagreement","offensive language detection","LLM confidence","overconfidence","self-consistency","instruction fine-tuning","few-shot learning","MD-Agreement"],"falsifier":"Take the low-agreement subset and measure each model's confidence with internal token probabilities or a proper scoring rule such as the Brier score or expected calibration error, instead of temperature-averaged hard predictions. If the paper's overconfidence claim is right, those calibrated confidences will still sit near 1.0 on tweets whose human agreement is only around 0.6; if the claim is a measurement artifact, the gap will shrink or disappear.","tokens_in":22149,"feed_emoji":"⚖️","tokens_out":6069,"duration_ms":52363,"temperature":0.7,"pith_summary":"The paper asks whether large language models behave like reliable judges of offensive language when the humans who labeled the data could not agree. It finds that they do not: on tweets where five annotators split, model accuracy falls below 65 percent, yet the models still emit high confidence, so an automated moderator would be most assertive precisely where human judgment is most divided. The paper further argues that this is not a fixed flaw: showing models disagreement samples in few-shot prompts or instruction fine-tuning improves both binary accuracy and the match between model confidence and the human agreement level. If this is right, treating offensive-language moderation as binary classification misses the central failure mode, and disagreement-aware training is a concrete repair.","feed_headline":"LLMs stay overconfident on tweets humans dispute","feed_subtitle":"On split votes accuracy falls below 65% while confidence stays high; training on those cases restores alignment.","key_machinery":"The load-bearing objects are the MD-Agreement dataset's soft labels and the self-consistency confidence estimate. MD-Agreement contains 10,753 tweets each annotated by five trained raters; soft labels are the averaged zero-one votes, bucketed into unanimous, mild, and weak agreement, and the corpus is used because prior validation showed its disagreements come from genuine ambiguity rather than labeling error. Confidence is derived by resampling each model at temperatures 0, 0.25, 0.5, 0.75, and 1 and averaging the binary predictions, following the self-consistency method. The alignment claim is carried by the mean squared error between that confidence and the soft label, supported by Spearman's rank correlation and Cohen's kappa; the few-shot and fine-tuning results show these metrics improving when disagreement samples enter training.","core_discovery":"On the paper's own terms, the discovery is a pattern and a prescription. Across twelve open- and closed-source LLMs on the MD-Agreement corpus, zero-shot accuracy and F1 are strong on unanimously labeled tweets and collapse on weakly agreed tweets, with every model scoring below 65 percent on A0 samples and with models biased toward calling ambiguous content offensive. Confidence measured by self-consistency, averaging hard predictions over five temperature settings, stays high on low-agreement samples, so the mean squared error against soft labels roughly quadruples when moving from unanimous to weak agreement, and even the best model, GPT-4o, assigns high confidence across the whole agreement spectrum. The corollary the paper establishes is training-side: including disagreement samples in few-shot learning improves generalization, and instruction fine-tuning with a balanced mix of agreement levels, with medium-agreement A+ samples offering the best single-category trade-off, improves both detection accuracy and the alignment of model confidence with human disagreement.","pith_inferences":["If this result transfers to other subjective annotation tasks such as humor, misogyny, or stance detection, then the disagreement rate of a sample could serve as a cheap, annotation-free curriculum signal for selecting training data, since the paper shows agreement level tracks both accuracy and confidence alignment.","The paper compares averaged binary model outputs to averaged binary human labels; a stronger test of the overconfidence claim would use token-level probabilities or proper scoring rules such as the Brier score or expected calibration error, and until that is done the magnitude of the overconfidence gap may be partly a measurement choice, even though the accuracy collapse is not.","In deployment, the finding implies a two-stage moderation design: act automatically only on inputs whose agreement profile is learnable, and explicitly model annotation distributions rather than majority vote, which requires datasets with unaggregated labels.","A testable extension is that models trained on disagreement samples should show a sharper drop in false positives on ambiguous non-offensive tweets, since the paper identifies that category as the main error type."],"forward_implications":["A moderation system that trusts an LLM's confidence is most likely to be wrong, and most firmly wrong, on borderline content, so confidence scores cannot be used on their own to route ambiguous cases to human review.","Low-agreement samples behave as a hard floor for scaling: increasing model size helps unanimous cases substantially but barely moves accuracy on ambiguous cases, so scaling alone will not resolve subjective disagreement.","Training data composition is a lever: mixing unanimous and disagreement samples beats any single agreement level, while including too many agreement categories can slightly reduce accuracy, so selection should balance clarity and ambiguity.","Fine-tuning with medium-agreement samples gives the best accuracy-calibration trade-off in-domain and also generalizes best out-of-distribution, suggesting that medium agreement is a useful proxy for moderate data complexity."],"supporting_citations":[{"why":"Supplies the MD-Agreement dataset, the corpus of 10,753 tweets with five annotations per tweet and the A++, A+, and A0 agreement schema that the entire evaluation rests on.","marker":"(Leonardelli et al., 2021)"},{"why":"Defines the SemEval 2023 Task 11 learning-with-disagreements framework and the MSE-based protocol for comparing model outputs with disagreement labels used in the alignment analysis.","marker":"(Leonardelli et al., 2023)"},{"why":"Provides the self-consistency method that the paper adapts to estimate LLM confidence by averaging hard predictions across temperature resampling.","marker":"(Wang et al., 2023b)"},{"why":"Validates that MD-Agreement's disagreements reflect genuine linguistic ambiguity rather than annotation error, and supplies the linguistic-feature annotations used in the error analysis.","marker":"(Sandri et al., 2023)"},{"why":"Supplies the offensive-language prompt template that the study reuses in zero-shot, few-shot, and fine-tuning settings.","marker":"(Roy et al., 2023)"},{"why":"Establishes the learning-with-disagreements evaluation tradition and the criterion that a well-aligned model should show lower confidence on ambiguous samples.","marker":"(Uma et al., 2021a)"}],"fun_headline_variants":["LLMs overconfident on contested tweets","Overconfident LLMs miss human disagreement","Training on disputed tweets fixes overconfidence","LLMs' false confidence on ambiguous insults","Low agreement stumps LLMs; training helps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The overconfidence result assumes that the correct confidence for a model on a disputed tweet is exactly the fraction of human annotators who called it offensive, and that averaging the model's yes-no predictions over a few random samples faithfully measures its confidence; if either assumption fails, the gap between model confidence and human agreement is partly an artifact of comparing averaged binary outputs with fractional human labels.","fun_headline_variants_meta":{"raw":{"variants":["LLMs overconfident on contested tweets","Overconfident LLMs miss human disagreement","Training on disputed tweets fixes overconfidence","LLMs' false confidence on ambiguous insults","Low agreement stumps LLMs; training helps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1287,"prompt_tokens":910,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":311}},"tokens_in":526,"tokens_out":377,"duration_ms":3660,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:23:30.694815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the low-agreement subset and measure each model's confidence with internal token probabilities or a proper scoring rule such as the Brier score or expected calibration error, instead of temperature-averaged hard predictions. If the paper's overconfidence claim is right, those calibrated confidences will still sit near 1.0 on tweets whose human agreement is only around 0.6; if the claim is a measurement artifact, the gap will shrink or disappear.","supporting_citations":[],"review_version":1}