{"id":"3e688f36-78c4-4196-8157-c524ad8bdfc6","arxiv_id":"2501.06826","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"PAIR, a post-stratification-based replication method, improves calibration of NLP models trained on non-representative annotator pools in simulations, but the effect is not uniform for rare hate-speech labels.","lead":"This paper introduces a post-processing method called PAIR that rebalances NLP training data by duplicating annotations from underrepresented annotator groups so the data better reflects a target population. The method improves model calibration in simulated offensive-language and hate-speech tasks without collecting new annotations, which matters because annotator pools are often not representative of the populations models serve.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PAIR's validity rests on the homogeneous additive annotator-bias model; heterogeneous or content-dependent bias could break the replication weights.","rationale":"The reader's verdict is CONDITIONAL, and the weakest assumption identified there—the fixed additive shift β uniform across all tweets—is exactly the load-bearing concern I find. The paper's own Limitations section concedes that 'real-world annotator biases may be more nuanced or context-dependent,' and the sensitivity analysis on difficult tweets does not address heterogeneity in β, only floor/ceiling effects. The central claim that PAIR 'corrects these calibration problems' is therefore only directly supported for the stylized, homogeneous-bias simulation. My proposed concrete test would directly probe whether PAIR's benefit degrades when β varies per tweet; if it does, the abstract's unqualified claim needs tightening. Since the reader already conditioned the verdict on exactly such a qualification, no change to the verdict is needed. I do not see a more fundamental flaw: the evaluation metric ACB aligns with the population average under the symmetric additive model, the algorithm is transparent and reproducible, and the authors honestly report the HS full-set failure. The single-realization concern is real but secondary; it affects confidence intervals, not the qualitative pattern. Therefore I agree with the reader's assessment and recommend keeping the CONDITIONAL verdict.","tokens_in":12070,"tokens_out":4201,"duration_ms":45157,"concrete_test":"Run a simulation identical to §4 but draw a per-tweet bias β_i (e.g., β_i ~ U[0, 0.3], or β_i = c·(1 - |2pi - 1|) so ambiguous tweets have larger bias) while keeping the PAIR weights fixed at the marginal population shares (50/50). Generate the Representative, Non-representative 1, Non-representative 2, and Adjusted datasets, train the same RoBERTa models, and compare ACB(Adjusted) to ACB(Representative). If the gap exceeds the gap observed under the homogeneous-β simulation (Figures 5 and 11) by a practically meaningful margin, the homogeneity assumption is load-bearing and the central claim must be qualified to homogeneous-bias settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PAIR corrects calibration problems depends on the simulation's generative assumption that annotator bias is a single additive shift β on the probability scale, identical across all tweets and with only two annotator types (Eq. 4: pA_i = max(pi - β, 0); Eq. 5: pB_i = min(pi + β, 1)). Under this model, the population-level probability for tweet i is exactly pi (away from boundary truncation), and reweighting the A/B groups to 50/50 perfectly recovers the population distribution. But real-world annotator disagreement is likely heterogeneous: group differences may be larger for ambiguous, identity-salient, or culturally specific content, and may vary in direction across tweets. When β varies per tweet, the post-stratification weights in PAIR—which depend only on marginal group shares, not on tweet-level features or the form of the bias—do not guarantee that the weighted training distribution matches the population's conditional label distribution. Consequently, a model trained on the PAIR-adjusted dataset could remain miscalibrated for exactly the tweets where calibration matters most. The paper's sensitivity analysis (§5.3) does not resolve this: it filters to tweets with 0.4 ≤ pi ≤ 0.6 but keeps the same homogeneous β, so the heterogeneous-bias scenario is never tested. Thus the strongest empirical support for PAIR is obtained under an assumption that the authors themselves flag as an oversimplification, and the method's transportability to real annotator pools remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PAIR (Population-Aligned Instance Replication), a post-processing method that reweights training data from non-representative annotator pools to match target population proportions by replicating annotations from underrepresented groups. The method is tested in a simulation study on offensive language and hate speech detection using 3,000 tweets with simulated A/B annotator types and a bias parameter beta. The authors fine-tune RoBERTa on representative, two non-representative, and PAIR-adjusted datasets and evaluate calibration (ACB) and accuracy (F1). They find that non-representative pools degrade calibration while leaving accuracy largely unchanged; PAIR-adjusted models track representative models for offensive language on the full set and for both tasks on a difficult-tweet subset, while hate speech results on the full set are inconclusive. The paper includes recommendations for collecting annotator characteristics and weighting training data.","tokens_in":12304,"tokens_out":3251,"duration_ms":35525,"significance":"If the central claim holds, PAIR offers a cheap and interpretable method for mitigating annotator-pool bias without additional data collection, bridging survey statistics and NLP. The paper is honest about limitations, releases code, and provides a reproducible simulation framework. The main value is in demonstrating that calibration, not just accuracy, is sensitive to annotator composition and that post-stratification can help in stylized settings. However, the strength of the evidence is limited by the homogeneity of the simulated bias model, the use of the original annotation proportion as the evaluation target, and the lack of repeated simulation draws.","major_comments":[{"comment":"The evaluation target in ACB (Eq. 6) is the original annotation proportion pi, but the simulated population proportion for tweet i is 0.5*pA_i + 0.5*pB_i, which equals pi only when neither pA_i nor pB_i is truncated at 0 or 1. For boundary tweets, especially in the rare hate-speech class, the population proportion differs from pi by up to beta/2. Thus the reported calibration of the PAIR-adjusted model is measured against pi, not against the target population distribution that PAIR is designed to match. Please report ACB against the true population proportion (0.5*pA_i + 0.5*pB_i) as well as against pi, or restrict the analysis to non-boundary tweets; otherwise the central claim that PAIR corrects calibration problems is not directly supported by the metric used.","section":"Sec. 4.1, Eqs. (4)-(6)"},{"comment":"The simulation assumes a single additive bias beta on the probability scale that is identical across all tweets and identical for all annotators within each type. The sensitivity analysis in Sec. 5.3 filters to difficult tweets (0.4 <= pi <= 0.6) but retains the same homogeneous beta; it does not test heterogeneous or content-dependent bias. If beta varies per tweet or by content, the post-stratification weights in Eqs. (1)-(3), which depend only on marginal group shares, do not guarantee that the weighted training distribution matches the population's conditional label distribution. Please add a simulation with tweet-level heterogeneous bias (e.g., beta_i drawn from a distribution or increasing with ambiguity) or state precisely the conditions under which marginal reweighting suffices; otherwise the transportability of PAIR to real annotator disagreement remains unsupported.","section":"Sec. 4.1 and Sec. 5.3"},{"comment":"For each beta value, only one realization of the Bernoulli annotation draws is used to construct each dataset, and the five random seeds average over model training only, not over annotation sampling. Figures 5, 7, 11, and 13 therefore show training-seed variability but not sampling variability in the annotation generation process. The Limitations section acknowledges this, but the paper does not quantify how much of the observed ACB differences between datasets could be due to a single draw. Please repeat the annotation simulation many times (e.g., 20-50 replicates) and report intervals over these replicates, or at least bootstrap the annotation draw; without this, the claim that PAIR improves calibration over non-representative datasets for OL cannot be distinguished from noise for small differences.","section":"Sec. 4.1 and Sec. 7"}],"minor_comments":[{"comment":"The legend in several figures uses 'Balanced' while the text and Table 1 use 'Representative'; please use consistent dataset names throughout.","section":"Figures 5, 7, 11, 13"},{"comment":"The sentence 'All models show lower average ACB values than the representative model across a wide range of the bias offset (0.10 - 0.20)' is confusing: if the adjusted model roughly tracks the representative model, it cannot be that all models have lower ACB than the representative model unless 'representative' is a typo for 'non-representative'. Please clarify the intended comparison.","section":"Sec. 5.2"},{"comment":"The choice of normalization constant K = 4/3 is explained arithmetically but it might be worth stating explicitly that any positive K leaves relative weights unchanged and that K was chosen to make the smaller weight equal to 1 for interpretability.","section":"Sec. 4.2"},{"comment":"The notation ws,i defines a stratum-level weight but the subscript i suggests unit-level variation; since the weight is constant within stratum s, consider writing w_s to avoid confusion.","section":"Eq. (1)"},{"comment":"The abstract states that PAIR 'corrects these calibration problems' without noting that the hate-speech full-data results in Sec. 5.2 do not show improvement; please qualify the claim to match the presented evidence.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is a simulation study with honest limitations, but the central claim is broader than the evidence: the evaluation target is the original annotation proportion rather than the simulated population target, the bias model is homogeneous, and the simulation is not replicated over annotation draws. These issues are fixable with additional experiments and metric definitions. The paper might also be better framed as a proof-of-concept for survey-style weighting in NLP rather than as an established correction method, given the mixed hate-speech results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PAIR is a sensible, clearly explained transposition of post-stratification from survey sampling to multi-annotator training data: replicate annotations from underrepresented groups so the training set matches population proportions. The paper is worth reading for that transfer, and the simulation for offensive language is convincing: PAIR-adjusted models track representative-data models on ACB while unbalanced pools drift with bias. The authors are also honest about limitations and ship code and data, which helps.\n\nWhat is new: earlier reweighting and resampling methods in fairness corrected class or observation-level feature imbalance; PAIR targets annotator group composition, which is the right target for subjective tasks. The deterministic replication step is simple and interpretable, and the discussion of when weighting helps—only when the weighting variable correlates with the target—is correct and grounded.\n\nSoft spots, in order.\n\n1. The hate speech results do not support the abstract's general claim. On the full HS set, PAIR-adjusted models are not better-calibrated than non-representative ones; the improvement appears only on the difficult-tweet subset. The abstract says PAIR corrects calibration problems without that qualification. That overstates.\n\n2. The evaluation target is the original pi from the 12-annotator sample, not the simulated population probability (pA+pB)/2. With truncation at 0 and 1 these differ at the boundaries, so the ACB numbers measure distance to a slightly different quantity than the one the simulation claims. The comparison across datasets is still fair, but the headline 'calibration to population' is not exactly what is measured.\n\n3. The homogeneous additive bias model—one beta per group, uniform across tweets—is doing a lot of work. The sensitivity analysis filters to ambiguous tweets but keeps the same beta; it never tests heterogeneous or content-dependent bias. Under heterogeneous bias, marginal replication weights need not recover the conditional label distribution. This is the real open question for real annotator pools. The authors flag it, but the paper's strongest evidence is inside that assumption.\n\n4. Single simulation draw per condition, acknowledged; five seeds cover model variation, not annotation sampling variation.\n\nThe reweighting-to-population improvement is partly by design, but that is fine for a simulation study; the useful result is that model-level calibration follows. Don't treat that as a flaw.\n\nBottom line: solid initial demonstration, honest limitations, but the abstract needs to be narrowed, and a revision should add repetition over simulation draws and at least one heterogeneous-bias scenario. I'd send it to a serious referee and come back with those asks.","headline":"PAIR is a clean transfer of post-stratification from survey sampling to annotator composition; the offensive-language simulation is convincing, but the paper overclaims hate speech and never tests heterogeneous bias.","tokens_in":745,"tokens_out":798,"would_cite":true,"duration_ms":36723,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PAIR replication fixes calibration skew from biased annotators","keywords":["PAIR","calibration","annotator bias","post-stratification","data replication","hate speech detection","offensive language detection","simulation study"],"falsifier":"Train pairs of models on the same texts and same annotator pool, but with weights computed from two different sets of characteristics; if calibration does not improve when the weighted characteristics are the ones that actually drive disagreement, the replication step is not doing the work. A concrete version: take an existing multi-annotator dataset with rich demographic metadata, sub-sample the annotators to create known imbalance, apply PAIR, and measure test-set ACB on held-out items whose disagreement was driven by annotator identity rather than by the single additive shift.","tokens_in":1359,"feed_emoji":"🎯","tokens_out":3933,"duration_ms":77411,"temperature":0.7,"pith_summary":"Models trained on crowdsourced annotations can inherit the blind spots of the people who label the data, especially for subjective tasks like offensive language and hate speech detection. This paper argues that poorly calibrated predictions often trace back to an annotator pool that does not mirror the target population, and proposes PAIR, a post-processing fix that requires no additional annotations: replicate annotations from underrepresented annotator groups until the training data matches population proportions. In simulation studies built from 3,000 tweets with two simulated annotator types and bias offsets from 5 to 30 percentage points, models trained on non-representative pools show calibration error (Absolute Calibration Bias) that grows with the bias, while accuracy stays roughly flat. Applying PAIR brings calibration close to models trained on a representative pool, and the improvement is clearest on ambiguous tweets, where calibration matters most.","feed_headline":"PAIR replication fixes calibration skew from biased annotators","feed_subtitle":"Simulation shows models trained on PAIR-adjusted data match representative-pool calibration.","key_machinery":"The central object is the PAIR algorithm, a post-stratification weighting scheme that constructs a pseudo-population from the training data. Each annotator stratum $s$ gets weight $w_{s,i} = P_s / S_s$, where $P_s$ is the population share and $S_s$ the sample share; weights are normalized by a constant $K$ and each annotation is deterministically replicated $\\mathrm{round}(w_i^{\\mathrm{normalized}}) - 1$ times. The algorithm assumes access to annotator-level characteristics, population benchmark shares, and a simulated two-type annotator model (Eqs. 4--5) with bias $\\beta$ shifting the probability of a positive label. The evaluation load-bearing piece is the Absolute Calibration Bias (ACB), which compares model predicted probabilities to the annotator-agreement proportion $p_i$, avoiding binning and giving a direct L1 view of miscalibration.","core_discovery":"On the paper's own terms, PAIR establishes that a survey-style post-stratification adjustment applied to annotations, rather than to people, transfers the benefits of representative training data to models trained on skewed annotator pools. For offensive language, models trained on the PAIR-adjusted dataset track the representative dataset's Absolute Calibration Bias across every bias level, while both unbalanced datasets show higher ACB that rises with $\\beta$. For hate speech, the full-set results are noisier because the class is rare, but restricting to difficult tweets with $0.4 \\leq p_i \\leq 0.6$ reveals the same pattern: adjusted models perform similarly to representative models and better than non-representative ones on both calibration and F1. The paper also reports that binary accuracy metrics obscure annotator-pool effects that calibration metrics expose, and it closes with a four-step recipe: identify annotator characteristics that matter, collect them alongside annotations, obtain population benchmarks, and use the resulting weights in training.","pith_inferences":["The uniform additive-shift bias model is the hinge: if real annotator disagreement is concentrated on ambiguous items and heterogeneous across annotators, fixed replication weights may under- or over-correct; a natural extension is to estimate $\\beta$ per tweet-difficulty strata and use strata-specific weights.","PAIR is a data-space analogue of importance weighting in the loss; one testable extension is comparing replication to soft weights during training, which would avoid inflating dataset size while achieving the same calibration target.","The same pseudo-population logic could be applied to multi-class and multi-label tasks, where bias might shift different labels in different directions, and to align with intersectional demographic strata.","Because PAIR corrects calibration rather than accuracy, it is most consequential for downstream systems that threshold confidence scores, such as content moderation triage systems."],"forward_implications":["Models trained on non-representative annotator pools carry a calibration penalty that grows with annotator bias; accuracy metrics alone will miss this penalty.","Applying PAIR to skewed training data yields calibration close to a representative dataset's without collecting new annotations, so annotation budgets can be spent on breadth rather than re-collection.","For rare classes such as hate speech, PAIR's benefit is clearest on ambiguous instances, which suggests practitioners should evaluate calibration on difficult subsets rather than only on the full test set.","The four-step recipe (identify influential annotator characteristics, collect them, obtain population benchmarks, weight and replicate) gives an actionable pipeline for reducing annotator-pool bias in subjective NLP tasks."],"supporting_citations":[{"why":"Supplies the 3,000-tweet dataset with 15 annotations per tweet on which the entire simulation is built.","marker":"Kern et al., 2023"},{"why":"Provides the post-stratification methodology that PAIR adapts from survey statistics.","marker":"Bethlehem et al., 2011"},{"why":"Gives the practical weighting and normalization guidance used in the PAIR algorithm.","marker":"Valliant et al., 2013"},{"why":"Offers the resampling and reweighting approach that PAIR extends to balance annotator characteristics rather than class labels.","marker":"Calders et al., 2009"},{"why":"Provides the data pre-processing framework for classification without discrimination that PAIR adapts.","marker":"Kamiran and Calders, 2012"},{"why":"Shows that annotator beliefs and identities bias toxic language detection, motivating the need for population-aligned annotation pools.","marker":"Sap et al., 2022"},{"why":"Establishes the theory that weighting reduces bias only when the weighting characteristics correlate with annotations, a condition PAIR relies on.","marker":"Eckman et al., 2024"},{"why":"Forms the basis for the Absolute Calibration Bias metric that avoids binning and gives a direct L1 calibration view.","marker":"Roelofs et al., 2022"}],"fun_headline_variants":["PAIR adjusts training data to fix annotator-bias calibration","Replicate underrepresented annotations to realign model calibration","Simulation shows PAIR restores representative-pool calibration","PAIR post-hoc annotation replication improves calibration, not accuracy","Match population views by replicating rare annotator responses"],"cache_read_input_tokens":14976,"weakest_assumption_plain":"The whole method rests on the assumption that annotator differences can be captured by a single shift in the probability of a positive label that is the same for every item; if the characteristics that drive disagreement vary from item to item or interact with context, the replication weights calculated from this simple model may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["PAIR adjusts training data to fix annotator-bias calibration","Replicate underrepresented annotations to realign model calibration","Simulation shows PAIR restores representative-pool calibration","PAIR post-hoc annotation replication improves calibration, not accuracy","Match population views by replicating rare annotator responses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1734,"prompt_tokens":839,"completion_tokens":895,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":816}},"tokens_in":455,"tokens_out":895,"duration_ms":9203,"temperature":1.0,"reasoning_tokens":816,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:18.369349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train pairs of models on the same texts and same annotator pool, but with weights computed from two different sets of characteristics; if calibration does not improve when the weighted characteristics are the ones that actually drive disagreement, the replication step is not doing the work. A concrete version: take an existing multi-annotator dataset with rich demographic metadata, sub-sample the annotators to create known imbalance, apply PAIR, and measure test-set ACB on held-out items whose disagreement was driven by annotator identity rather than by the single additive shift.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the post-stratification methodology that PAIR adapts from survey statistics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the practical weighting and normalization guidance used in the PAIR algorithm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the theory that weighting reduces bias only when the weighting characteristics correlate with annotations, a condition PAIR relies on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Forms the basis for the Absolute Calibration Bias metric that avoids binning and gives a direct L1 calibration view."}],"review_version":1}