{"id":"c2646979-9614-4ac0-838f-4f33f6419ead","arxiv_id":"2411.12147","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Ensembling pretrained language models with varied layers and anisotropy removal, then taking the standard deviation of their continuous relatedness scores, predicts human annotator disagreement in the CoMeDi shared task better than discrete-label aggregation metrics.","lead":"This shared-task paper uses ensembles of pretrained language models, treated as virtual annotators, to predict human annotator disagreement for word-in-context relatedness judgments, and finds that the standard deviation of continuous similarity scores across model versions tracks human disagreement better than metrics on discrete labels. If it generalizes, this offers a training-free way to flag ambiguous annotations and estimate disagreement in lexical semantics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Subtask 2 claim is vulnerable to a mean-score confound: STD across model manipulations may correlate with human disagreement only because both peak at intermediate relatedness; no partial-correlation or extremeness-baseline control is reported.","rationale":"The reader's CONDITIONAL verdict is well-calibrated, and I agree with the overall assessment. The specific weakest point is not just that model variance might not generalize to human variance, but that the observed development-set correlation may be an artifact of the mean score. Human disagreement in ordinal tasks is typically U-shaped relative to the scale midpoint, while model-relatedness scores have a bounded range, so variance across model manipulations is also likely highest near the midpoint. Without controlling for μ, the STD measure could simply be a nonlinear transform of the mean. The paper does not report any such control, nor significance tests; the post-evaluation test scores are competitive, which is real evidence of empirical utility, but they do not establish the mechanism claimed in the abstract. The proposed partial-correlation test is cheap and decisive: it can be run on the already-collected development predictions and would either rescue the models-as-annotators interpretation or reduce the claim to a known scale-extremeness effect. I therefore keep the reader's CONDITIONAL verdict, with the explicit condition that this control be reported.","tokens_in":10640,"tokens_out":4757,"duration_ms":52297,"concrete_test":"On the CoMeDi development data, for each item compute the four model-configuration relatedness scores, their mean μ and STD σ, and the human disagreement score. (1) Compute Spearman ρ(σ, disagreement). (2) Compute the partial Spearman correlation between σ and disagreement controlling for μ (residualize both on μ, then correlate residuals). (3) Compare with a trivial extremeness baseline, ρ(|μ − midscale|, disagreement), and with a null in which configuration labels are permuted to break any model-specific signal while preserving μ. If the partial correlation falls to near zero or the extremeness baseline matches or exceeds the STD correlation, the 'models as annotators' claim is not supported; the result should be reframed as a mean-score artifact. Report bootstrap confidence intervals for all three quantities.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline Subtask 2 result (Abstract; §5.2, Figure 3) is that the standard deviation (STD) of continuous relatedness scores across model manipulations correlates with human disagreement better than discrete-label measures (MPD, VR). The load-bearing assumption is that this spread reflects something like annotator disagreement. The paper never tests the simpler explanation: in ordinal rating data, human disagreement is typically maximal near the middle of the scale and minimal at the extremes, and model relatedness scores are also bounded, so their variance across layers and anisotropy treatments may be largest for intermediate mean scores. If STD and human disagreement are both driven by the item's mean relatedness, the reported correlation is a confound rather than evidence for 'models as annotators'. The paper's own Limitations (§7) concedes that the model is trained on the median annotation, not individual annotations, further weakening the annotator-simulation analogy. No significance tests accompany Figure 3, and the best measure and model groups are selected on the development set, so the comparison may be optimistically selected. A partial-correlation control is needed before the central claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the JuniperLiu system for the CoMeDi shared task. It frames Subtask 1 (median judgment classification) and Subtask 2 (disagreement prediction) as the estimation of the mean and variance of a hypothetical Gaussian judgment distribution. The system combines threshold-based and MLP-based probes of pretrained language models with anisotropy-removal techniques, and for Subtask 2 it treats model manipulations (model, layer, anisotropy treatment) as virtual annotators. The paper compares three disagreement measures—standard deviation (STD) of continuous relatedness scores, mean pairwise absolute judgment differences (MPD), and variation ratio (VR)—and reports that STD correlates better with human disagreement than the discrete-label measures. Post-evaluation ensembling configurations are reported to reach near-top test performance on several languages.","tokens_in":10814,"tokens_out":5604,"duration_ms":60782,"significance":"If the headline correlation is real and not an artifact of mean relatedness, the result has practical value: a simple, unsupervised ensemble-spread statistic that predicts annotator disagreement across languages would be a useful baseline and proxy for lexical-semantics annotation. The paper's strengths include the 500-run development study, the multilingual coverage across seven languages, the explicit statement of limitations, and the plan to release code. The central claim, however, currently lacks the controls needed to distinguish a genuine model-variance signal from a trivial mean-score confound, and the absence of significance testing makes the measure comparisons difficult to evaluate.","major_comments":[{"comment":"The central claim that STD of continuous relatedness scores 'correlates with human disagreement annotations' is not protected against the mean-score confound. For ordinal judgment scales, human disagreement is typically highest for items near the middle of the scale and lowest at the extremes; bounded model similarity scores will also show larger across-manipulation variance at intermediate mean scores. The reported Spearman correlation between STD and human disagreement may therefore be driven by the common dependence of both variables on the item's mean score. I ask for a partial correlation controlling for the mean relatedness score (and, if possible, for the label mean), reported per language and overall, or for an extremeness-matched baseline analysis. Without this control, the conclusion that STD captures disagreement beyond a mean artifact is not established.","section":"§5.2, Figure 3"},{"comment":"The measure and model-group comparisons are selected on the development set, yet Figure 3 and Table 3 do not report the variance of the 500 runs or any significance tests. Only the top five results of each ensemble strategy are shown, and the post-evaluation 'ensembling*' method uses language-specific groups chosen on the development set; the test-set advantage over 'ensembling' (0.220 vs 0.205 mean over languages in Table 2) is small and could reflect selection bias. Please report the full distribution of Spearman's rho over the 500 runs (mean, standard deviation, confidence intervals), paired significance tests for STD vs MPD/VR and for homo vs hete/mixed, and a correction for multiple comparisons or an out-of-sample validation of the selection.","section":"§5.2, Table 3 and §4.4"},{"comment":"The 'models as annotators' framing is load-bearing for the paper's interpretation, but the Limitations section concedes that the model is trained on the median annotation rather than individual annotations, which is inconsistent with treating model outputs as simulated annotators. The STD claim does not require this equivalence to be true, but if the paper wants to maintain the framing it should provide an item-level test that model-manipulation variance tracks human annotator variance (for example, by comparing predicted score distributions to human score distributions or by predicting individual judgments), or explicitly downgrade the framing to a metaphor and state that the contribution is only the empirical ranking of the three aggregation measures.","section":"§3.3 and §7"}],"minor_comments":[{"comment":"'Spearsman'r' is a typo for Spearman's rho, and the x-axis label 'mixed run' should be defined: is each point one of 500 random subsets, and are the runs sorted by result?","section":"Figure 3"},{"comment":"The abbreviation MPD is written as 'MDP' in Figure 3 and in the text of §5.2; please unify the spelling.","section":"Figure 3 and §5.2"},{"comment":"The row 'ours Thr* (XLM-R-B)' is confusing because §4.3 states that the language-specific selection uses BERT-multi-base for Chinese and Russian and LERT-base-chinese for Norwegian; the table legend should explain which model and layer are used for each language.","section":"Table 1 and §4.3"},{"comment":"The letter scheme (A–D, h–k, X–W) is introduced only in Appendix 10.2; please define it at the first mention of Table 6 and state explicitly that Table 6 corresponds to the 'ensembling*' rows in Table 2.","section":"Tables 6 and 7"},{"comment":"The sentence 'The selector employs the Nelder-Mead method to optimize bin edges for Krippendorff's α' needs a brief explanation of how α is computed during bin optimization, since the threshold selector is not described in the baseline reference alone.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"This is a shared-task system description rather than a full research paper, and the central empirical claim is interesting but currently under-supported. The key risk is the mean-score confound in the STD claim; if the authors cannot supply the partial-correlation control, the claim should be substantially softened. The selection-on-development-set issue also needs to be addressed with confidence intervals or held-out validation before the comparison between measures can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a system description from the CoMeDi shared task, and it's a decent one. The genuinely new piece is the Subtask 2 comparison: across 500 random model subsets, the standard deviation of continuous relatedness scores (STD) beats discrete-label agreement metrics (MPD, VR) for correlating with human disagreement. The development study is systematic, and the post-eval test scores sit near the top of the leaderboard. If that comparison holds up, it's a practical, training-free proxy for disagreement detection, useful for annotation quality control.\n\nThe paper is honest about its own limits. The limitations section admits that the model is trained on the median annotation rather than individual judgments, which undercuts the 'models as annotators' analogy, and the code is only promised. The Gaussian mean/variance framing is decorative, not load-bearing.\n\nThe soft spot that matters: STD may correlate with human disagreement only because both are driven by the mean relatedness score. Human disagreement tends to peak in the middle of an ordinal scale, and model score variance across layers and anisotropy treatments likely also peaks for intermediate means. The paper never partials out mean score or tests an extremeness baseline. That is a genuine confound for the central claim. It doesn't kill the paper—the comparison to MPD and VR is still interesting, and those discrete metrics probably have a similar issue—but the 'models as annotators' interpretation should be tempered until the confound is addressed. No significance tests anywhere either, and the language-specific ensemble groups were selected on the dev set, so the test numbers are optimistically chosen.\n\nWho gets value: people working on annotator disagreement, uncertainty estimation for lexical semantics, and shared-task practitioners. It's a legitimate empirical contribution, not a theoretical one. I'd send it to peer review as a system report, but I'd ask the authors to add a mean-score partial-correlation control and release the code. The central claim is plausible, not yet proven.\n\nRecommendation: engage with it, but treat the headline result as provisional until the confound check is done.","headline":"A genuinely useful shared-task write-up with one interesting empirical comparison, but the headline disagreement result needs a mean-score control before I'd trust the 'models as annotators' framing.","tokens_in":11353,"tokens_out":1966,"would_cite":true,"duration_ms":21474,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the standard deviation of continuous relatedness scores across model manipulations predicts human annotator disagreement better than discrete-label disagreement metrics.","keywords":["annotator disagreement","lexical semantics","word-in-context relatedness","uncertainty estimation","model ensembling","anisotropy removal","continuous relatedness scores"],"falsifier":"A concrete falsifier: find a word pair where the model ensemble shows high standard deviation in continuous relatedness scores but human annotators nearly agree, or low model spread where human disagreement is high; if such mismatches are common, or if the reported Spearman correlation between model STD and human disagreement drops to chance on a held-out language or dataset, the proxy fails.","tokens_in":10392,"feed_emoji":"📊","tokens_out":9008,"duration_ms":88521,"temperature":0.7,"pith_summary":"This paper argues that annotator disagreement about the relatedness of a word across two contexts can be estimated without training on disagreement labels: vary a small set of pretrained language models, gather their continuous relatedness scores, and take the standard deviation. In the shared task's Subtask 2, this model-spread measure correlates with human disagreement better than mean pairwise difference (MPD) or variation ratio (VR) computed from discretized labels. The win matters because continuous scores preserve information that binning into labels throws away, and because a cheap ensemble-spread proxy could replace expensive human disagreement annotation. The paper wraps the two subtasks into one Gaussian picture: the median or majority judgment is the mean of an assumed judgment distribution, and disagreement is its variance, with each model manipulation playing the role of a virtual annotator.","feed_headline":"Model spread predicts human disagreement in word meaning","feed_subtitle":"Word-in-context relatedness scores from varied pretrained models track annotator disagreement without training on labels.","key_machinery":"The central object is the 'models as annotators' ensemble: 64 configurations formed from four pretrained models (XLM-RoBERTa-base, XLM-RoBERTa-large, BERT-multi-base, Llama-7B), four layer levels, and four anisotropy treatments (none, standardization, centering, all-but-the-top). Each configuration yields a continuous relatedness score and, by a Nelder-Mead threshold selector, a discrete 1–4 label. The paper samples 500 random subsets of four configurations for each ensemble strategy, then compares three disagreement measures: STD on continuous scores, and MPD and VR on discrete labels. The Gaussian framing (judgments are treated as drawn from $N(\\mu, \\sigma^2)$) is what lets the consensus subtask and the disagreement subtask be handled as estimation of the mean and the variance of the same judgment distribution.","core_discovery":"On its own terms, the paper's central discovery is that model manipulations can double as annotators for disagreement: using STD on continuous relatedness scores across model manipulations outperforms MPD and VR on aggregated discrete labels in predicting human disagreement. The authors report this on the development set across languages, select STD as their final measure, and show that their ensembling results on the test set match top evaluation-phase performance. The explanation offered is that discrete labels are a lossy discretization of an underlying continuous score, so aggregations of labels lose the precision needed to reflect how much annotators diverge; homogeneous manipulations of one strong model also beat heterogeneous model mixes, aligning with dropout-style stochastic uncertainty.","pith_inferences":["If the model-spread proxy generalizes, annotator disagreement for lexical semantics could be produced from model outputs alone at near-zero marginal annotation cost, and the same recipe might transfer to other annotation tasks where disagreement is a signal.","The paper leaves implicit that the 64-configuration spread only helps if the manipulated dimensions (layers, anisotropy) align with the dimensions along which human judgments vary; a direct test is to apply the STD measure to a different semantic dataset with known disagreement and see whether the Spearman correlation persists.","A practical extension suggested by the results is to use a single large decoder-only model, vary its layers and anisotropy corrections, and use score spread as an unsupervised disagreement signal, with no fine-tuning on disagreement labels required."],"forward_implications":["Human disagreement on word-in-context relatedness can be approximated without collecting disagreement labels: the standard deviation of continuous scores from a varied model ensemble is enough.","Continuous relatedness scores should be kept rather than discretized into labels when the goal is predicting disagreement, because label aggregation loses precision and consistently underperforms.","Homogeneous manipulations of a single strong model (layer and anisotropy variations) provide a better disagreement signal than mixing different models, supporting a dropout-style interpretation of model stochasticity.","Anisotropy removal is a significant ingredient for the consensus-judgment subtask, with standardization performing best across layers.","The mean-variance Gaussian framing connects the two subtasks, so inferring the consensus label and inferring disagreement are two moments of the same judgment distribution."],"supporting_citations":[{"why":"Defines the shared task, its two subtasks, the multilingual data, and the baseline threshold selector the authors build on.","marker":"(Schlechtweg et al., 2025)"},{"why":"Frames annotator disagreement as uncertainty estimation and supplies the variation-ratio measure and the dropout analogy.","marker":"(Gal, 2016)"},{"why":"Justifies the ensembling approach as a simple and scalable way to estimate predictive uncertainty.","marker":"(Lakshminarayanan et al., 2017)"},{"why":"Documents the anisotropy of contextual embeddings that the paper corrects before scoring.","marker":"(Ethayarajh, 2019)"},{"why":"Contributes the all-but-the-top postprocessing used as one anisotropy-removal manipulation.","marker":"(Mu and Viswanath, 2018)"},{"why":"Supplies the standardization and rogue-dimension removal used as another manipulation.","marker":"(Timkey and van Schijndel, 2021)"},{"why":"Establishes the word-in-context relatedness task and the threshold-based probe framework the system adapts.","marker":"(Pilehvar and Camacho-Collados, 2019)"},{"why":"Provides the prompt-based method for extracting lexical representations from decoder-only LLMs and links ambiguity to uncertainty.","marker":"(Liu and Liu, 2023)"}],"fun_headline_variants":["Model spread predicts annotator disagreement in word meaning","Models as virtual annotators: spread beats labels for disagreement","Continuous model scores forecast human disagreement on word senses","STD of model scores mirrors human disagreement in lexical semantics","Virtual annotators: model variance signals human disagreement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the variance across a small set of pretrained model manipulations mirrors the variance of human annotators — that model disagreement reflects the same ambiguities in word meaning that make people disagree, not implementation quirks of the chosen models and layers.","fun_headline_variants_meta":{"raw":{"variants":["Model spread predicts annotator disagreement in word meaning","Models as virtual annotators: spread beats labels for disagreement","Continuous model scores forecast human disagreement on word senses","STD of model scores mirrors human disagreement in lexical semantics","Virtual annotators: model variance signals human disagreement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000378,"raw_usage":{"total_tokens":1949,"prompt_tokens":824,"completion_tokens":1125,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":1060}},"tokens_in":440,"tokens_out":1125,"duration_ms":9292,"temperature":1.0,"reasoning_tokens":1060,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:51:41.813189+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: find a word pair where the model ensemble shows high standard deviation in continuous relatedness scores but human annotators nearly agree, or low model spread where human disagreement is high; if such mismatches are common, or if the reported Spearman correlation between model STD and human disagreement drops to chance on a held-out language or dataset, the proxy fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames annotator disagreement as uncertainty estimation and supplies the variation-ratio measure and the dropout analogy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the anisotropy of contextual embeddings that the paper corrects before scoring."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the all-but-the-top postprocessing used as one anisotropy-removal manipulation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standardization and rogue-dimension removal used as another manipulation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the word-in-context relatedness task and the threshold-based probe framework the system adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prompt-based method for extracting lexical representations from decoder-only LLMs and links ambiguity to uncertainty."}],"review_version":1}