Pith. sign in

REVIEW 3 major objections 5 minor 42 references

JuniperLiu at CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the standard deviation of continuous relatedness scores across model manipulations predicts human annotator disagreement better than discrete-label disagreement metrics.

desk verdict A genuinely useful shared-task write-up with one interesting empirical comparison, but the headline disagreement result needs a mean-score control before I'd trust the 'models as annotators' framing. read the letter →

arxiv 2411.12147 v2 pith:ADO4HYM3 submitted 2024-11-19 cs.CL

Zhu Liu , Zhen Hu , Ying Liu This is my paper · ORCID
classification cs.CL
keywords annotatordisagreementlexicalsemanticsword-in-contextrelatednessuncertaintyestimationmodelensemblinganisotropyremovalcontinuousscores
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that annotator disagreement about the relatedness of a word across two contexts can be estimated without training on disagreement labels: vary a small set of pretrained language models, gather their continuous relatedness scores, and take the standard deviation. In the shared task's Subtask 2, this model-spread measure correlates with human disagreement better than mean pairwise difference (MPD) or variation ratio (VR) computed from discretized labels. The win matters because continuous scores preserve information that binning into labels throws away, and because a cheap ensemble-spread proxy could replace expensive human disagreement annotation. The paper wraps the two subtasks into one Gaussian picture: the median or majority judgment is the mean of an assumed judgment distribution, and disagreement is its variance, with each model manipulation playing the role of a virtual annotator.

What carries the argument

The central object is the 'models as annotators' ensemble: 64 configurations formed from four pretrained models (XLM-RoBERTa-base, XLM-RoBERTa-large, BERT-multi-base, Llama-7B), four layer levels, and four anisotropy treatments (none, standardization, centering, all-but-the-top). Each configuration yields a continuous relatedness score and, by a Nelder-Mead threshold selector, a discrete 1–4 label. The paper samples 500 random subsets of four configurations for each ensemble strategy, then compares three disagreement measures: STD on continuous scores, and MPD and VR on discrete labels. The Gaussian framing (judgments are treated as drawn from $N(\mu, \sigma^2)$) is what lets the consensus subtask and the disagreement subtask be handled as estimation of the mean and the variance of the same judgment distribution.

What would settle it

A concrete falsifier: find a word pair where the model ensemble shows high standard deviation in continuous relatedness scores but human annotators nearly agree, or low model spread where human disagreement is high; if such mismatches are common, or if the reported Spearman correlation between model STD and human disagreement drops to chance on a held-out language or dataset, the proxy fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that model manipulations can double as annotators for disagreement: using STD on continuous relatedness scores across model manipulations outperforms MPD and VR on aggregated discrete labels in predicting human disagreement. The authors report this on the development set across languages, select STD as their final measure, and show that their ensembling results on the test set match top evaluation-phase performance. The explanation offered is that discrete labels are a lossy discretization of an underlying continuous score, so aggregations of labels lose the precision needed to reflect how much annotators diverge; homogeneous manipulations of one strong model also beat heterogeneous model mixes, aligning with dropout-style stochastic uncertainty.

Load-bearing premise

The load-bearing premise is that the variance across a small set of pretrained model manipulations mirrors the variance of human annotators — that model disagreement reflects the same ambiguities in word meaning that make people disagree, not implementation quirks of the chosen models and layers.

Editorial extensions

If this is right

  • Human disagreement on word-in-context relatedness can be approximated without collecting disagreement labels: the standard deviation of continuous scores from a varied model ensemble is enough.
  • Continuous relatedness scores should be kept rather than discretized into labels when the goal is predicting disagreement, because label aggregation loses precision and consistently underperforms.
  • Homogeneous manipulations of a single strong model (layer and anisotropy variations) provide a better disagreement signal than mixing different models, supporting a dropout-style interpretation of model stochasticity.
  • Anisotropy removal is a significant ingredient for the consensus-judgment subtask, with standardization performing best across layers.
  • The mean-variance Gaussian framing connects the two subtasks, so inferring the consensus label and inferring disagreement are two moments of the same judgment distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the model-spread proxy generalizes, annotator disagreement for lexical semantics could be produced from model outputs alone at near-zero marginal annotation cost, and the same recipe might transfer to other annotation tasks where disagreement is a signal.
  • The paper leaves implicit that the 64-configuration spread only helps if the manipulated dimensions (layers, anisotropy) align with the dimensions along which human judgments vary; a direct test is to apply the STD measure to a different semantic dataset with known disagreement and see whether the Spearman correlation persists.
  • A practical extension suggested by the results is to use a single large decoder-only model, vary its layers and anisotropy corrections, and use score spread as an unsupervised disagreement signal, with no fine-tuning on disagreement labels required.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports the JuniperLiu system for the CoMeDi shared task. It frames Subtask 1 (median judgment classification) and Subtask 2 (disagreement prediction) as the estimation of the mean and variance of a hypothetical Gaussian judgment distribution. The system combines threshold-based and MLP-based probes of pretrained language models with anisotropy-removal techniques, and for Subtask 2 it treats model manipulations (model, layer, anisotropy treatment) as virtual annotators. The paper compares three disagreement measures—standard deviation (STD) of continuous relatedness scores, mean pairwise absolute judgment differences (MPD), and variation ratio (VR)—and reports that STD correlates better with human disagreement than the discrete-label measures. Post-evaluation ensembling configurations are reported to reach near-top test performance on several languages.

Significance. If the headline correlation is real and not an artifact of mean relatedness, the result has practical value: a simple, unsupervised ensemble-spread statistic that predicts annotator disagreement across languages would be a useful baseline and proxy for lexical-semantics annotation. The paper's strengths include the 500-run development study, the multilingual coverage across seven languages, the explicit statement of limitations, and the plan to release code. The central claim, however, currently lacks the controls needed to distinguish a genuine model-variance signal from a trivial mean-score confound, and the absence of significance testing makes the measure comparisons difficult to evaluate.

major comments (3)
  1. [§5.2, Figure 3] The central claim that STD of continuous relatedness scores 'correlates with human disagreement annotations' is not protected against the mean-score confound. For ordinal judgment scales, human disagreement is typically highest for items near the middle of the scale and lowest at the extremes; bounded model similarity scores will also show larger across-manipulation variance at intermediate mean scores. The reported Spearman correlation between STD and human disagreement may therefore be driven by the common dependence of both variables on the item's mean score. I ask for a partial correlation controlling for the mean relatedness score (and, if possible, for the label mean), reported per language and overall, or for an extremeness-matched baseline analysis. Without this control, the conclusion that STD captures disagreement beyond a mean artifact is not established.
  2. [§5.2, Table 3 and §4.4] The measure and model-group comparisons are selected on the development set, yet Figure 3 and Table 3 do not report the variance of the 500 runs or any significance tests. Only the top five results of each ensemble strategy are shown, and the post-evaluation 'ensembling*' method uses language-specific groups chosen on the development set; the test-set advantage over 'ensembling' (0.220 vs 0.205 mean over languages in Table 2) is small and could reflect selection bias. Please report the full distribution of Spearman's rho over the 500 runs (mean, standard deviation, confidence intervals), paired significance tests for STD vs MPD/VR and for homo vs hete/mixed, and a correction for multiple comparisons or an out-of-sample validation of the selection.
  3. [§3.3 and §7] The 'models as annotators' framing is load-bearing for the paper's interpretation, but the Limitations section concedes that the model is trained on the median annotation rather than individual annotations, which is inconsistent with treating model outputs as simulated annotators. The STD claim does not require this equivalence to be true, but if the paper wants to maintain the framing it should provide an item-level test that model-manipulation variance tracks human annotator variance (for example, by comparing predicted score distributions to human score distributions or by predicting individual judgments), or explicitly downgrade the framing to a metaphor and state that the contribution is only the empirical ranking of the three aggregation measures.
minor comments (5)
  1. [Figure 3] 'Spearsman'r' is a typo for Spearman's rho, and the x-axis label 'mixed run' should be defined: is each point one of 500 random subsets, and are the runs sorted by result?
  2. [Figure 3 and §5.2] The abbreviation MPD is written as 'MDP' in Figure 3 and in the text of §5.2; please unify the spelling.
  3. [Table 1 and §4.3] The row 'ours Thr* (XLM-R-B)' is confusing because §4.3 states that the language-specific selection uses BERT-multi-base for Chinese and Russian and LERT-base-chinese for Norwegian; the table legend should explain which model and layer are used for each language.
  4. [Tables 6 and 7] The letter scheme (A–D, h–k, X–W) is introduced only in Appendix 10.2; please define it at the first mention of Table 6 and state explicitly that Table 6 corresponds to the 'ensembling*' rows in Table 2.
  5. [§4.2] The sentence 'The selector employs the Nelder-Mead method to optimize bin edges for Krippendorff's α' needs a brief explanation of how α is computed during bin optimization, since the threshold selector is not described in the baseline reference alone.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the Subtask 2 STD measure is an empirical ensemble-spread statistic compared against external human disagreement labels, not fitted to them.

full rationale

The central Subtask 2 claim (Abstract; §5.2, Figure 3) is that the standard deviation of continuous relatedness scores across model manipulations correlates with human disagreement. This is not circular: the STD is computed from pretrained model outputs across layers, anisotropy treatments, and models, then correlated with the external CoMeDi human disagreement labels. No parameter of the STD statistic itself is fitted to those labels. The discrete-label competitors (MPD, VR) do depend on threshold bins fitted for Krippendorff's α, but the continuous relatedness scores used for STD are produced directly from embeddings before that discretization, so the winning measure does not reduce to a fitted quantity. The paper's own framing, 'we treat each model or its manipulation as an annotator' (§3.3), is an analogy that is then tested empirically; the human disagreement labels are not defined in terms of the model STD. The only target-dependent choices are selecting which measure and which model groups to report from the development set (Tables 3, 6, 7); this is ordinary model selection on the dev split, and the reported test-set results (§5, Table 2) remain external evaluations. The self-citations (Liu et al., 2024; Liu and Liu, 2023) are methodological precedents for threshold-based probing and Llama prompt extraction; they are not used as a uniqueness theorem or as the justification that model spread proxies human disagreement. The Limitations section explicitly concedes that Subtask 2 training uses the median label rather than individual annotations, but that is a caveat about the 'models as annotators' analogy, not a definitional identification of model spread with human disagreement. A possible mean-score confound (human disagreement and model variance both peaking at intermediate relatedness) is a validity threat requiring a partial-correlation control, not a circularity. Overall, no load-bearing step reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two domain assumptions: that model manipulation variance mirrors human disagreement, and that cosine similarity from pretrained representations is a valid relatedness proxy. The fitted components are the threshold bins for label mapping, the layer selections, and the language-specific ensemble groups, all tuned on development data. No new entities are introduced.

free parameters (3)
  • threshold bin edges for label mapping = not reported (optimized via Nelder-Mead)
    Optimized to maximize Krippendorff's alpha on training data; used to convert continuous similarity scores to discrete 1-4 labels for Subtask 1 and for the MPD/VR measures in Subtask 2.
  • layer index for representation extraction per model = e.g., layer 10 for XLM-RoBERTa-base, layer 25 for Llama-7B (post-evaluation), layer 11 for the MLP
    Chosen by development-set ablation (Figures 1-2, Appendix 10.1); affects all downstream similarity scores and therefore the ensembling results.
  • language-specific ensemble groups = Table 6 (e.g., Chinese AiX-AkX-AhX-AkW)
    Selected on the development set to maximize Spearman correlation for each language; used in the ensembling* test results.
assumptions (3)
  • domain assumption Annotator scores for a word pair are Gaussian-distributed, p ~ N(mu, sigma^2)
    Section 3.1 introduces this as a natural statistical choice. It is used only as a conceptual framing to unify Subtask 1 (mean) and Subtask 2 (variance); the implemented methods do not estimate a Gaussian, so this assumption is not load-bearing.
  • domain assumption Model manipulations act as virtual annotators whose variance reflects human disagreement
    Section 3.3 and the abstract state that each model or its manipulation is treated as an annotator. This is the load-bearing premise for Subtask 2; the paper provides empirical evidence on CoMeDi but no mechanism-level justification.
  • domain assumption Cosine similarity of contextualized target-word embeddings is a valid proxy for human relatedness
    Implied in the threshold-based method (Sections 3.3 and 4.2); standard in the lexical-semantics literature but treated as given without independent validation in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JuniperLiu at CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements." pith.science (2026). https://pith.science/paper/ADO4HYM3

@misc{pith2026241112147,
  author       = {Pith},
  title        = {Pith review of: JuniperLiu at CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ADO4HYM3}},
  note         = {Machine review of arXiv:2411.12147}
}
read the original abstract

We present the results of our system for the CoMeDi Shared Task, which predicts majority votes (Subtask 1) and annotator disagreements (Subtask 2). Our approach combines model ensemble strategies with MLP-based and threshold-based methods trained on pretrained language models. Treating individual models as virtual annotators, we simulate the annotation process by designing aggregation measures that incorporate continuous relatedness scores and discrete classification labels to capture both majority and disagreement. Additionally, we employ anisotropy removal techniques to enhance performance. Experimental results demonstrate the effectiveness of our methods, particularly for Subtask 2. Notably, we find that standard deviation on continuous relatedness scores among different model manipulations correlates with human disagreement annotations compared to metrics on aggregated discrete labels. The code will be published at https://github.com/RyanLiut/CoMeDi_Solution.

Figures

Figures reproduced from arXiv: 2411.12147 by the authors.

Figure 1
Figure 1. Performance of different types of anisotropy [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Performance of different models as the layer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 20 canonical work pages

  1. [1]

    Anna Aksenova, Ekaterina Gavrishina, Elisei Rykov, and Andrey Kutuzov. 2022. https://aclanthology.org/2022.textgraphs-1.9 R u DSI : Graph-based word sense induction dataset for R ussian . In Proceedings of TextGraphs-16: Graph-based Methods for Natural Language Processing, pages 77--88, Gyeongju, Republic of Korea. Association for Computational Linguistics

    2022

  2. [2]

    Jing Chen, Emmanuele Chersoni, Dominik Schlechtweg, Jelena Prokic, and Chu-Ren Huang. 2023. https://aclanthology.org/2023.lchange-1.10/ ChiWUG : A graph-based evaluation dataset for C hinese lexical semantic change detection . In Proceedings of the 4th International Workshop on Computational Approaches to Historical Language Change, Singapore. Association...

    2023

  3. [3]

    Timothy Chklovski and Rada Mihalcea. 2003. Exploiting agreement and disagreement of human annotators for word sense disambiguation. In Recent Advances in Natural Language Processing

    work page 2003

  4. [4]

    Simone Conia and Roberto Navigli. 2021. Framing word sense disambiguation as a multi-label problem for model-agnostic knowledge integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3269--3275

    work page 2021

  5. [5]

    A Conneau. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116

    arXiv 2019

  6. [6]

    Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55--65, Hong Kong, China. Assoc...

    work page 2019

  7. [7]

    Yarin Gal. 2016. Uncertainty in Deep Learning. Phd thesis, University of Cambridge

    work page 2016

  8. [8]

    Anna H \"a tty, Dominik Schlechtweg, and Sabine Schulte im Walde. 2019. https://doi.org/10.18653/v1/S19-1001 SUR el: A gold standard for incorporating meaning shifts into term extraction . In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (* SEM 2019) , pages 1--8, Minneapolis, Minnesota. Association for Computational Li...

    work page doi:10.18653/v1/s19-1001 2019

Show all 42 references
  1. [9]

    Aminul Islam and Diana Inkpen. 2008. Semantic text similarity using corpus-based word similarity and string similarity. ACM Transactions on Knowledge Discovery from Data, 2(2):10:1--10:25

    2008

  2. [10]

    Edwin T Jaynes. 2003. Probability theory: The logic of science. Cambridge university press

    2003

  3. [11]

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.181 Scaling sentence embeddings with large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3182--31...

    2024 doi

  4. [12]

    Sinan Kurtyigit, Maike Park, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. https://aclanthology.org/2021.acl-long.543/ Lexical Semantic Change Discovery . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1...

    2021

  5. [13]

    Andrey Kutuzov and Lidia Pivovarova. 2021. Rushifteval: a shared task on semantic shift detection for russian. Komp'yuternaya Lingvistika i Intellektual'nye Tekhnologii: Dialog conference

    2021

  6. [14]

    Andrey Kutuzov, Samia Touileb, Petter M hlum, Tita Enstad, and Alexandra Wittemann. 2022. https://aclanthology.org/2022.lrec-1.274 N or D ia C hange: Diachronic semantic change dataset for N orwegian . In Proceedings of the Thirteenth Language Resources and Evaluation Conferen...

    2022

  7. [15]

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30

    2017

  8. [16]

    Zhu Liu, Cunliang Kong, Ying Liu, and Maosong Sun. 2024. https://doi.org/10.18653/v1/2024.findings-acl.866 Fantastic semantics and where to find them: Investigating which layers of generative LLM s reflect lexical semantics . In Findings of the Association for Computational Li...

    2024 doi

  9. [17]

    Zhu Liu and Ying Liu. 2023. Ambiguity meets uncertainty: Investigating uncertainty estimation for word sense disambiguation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3963--3977

    2023

  10. [18]

    Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations

    2018

  11. [19]

    Roberto Navigli. 2008. A structural approach to the automatic adjudication of word sense disagreements. Natural Language Engineering, 14(4):547--573

    2008

  12. [20]

    Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM computing surveys (CSUR), 41(2):1--69

    2009

  13. [21]

    Nelder and Roger Mead

    John A. Nelder and Roger Mead. 1965. A simplex method for function minimization. The Computer Journal, 7(4):308--313

    1965

  14. [22]

    Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of NAACL-HLT, pages 1267--1273

    2019

  15. [23]

    T Pires. 2019. How multilingual is multilingual bert. arXiv preprint arXiv:1906.01502

    2019 arXiv

  16. [24]

    Julia Rodina and Andrey Kutuzov. 2020. https://doi.org/10.18653/v1/2020.coling-main.90 R u S em S hift: a dataset of historical lexical semantic change in R ussian . In Proceedings of the 28th International Conference on Computational Linguistics, pages 1037--1047, Barcelona, ...

    2020 doi

  17. [25]

    Magnus Sahlgren, Amaru Cuba Gyllensten, Fredrik Espinoza, Ola Hamfors, Jussi Karlgren, Fredrik Olsson, Per Persson, Akshay Viswanathan, and Anders Holst. 2016. The gavagai living lexicon. In Proceedings of the Tenth International Conference on Language Resources and Evaluation...

    2016

  18. [26]

    Dominik Schlechtweg. 2023. https://d-nb.info/1285256050 Human and computational measurement of lexical semantic change . Ph.D. thesis, University of Stuttgart, Germany

    2023

  19. [27]

    Dominik Schlechtweg, Pierluigi Cassotti, Bill Noble, David Alfter, Sabine Schulte im Walde, and Nina Tahmasebi. 2024. More DWUGs : Extending and evaluating word usage graph datasets in multiple languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural La...

    2024

  20. [28]

    Dominik Schlechtweg, Tejaswi Choppa, Wei Zhao, and Michael Roth. 2025. The CoMeDi shared task: Median judgment classification & mean disagreement ranking with ordinal word-in-context judgments. In Proceedings of the 1st Workshop on Context and Meaning--Navigating Disagreements...

    2025

  21. [29]

    Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanie Eckmann. 2018. https://doi.org/10.18653/v1/N18-2027 Diachronic usage relatedness ( DUR el): A framework for the annotation of lexical semantic change . In Proceedings of the 2018 Conference of the North A merican Chapt...

    2018 doi

  22. [30]

    Dominik Schlechtweg, Nina Tahmasebi, Simon Hengchen, Haim Dubossarsky, and Barbara McGillivray. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.567 DWUG : A large resource of diachronic word usage graphs in four languages . In Proceedings of the 2021 Conference on Empirical ...

    2021 doi

  23. [31]

    a ndische Universit \

    David Stutz. 2022. Understanding and improving robustness and uncertainty estimation in deep learning. Saarl \"a ndische Universit \"a ts-und Landesbibliothek

    2022

  24. [32]

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818--2826

    2016

  25. [33]

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th Internationa...

    2019

  26. [34]

    William Timkey and Marten van Schijndel. 2021. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4527--4546, Online an...

    2021

  27. [35]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  28. [36]

    Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385--1470

    2021

  29. [37]

    Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, et al. 2022. Uncertainty estimation of transformer predictions for misclassification detection. In Proceedings of th...

    2022

  30. [38]

    Ivan Vuli \'c , Edoardo Maria Ponti, Robert Litschko, Goran Glava s , and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222--7240, Online....

    2020

  31. [39]

    Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg

    Frank D. Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022. https://aclanthology.org/2022.lchange-1.16/ LSCDiscovery : A shared task on semantic change discovery and detection in S panish . In Proceedings of the 3rd International Workshop on Computational Appro...

    2022

  32. [40]

    Hua Zheng, Lei Li, Damai Dai, Deli Chen, Tianyu Liu, Xu Sun, and Yang Liu. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.78 Leveraging word-formation knowledge for C hinese word sense disambiguation . In Findings of the Association for Computational Linguistics: EMNLP ...

    2021 doi

  33. [41]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.