Pith. sign in

REVIEW 3 major objections 4 minor 65 references

Direct training beats inference for spotting subjective arguments

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:11 UTC pith:6NB6JO3P

load-bearing objection Direct subjectivity training beats inference on the ValueEval subset, but the four-annotator target leaves the general claim only partly supported. the 3 major comments →

arxiv 2509.06704 v1 pith:6NB6JO3P submitted 2025-09-08 cs.CL

Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments

Amir Homayounirad , Enrico Liscio , Tong Wang , Catholijn M. Jonker , Luciano C.Siebert This is my paper
classification cs.CL
keywords subjectivity identificationannotator disagreementhuman valuesvalue-laden argumentsmulti-annotator NLPcontrastive learningargumentationlabel variation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that, for arguments carrying human values, the best way to predict whether annotators will disagree is to train a model directly on disagreement labels, not to predict each annotator's value choices and then compare them. On the Touché23-ValueEval dataset, direct subjectivity models reach F1 0.80, while inference-based models stay at 0.36 and random guessing at 0.51. Adding contrastive losses does not improve F1 but produces embeddings that separate subjective from non-subjective arguments more cleanly and reduces the model's tendency to do better only on less subjective values. If the result holds, annotation pipelines and deliberation tools can flag arguments likely to be interpreted differently, prompting extra annotations or moderator questions.

Core claim

On the Touché23-ValueEval dataset, the paper defines an argument-value pair as subjective when at least one of four annotators assigns a different set of values for that pair. It compares three inference-based methods—dedicated per-annotator models, shared embeddings with per-annotator heads, and a single model with annotator IDs—against three direct methods that train binary classifiers on the disagreement labels, two of which add contrastive objectives. The direct methods score far higher (best F1 0.80 with DS-simple on Llama versus 0.36 for the best inference approach and 0.51 random), showing that disagreement is better learned as a label in its own right than reconstructed from predicte

What carries the argument

The central object is the binary subjectivity classifier trained directly on disagreement labels, one per value (DS-simple), with optional auxiliary contrastive losses—triplet loss (supervised) and contrastive tension (unsupervised)—added to binary cross-entropy. The direct label is what carries the argument: it bypasses the multi-label value-prediction task entirely, and the comparison with inference-based methods isolates the effect of the learning objective rather than the architecture.

Load-bearing premise

The entire evaluation rests on defining 'subjective' as disagreement among four specific annotators chosen for high overlap; if those four are too similar to each other, or if their disagreements are mostly noise, the models are learning how that particular group annotates, not a general property of the arguments.

What would settle it

Take the same arguments and collect value annotations from a fresh set of, say, twenty annotators with diverse demographics; if the DS-simple model's subjectivity scores do not predict which items the new group disagrees on, the method has captured the original four annotators' quirks rather than disagreement itself.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A model trained to predict disagreement directly can flag arguments that would benefit from additional annotations before they are sent to more annotators.
  • Annotation projects can keep individual value predictions for interpretability while using a separate direct subjectivity head for accuracy, since the two objectives are not interchangeable.
  • The supervised contrastive variant produces embeddings that separate subjective from non-subjective arguments more cleanly, which can support active learning sampling near the decision boundary.
  • Scaling the model used for direct subjectivity identification (BERT to Llama) improves recall substantially, while the same scaling does not rescue inference-based methods—suggesting the inference bottleneck is structural rather than a matter of model size.
  • The weaker dependence of DS-sup on per-label subjectivity means combined losses yield more consistent performance across values with different disagreement rates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's comparison does not isolate whether the gain comes from the direct label or from the paraphrasing data augmentation used only for DS; a testable follow-up would train DS without augmentation to separate these effects.
  • The strong negative correlation for DS-simple (Llama) suggests the model may excel on values where disagreement is rare by exploiting distributional cues; probing with counterfactual rewrites would test whether it learns generic subjectivity or value-specific surface patterns.
  • The approach could transfer to other subjective annotation tasks, such as hate speech or emotion detection, where per-item disagreement labels exist; the paper demonstrates it only for value-laden arguments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses subjectivity in recognizing human values behind arguments. It formalizes an argument as subjective for a value when at least one annotator assigns a different set of values than the others. Two families of methods are proposed: inferred subjectivity identification (IS), which trains per-annotator value prediction models and derives disagreement from predicted value differences, and direct subjectivity identification (DS), which trains binary classifiers per value to predict disagreement directly. The methods are evaluated on a subset of Touché23-ValueEval consisting of eight level-2 values annotated by four selected annotators, using BERT and Llama models. The best result is DS-simple (Llama) at F1 0.80, well above the random baseline of 0.51, while all IS methods are near or below the random baseline. The paper also investigates supervised and unsupervised contrastive losses, finding that they do not improve F1 but affect embedding structure and the correlation with per-value subjectivity.

Significance. If the central claim holds, the work is a useful contribution to the label-variation literature: it provides a concrete comparison between direct subjectivity classification and inference from per-annotator value prediction, and it identifies a large practical gap between the two. Strengths include the public code release, a fixed test set, standard deviations reported for most BERT experiments, a random baseline, and a systematic comparison of several architectural variants. However, the external validity of the main claim—that the method predicts whether annotators will disagree—is not yet established, because the subjectivity labels are derived from only four annotators selected for high overlap, with no held-out annotator validation. The internal DS-versus-IS comparison is coherent, but the title and abstract promise more than the current experimental design can support.

major comments (3)
  1. [§4.2, App. A.1, §2.2] The DS labels are built from exactly four annotators selected for high annotation overlap. Section 2.2 states the method is “not being reliant on specific individuals or groups,” but the binary subjectivity targets are tied to this particular group. No held-out annotator set is used: a model trained on these four annotators may learn the disagreement pattern of a small, similar group rather than generalizable subjectivity. The paper itself concedes in Section 7 that annotation disagreement can be confounded with noise. I recommend annotator-level cross-validation (e.g., train on three annotators, evaluate on the fourth; or resample annotator subsets) and a report of how stable DS performance is across annotator configurations. In addition, Appendix A.1 says the selection was “the first four annotators who annotated the most,” while Section 4.2 says “highest annotation overlap”; these are
  2. [§5, Table 1, App. B] The abstract and Section 5.1 state that direct subjectivity identification “significantly improves” performance, but no statistical significance test is reported. Appendix B provides means and standard deviations over five runs for most BERT methods, yet there is no paired test (e.g., paired bootstrap, McNemar) or confidence intervals for the comparisons that drive the conclusions (DS vs. IS, DS-simple vs. DS-sup). Moreover, the Llama results, which include the best overall system, are reported without standard deviations. Since the central claim is comparative, the authors should add explicit significance tests or at least report per-seed variation for the Llama experiments.
  3. [§3.1, §7] The operational definition of subjectivity as “at least one annotator assigns a different set of values compared to others” is very sensitive with only four annotators: a single divergent annotation makes the argument subjective. The paper acknowledges in Section 7 that disagreement can be confounded with noise, but it does not attempt to distinguish systematic disagreement from isolated error. This is load-bearing because the DS models are trained to reproduce this binary signal. A robustness analysis varying the disagreement threshold (e.g., requiring two or more annotators to differ) would show whether the reported gains depend on the current dichotomization.
minor comments (4)
  1. [§4.2 vs. App. A.1] The main text reports 9,324 natural language arguments, while Appendix A.1 reports 5,270. Please reconcile the dataset size.
  2. [§8] The Limitations section contains a garbled sentence: “The first two methods from ISV including —ISV-each, ISV-shared, and DS-simple are computationally expensive,” and the names ISV-each/ISV-shared do not match the method names introduced in Section 3. Please correct.
  3. [App. B] The appendix tables use labels DBCV, SBDCV, SBCV, DBV, SBV without mapping them to the method names in Section 3. A legend or renaming would help readers.
  4. [Figure 4] The t-SNE visualizations are illustrative but the claims about embedding separation and grouping would be stronger with a quantitative metric (e.g., silhouette score or nearest-neighbor agreement).

Circularity Check

0 steps flagged

No significant circularity: standard supervised evaluation with externally sourced annotations and a held-out test set.

full rationale

The paper's central claim—that direct subjectivity identification (DS) outperforms inference from value prediction (IS)—rests on a standard supervised learning comparison. Subjectivity labels are derived from the Touché23-ValueEval crowd annotations (Section 4.2), and DS models are trained with binary cross-entropy and optional contrastive losses to predict those labels; IS models are trained to predict per-annotator value labels, and the same operational definition of disagreement is applied to their outputs. Evaluation uses a fixed held-out test set with hyperparameters chosen on validation, and Appendix B reports standard deviations over five runs with different train/validation splits. No fitted parameter is renamed as a prediction, no equation reduces the target to the model's own outputs by construction, and no load-bearing premise is justified only by a self-citation. The selection of four high-overlap annotators and the possible confound between subjectivity and annotation noise are real external-validity limitations, explicitly acknowledged in Section 7 and the Limitations section, but these concern construct validity rather than circularity: the model's target is defined by annotator disagreement, not by the model's predictions. The comparison between DS and IS is therefore self-contained and empirically meaningful.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities. It relies on a specific definition of subjectivity, a small selected subset of annotators and values, and standard transformer assumptions. The main free parameters are the contrastive loss weights and margin, plus data augmentation settings, all chosen by hand on validation. These choices are not fitted to the test set, so they do not inflate circularity, but they do affect how well the results generalize.

free parameters (4)
  • contrastive_loss_weight_alpha = 1.0 (DS-sup), 5.0 (DS-unsup)
    Weight on the auxiliary contrastive loss in the combined loss L = L_BCE + alpha * L_CL; chosen by hand, affects final embeddings and performance.
  • triplet_margin_m = 1.0
    Margin in the triplet loss for DS-sup; enforces minimum separation between positive and negative pairs, set by hand.
  • contrastive_temperature_tau = not reported
    Temperature in the contrastive tension loss for DS-unsup; not listed in the hyperparameter table, likely left at a default value.
  • paraphrase_generation_temperature = 2.0
    Used for data augmentation of the minority class in DS methods (Appendix A.2); impacts the distribution of synthetic samples used in training.
axioms (4)
  • ad hoc to paper An argument is subjective if at least one annotator assigns a different set of values than the others (Section 3.1).
    This binary operationalization is arbitrary; other thresholds (e.g., majority disagreement, variance-based) would change the labels and potentially the results.
  • domain assumption Disagreement among annotators reflects subjectivity and not annotation noise (Section 3.1, acknowledged as a confound in Section 7).
    The paper itself acknowledges that differences in annotation can be confounded with noise. If noise dominates, the trained models are learning noise, not subjectivity.
  • ad hoc to paper The four annotators with highest annotation overlap and the eight most frequently annotated values are representative enough to evaluate subjectivity.
    This selection reduces the dataset and biases toward annotators who agree more, likely understating true subjectivity and limiting generalizability.
  • domain assumption Standard transformer embeddings (BERT, Llama) capture the semantic information needed for subjectivity detection.
    The methods rely on fine-tuned language models without any external knowledge of annotator backgrounds or deliberation context.

pith-pipeline@v1.3.0-alltime-deepseek · 20029 in / 8541 out tokens · 85764 ms · 2026-08-04T23:11:58.348731+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments." pith.science (2026). https://pith.science/paper/6NB6JO3P

@misc{pith2026250906704,
  author       = {Pith},
  title        = {Pith review of: Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NB6JO3P}},
  note         = {Machine review of arXiv:2509.06704}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.

Figures

Figures reproduced from arXiv: 2509.06704 by Amir Homayounirad, Catholijn M. Jonker, Enrico Liscio, Luciano C.Siebert, Tong Wang.

Figure 1
Figure 1. Figure 1: Representation of how the method processes input text to predict values for individual annotators and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representation of how the method processes input text to identify subjectivity for each value label [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Counts and ratio of subjective (S, red) and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of 2D sentence embeddings visu [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 38 canonical work pages · 1 internal anchor

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sai Vamsi Alisetti. 2021. Paraphrase generator with t5

    work page 2021

  4. [4]

    Milad Alshomary, Roxanne El Baff, Timon Gurcke, and Henning Wachsmuth. 2022. https://doi.org/10.18653/v1/2022.acl-long.601 The moral debater: A study on the computational generation of morally framed arguments . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8782--8797, Dublin, Ire...

    work page doi:10.18653/v1/2022.acl-long.601 2022

  5. [5]

    Oscar Araque, Lorenzo Gatti, and Kyriaki Kalimeri. 2020. Moralstrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction. Knowledge-based systems, 191:105184

    work page 2020

  6. [6]

    Taylor, Mark D \'i az, Christopher Michael Homan, Alicia Parrish, Greg Serapio-Garc \'i a, Vinodkumar Prabhakaran, and Ding Wang

    Lora Aroyo, Alex S. Taylor, Mark D \'i az, Christopher Michael Homan, Alicia Parrish, Greg Serapio-Garc \'i a, Vinodkumar Prabhakaran, and Ding Wang. 2023. https://api.semanticscholar.org/CorpusID:259203842 Dices dataset: Diversity in conversational ai evaluation for safety . ArXiv, abs/2306.11247

    Pith/arXiv arXiv 2023

  7. [7]

    Connor Baumler, Anna Sotnikova, and Hal Daum \'e III. 2023. https://doi.org/10.18653/v1/2023.findings-acl.658 Which examples should be multiply annotated? active learning when annotators may disagree . In Findings of the Association for Computational Linguistics: ACL 2023, pages 10352--10371, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-acl.658 2023

  8. [8]

    Trevor J. M. Bench-Capon. 2003. https://doi.org/10.1093/logcom/13.3.429 Persuasion in Practical Argument Using Value-based Argumentation Frameworks . Journal of Logic and Computation, 13(3):429--448

    doi:10.1093/logcom/13.3.429 2003

  9. [9]

    Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Ar \'e chiga, and Tengyu Ma. 2019. https://api.semanticscholar.org/CorpusID:189998981 Learning imbalanced datasets with label-distribution-aware margin loss . In Neural Information Processing Systems

    work page 2019

  10. [10]

    Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylip \"a \"a Hellqvist, and Magnus Sahlgren. 2021. https://api.semanticscholar.org/CorpusID:235613354 Semantic re-tuning with contrastive tension . In International Conference on Learning Representations

    work page 2021

  11. [11]

    J. D. Chandler and S. L. Vargo. 2011. https://doi.org/10.1177/1470593110393713 Contextualization and value-in-context: how context frames exchange . Marketing Theory, 11:35--49

    work page doi:10.1177/1470593110393713 2011

  12. [12]

    Naihao Deng, Xinliang Zhang, Siyang Liu, Winston Wu, Lu Wang, and Rada Mihalcea. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.832 You are what you annotate: Towards better models through annotator representations . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12475--12498, Singapore. Association for Computationa...

    doi:10.18653/v1/2023.findings-emnlp.832 2023

  13. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://api.semanticscholar.org/CorpusID:52967399 Bert: Pre-training of deep bidirectional transformers for language understanding . In North American Chapter of the Association for Computational Linguistics

    2019

  14. [14]

    Edvardsson, B

    B. Edvardsson, B. Tronvoll, and T. Gruber. 2010. https://doi.org/10.1007/s11747-010-0200-y Expanding understanding of service exchange and value co-creation: a social construction approach . Journal of the Academy of Marketing Science, 39:327--339

    work page doi:10.1007/s11747-010-0200-y 2010

  15. [15]

    Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.415 When the majority is wrong: Modeling annotator disagreement for subjective tasks . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6715--6726, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.415 2023

  16. [16]

    Joseph L. Fleiss. 1971. https://api.semanticscholar.org/CorpusID:143544759 Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378--382

    work page 1971

  17. [17]

    Nitesh Goyal, Ian Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. https://arxiv.org/abs/2205.00501 Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation . Preprint, arXiv:2205.00501

    Pith/arXiv arXiv 2022

  18. [18]

    Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Advances in experimental social psychology, volume 47, pages 55--130. Elsevier

    work page 2013

  19. [19]

    Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020. https://api.semanticscholar.org/CorpusID:226237047 Supervised contrastive learning for pre-trained language model fine-tuning . ArXiv, abs/2011.01403

    Pith/arXiv arXiv 2020

  20. [20]

    Amy Gutmann and Dennis F Thompson. 2004. Why deliberative democracy? Princeton University Press

    work page 2004

  21. [21]

    Hadsell, S

    R. Hadsell, S. Chopra, and Y. LeCun. 2006. https://doi.org/10.1109/CVPR.2006.100 Dimensionality reduction by learning an invariant mapping . In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2, pages 1735--1742

    doi:10.1109/cvpr.2006.100 2006

  22. [22]

    Jonathan Haidt and Jesse Graham. 2007. When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize. Social justice research, 20(1):98--116

    work page 2007

  23. [23]

    Joe Hoover, Gwenyth Portillo-Wightman, Leigh Yeh, Shreya Havaldar, Aida Mostafazadeh Davani, Ying Lin, Brendan Kennedy, Mohammad Atari, Zahra Kamel, Madelyn Mendlen, et al. 2020. Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment. Social Psychological and Personality Science, 11(8):1057--1071

    work page 2020

  24. [24]

    Horbel, B

    C. Horbel, B. Popp, H. Woratschek, and B. Wilson. 2016. https://doi.org/10.1080/02642069.2016.1255730 How context shapes value co-creation: spectator experience of sport events . The Service Industries Journal, 36:510--531

    work page arXiv 2016

  25. [25]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    Pith/arXiv arXiv 2021

  26. [26]

    Wormley, and Adam H

    Xiaolei Huang, Alexandra S. Wormley, and Adam H. Cohen. 2022. https://api.semanticscholar.org/CorpusID:248227355 Learning to adapt domain shifts of moral values via instance weighting . Proceedings of the 33rd ACM Conference on Hypertext and Social Media

    work page 2022

  27. [27]

    John Kekes. 1996. The morality of pluralism

    work page 1996

  28. [28]

    Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. 2022. Identifying the human values behind arguments. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4459--4471

    work page 2022

  29. [29]

    Johannes Kiesel, Milad Alshomary, Nailia Mirzakhmedova, Maximilian Heinrich, Nicolas Handke, Henning Wachsmuth, and Benno Stein. 2023. https://doi.org/10.18653/v1/2023.semeval-1.313 S em E val-2023 task 4: V alue E val: Identification of human values behind arguments . In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)...

    doi:10.18653/v1/2023.semeval-1.313 2023

  30. [30]

    Johannes Kiesel et al. 2024. https://doi.org/10.1007/978-3-031-71908-0_14 Overview of Touch \'e 2024: Argumentation Systems . In Experimental IR Meets Multilinguality, Multimodality, and Interaction. 15th International Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science, Berlin Heidelberg New York. Springer

    work page doi:10.1007/978-3-031-71908-0_14 2024

  31. [31]

    Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. https://doi.org/10.18653/v1/2021.acl-long.197 Self-guided contrastive learning for BERT sentence representations . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa...

    doi:10.18653/v1/2021.acl-long.197 2021

  32. [32]

    Milkowski, and Przemyslaw Kazienko

    Jan Kocoń, Marcin Gruza, Julita Bielaniewicz, Damian Grimling, Kamil Kanclerz, P. Milkowski, and Przemyslaw Kazienko. 2021. https://api.semanticscholar.org/CorpusID:246288645 Learning personal human biases and representations for subjective tasks in natural language processing . 2021 IEEE International Conference on Data Mining (ICDM), pages 1168--1173

    work page 2021

  33. [33]

    Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 30(3):411--433

    work page 2004

  34. [34]

    Yuanyuan Lei, Md Messal Monem Miah, Ayesha Qamar, Sai Ramana Reddy, Jonathan Tong, Haotian Xu, and Ruihong Huang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.293 EMONA : Event-level moral opinions in news articles . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te...

    work page doi:10.18653/v1/2024.naacl-long.293 2024

  35. [35]

    Dondera, Andrei Gead a u, Catholijn M

    Enrico Liscio, Alin E. Dondera, Andrei Gead a u, Catholijn M. Jonker, and Pradeep K. Murukannaiah. 2022. https://aclanthology.org/2022.findings-naacl.209/ Cross-domain classification of moral values . In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2727--2745, Seattle, United States. Association for Computational Linguistics

    work page 2022

  36. [36]

    Enrico Liscio, Luciano C Siebert, Catholijn M Jonker, and Pradeep K Murukannaiah. 2025. https://www.jair.org/index.php/jair/article/view/14958 Value preferences estimation and disambiguation in hybrid participatory systems . Journal of Artificial Intelligence Research, 82:819--850

    work page 2025

  37. [37]

    Siebert, Catholijn M

    Enrico Liscio, Michiel van der Meer, Luciano C. Siebert, Catholijn M. Jonker, Niek Mouter, and Pradeep K. Murukannaiah. 2021. https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p799.pdf Axies: Identifying and Evaluating Context-Specific Values . In Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '21, pag...

    work page 2021

  38. [38]

    Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. 2016. https://api.semanticscholar.org/CorpusID:1829423 Large-margin softmax loss for convolutional neural networks . In International Conference on Machine Learning

    work page 2016

  39. [39]

    Charles G Lord, Lee Ross, and Mark R Lepper. 1979. Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of personality and social psychology, 37(11):2098

    work page 1979

  40. [40]

    John L Mackie. 1988. The subjectivity of values. Essays on moral realism, pages 95--118

    work page 1988

  41. [41]

    Nailia Mirzakhmedova, Johannes Kiesel, Milad Alshomary, Maximilian Heinrich, Nicolas Handke, Xiaoni Cai, Barriere Valentin, Doratossadat Dastgheib, Omid Ghahroodi, Mohammad Ali Sadraei, Ehsaneddin Asgari, Lea Kawaletz, Henning Wachsmuth, and Benno Stein. 2023. https://api.semanticscholar.org/CorpusID:256416203 The touch \'e 23-valueeval dataset for identi...

    work page 2023

  42. [42]

    Aida Mostafazadeh Davani, Mark Diaz, Dylan K Baker, and Vinodkumar Prabhakaran. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1029 D 3 CODE : Disentangling disagreements in data across cultures on offensiveness detection and evaluation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18511--18526, Miam...

    doi:10.18653/v1/2024.emnlp-main.1029 2024

  43. [43]

    Aida Mostafazadeh Davani, Mark D \' az, and Vinodkumar Prabhakaran. 2022. https://doi.org/10.1162/tacl_a_00449 Dealing with disagreements: Looking beyond the majority vote in subjective annotations . Transactions of the Association for Computational Linguistics, 10:92--110

    doi:10.1162/tacl_a_00449 2022

  44. [44]

    Matthias Orlikowski, Paul R \"o ttger, Philipp Cimiano, and Dirk Hovy. 2023. https://doi.org/10.18653/v1/2023.acl-short.88 The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1017--1029...

    doi:10.18653/v1/2023.acl-short.88 2023

  45. [45]

    Jeongwoo Park, Enrico Liscio, and Pradeep Murukannaiah. 2024. https://aclanthology.org/2024.findings-eacl.45 Morality is non-binary: Building a pluralist moral sentence embedding space using contrastive learning . In Findings of the Association for Computational Linguistics: EACL 2024, pages 654--673, St. Julian ' s, Malta. Association for Computational L...

    work page 2024

  46. [46]

    Frey, Michael B

    Desmond Upton Patton, Philipp Blandfort, William R. Frey, Michael B. Gaskell, and Svebor Karaman. 2019. https://api.semanticscholar.org/CorpusID:102352155 Annotating social media data from vulnerable populations: Evaluating disagreement between domain experts and graduate student annotators . In Hawaii International Conference on System Sciences

    work page 2019

  47. [47]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The `` problem '' of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    doi:10.18653/v1/2022.emnlp-main.731 2022

  48. [48]

    John Rawls. 2002. John Rawls: Political liberalism and the law of peoples, volume 2. Taylor & Francis

    work page 2002

  49. [49]

    Lee Ross and Andrew Ward. 2013. Naive realism in everyday life: Implications for social conflict and misunderstanding. In Values and knowledge, pages 103--135. Psychology Press

    work page 2013

  50. [50]

    Pierrehumbert

    Paul R \"o ttger, Bertie Vidgen, Dirk Hovy, and Janet B. Pierrehumbert. 2021. https://api.semanticscholar.org/CorpusID:245130931 Two contrasting data annotation paradigms for subjective nlp tasks . In North American Chapter of the Association for Computational Linguistics

    work page 2021

  51. [51]

    Mike Schaekermann, Joslin Goh, Kate Larson, and Edith Law. 2018. https://doi.org/10.1145/3274423 Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work . Proc. ACM Hum.-Comput. Interact., 2(CSCW)

    doi:10.1145/3274423 2018

  52. [52]

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. https://doi.org/10.1109/cvpr.2015.7298682 Facenet: A unified embedding for face recognition and clustering . In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 815–823. IEEE

    arXiv 2015

  53. [53]

    Schwartz

    Shalom H. Schwartz. 1994. https://doi.org/10.1111/j.1540-4560.1994.tb01196.x Are there universal aspects in the structure and contents of human values? Journal of Social Issues, 50(4):19--45

    arXiv 1994

  54. [54]

    Shalom H. Schwartz, Jan Cieciuch, Michele Vecchione, Eldad Davidov, Ronald Fischer, Constanze Beierlein, Alice Ramos, Markku Verkasalo, Jan-Erik L \"o nnqvist, Kursad Demirutku, Ozlem Dirilen-Gumus, and Mark Konty. 2012. https://api.semanticscholar.org/CorpusID:20456630 Refining the theory of basic individual values. Journal of personality and social psyc...

    work page 2012

  55. [55]

    Rithik Appachi Senthilkumar, Amir Homayounirad, and Luciano Cavalcante Siebert. 2024. Leveraging large language models to identify the values behind arguments. In International Workshop on Value Engineering in AI, pages 87--103. Springer

    work page 2024

  56. [56]

    Barry Stroud. 1988. The study of human nature and the subjectivity of value. The Tanner Lectures on Human Value

    work page 1988

  57. [57]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama: Open and efficient foundation language models . Preprint, arXiv:2302.13971

    Pith/arXiv arXiv 2023

  58. [58]

    T5 paraphrase paws

    Vamsi. T5 paraphrase paws. https://huggingface.co/Vamsi/T5_Paraphrase_Paws. Accessed: 2024-12-06

    work page 2024

  59. [59]

    Murukannaiah, and Enrico Liscio

    Michiel van der Meer, Neele Falk, Pradeep K. Murukannaiah, and Enrico Liscio. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1031 Annotator-centric active learning for subjective NLP tasks . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18537--18555, Miami, Florida, USA. Association for Computational L...

    doi:10.18653/v1/2024.emnlp-main.1031 2024

  60. [60]

    S. L. Vargo and R. F. Lusch. 2015. https://doi.org/10.1007/s11747-015-0456-3 Institutions and axioms: an extension and update of service-dominant logic . Journal of the Academy of Marketing Science, 44:5--23

    work page doi:10.1007/s11747-015-0456-3 2015

  61. [61]

    Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. 2023. https://doi.org/10.1609/aaai.v37i12.26698 Everyone's voice matters: quantifying annotation disagreement using demographic information . In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thir...

    doi:10.1609/aaai.v37i12.26698 2023

  62. [62]

    Xinpeng Wang and Barbara Plank. 2023. https://arxiv.org/abs/2310.14979 Actor: Active learning with annotator-specific classification heads to embrace human label variation . Preprint, arXiv:2310.14979

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  63. [63]

    Zeerak Waseem. 2016. https://doi.org/10.18653/v1/W16-5618 Are you a racist or am I seeing things? annotator influence on hate speech detection on T witter . In Proceedings of the First Workshop on NLP and Computational Social Science , pages 138--142, Austin, Texas. Association for Computational Linguistics

    doi:10.18653/v1/w16-5618 2016

  64. [64]

    Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.acl-long.123 V ari E rr NLI : Separating annotation error from human label variation . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256--2269, Bangkok, Thailand....

    doi:10.18653/v1/2024.acl-long.123 2024

  65. [65]

    Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christopher Homan. 2023. https://doi.org/10.18653/v1/2023.findings-acl.287 Disagreement matters: Preserving label diversity by jointly modeling item and annotator label distributions with D is C o . In Findings of the Association for Computational Linguistics: ACL 20...

    doi:10.18653/v1/2023.findings-acl.287 2023