Pith. sign in

REVIEW 5 major objections 6 minor 70 references

Bridging the Gap: In-Context Learning for Modeling Human Disagreement

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs collapse human disagreement into near-monolithic labels, even when prompted to consider multiple perspectives.

desk verdict Useful negative result on disaggregated labels; the advertised positive zero-shot result is selective reporting and should be reframed. read the letter →

arxiv 2506.06113 v1 pith:K3BWZCM4 submitted 2025-06-06 cs.CL

Benedetta Muscato , Yue Li , Gizem Gezici , Zhixue Zhao , Fosca Giannotti This is my paper
classification cs.CL
keywords in-contextlearninghumandisagreementsubjectiveannotationperspectivismhatespeechdetectionsoftlabelsdemonstrationselectionJensen-Shannondivergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can do more than reproduce majority-vote labels: can they predict the spread of human judgments on subjective tasks like hate speech and offensive language detection? Across four open-source LLMs and three LeWiDi datasets, the authors find that explicitly instructing a model to consider multiple perspectives improves its aggregated hard-label predictions in zero-shot settings, bringing open models close to fine-tuned systems on soft evaluation metrics. The same prompting fails for disaggregated labels: when asked to output individual annotator votes or probability distributions, models settle into near-monolithic outputs such as [1,0,0,0] or [0.9,0.1], missing the genuine disagreement in human annotations. Demonstration selection affects results, while example ordering has little effect. The paper matters because it tests, without fine-tuning, whether cheap prompting can make LLMs perspective-aware rather than majority-vote machines.

What carries the argument

The carrying mechanism is the multi-perspective prompt plus the evaluation geometry. A prompt has three controllable parts: a task definition that tells the model the task is subjective and names the perspectives to consider, a label space (aggregated hard, disaggregated hard, disaggregated soft), and, in few-shot settings, demonstrations selected by BM25 or PLM similarity, by annotator-disagreement entropy, by a two-stage ranking, and ordered randomly or by curriculum. The metric that exposes the failure mode is Jensen–Shannon divergence (JSD), the symmetric distance between the model's output distribution and the human annotation distribution, with cross-entropy reporting how overconfident the top prediction is. Comparing the three label spaces is what separates 'good at consensus' from 'good at disagreement.'

What would settle it

For test items where the human annotation list has high entropy, measure the Shannon entropy of the model's predicted disaggregated hard list or soft distribution. The central claim predicts near-zero entropy in those cases; if model entropy instead tracks the human spread across many items, the collapse result would be overturned.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is asymmetric: instructing an LLM to consider diverse perspectives—often with role-playing—improves its ability to predict the aggregated majority label of subjective texts, bringing zero-shot open models close to or better than fine-tuned baselines on Jensen–Shannon divergence (JSD) and cross-entropy. On MD-Agreement, the best multi-perspective zero-shot model reaches a JSD of 0.14, matching the best baseline on ConvAbuse. But when the same models are asked to reproduce the individual annotator labels (hard disaggregated) or the probability spread over classes (soft disaggregated), the outputs converge to near-point-mass distributions like [1,0,0,0] or [0.9,0.1], and JSD relative to human distributions is much worse. The paper thus establishes a boundary: current open-source LLMs can approximate consensus but not the full spectrum of human disagreement through in-context learning alone.

Load-bearing premise

The load-bearing assumption is that the human label distributions from the small LeWiDi annotator pools (3–8 annotators per item) are a valid gold standard for 'multiple perspectives', and that JSD/cross-entropy against them measures perspective capture.

Editorial extensions

If this is right

  • Zero-shot multi-perspective prompting with role-playing can serve as a fine-tuning-free route to better aggregated predictions on subjective classification.
  • LLM outputs that match individual human judgments should not be expected from simple disaggregated hard or soft prompting; the outputs collapse to point masses.
  • Demonstration selection matters: BM25 similarity and two-stage similarity-plus-disagreement ranking give measurable gains, while curriculum ordering has limited, inconsistent impact.
  • The best model differs across datasets, so prompt-strategy conclusions need to be checked per model and per task rather than assumed to transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reference distributions come from only 3–8 annotators per item, an untested alternative reading is that near-monolithic model outputs are being compared against noisy estimates of disagreement; larger annotation pools would sharpen or weaken the failure claim.
  • The same prompt matrix could be run on multi-class and multilingual subjective tasks, which the paper identifies as future work; the current conclusion is scoped to binary English tasks.
  • Entropy-only demonstration selection may work better when anchored by similarity; a systematic sweep of the similarity–disagreement trade-off could turn the paper's 'inconsistent' finding into a precise recipe.
  • Conditioning disaggregated generation on annotator identity or demographic prompts, rather than one generic multi-perspective instruction, would test whether the collapse is a limitation of the instruction format or of the models' latent representation of disagreement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper investigates whether in-context learning (ICL) can make large language models capture human disagreement in subjective NLP tasks. Using three LeWiDi benchmark datasets (HS-Brexit, MD-Agreement, ConvAbuse), four open-source instruction-tuned LLMs, and three label spaces (aggregated hard, disaggregated hard, disaggregated soft), the authors compare a standard baseline prompt with a multi-perspective (MP) prompt that explicitly asks the model to consider diverse viewpoints. In few-shot settings they also vary demonstration selection (BM25, PLM similarity, annotator-disagreement entropy, two-stage ranking) and ordering (random vs. curriculum learning). The paper claims that MP prompting improves aggregated hard-label prediction relative to baseline in zero-shot settings, while disaggregated hard and soft label predictions collapse to near-monolithic distributions, indicating that LLMs fail to capture the full spectrum of human judgments. The main text reports only the best-performing models per dataset, with full results deferred to appendices.

Significance. If the central positive result is robust, the paper provides a practically useful comparison of ICL strategies for subjective tasks and a valuable negative result about the difficulty of eliciting distributional judgments from open-source LLMs. Its strengths include the use of four open-source models, three datasets, soft evaluation metrics (JSD and CE), and the public release of prompts and detailed appendix tables, which enables independent verification. The negative finding on disaggregated labels aligns with related work (e.g., Pavlovic and Poesio 2024) and is a useful caution for perspectivist NLP. However, the main text's selective reporting and the absence of any significance testing make the headline claim about multi-perspective prompting's superiority unsupported as stated; the paper's own full tables contain several counterexamples. The study is therefore a useful empirical contribution that requires substantial revision to align claims with evidence.

major comments (5)
  1. [§5.1, Table 2 vs. Appendix B, Table 6] The claim that the multi-perspective approach 'outperforms the baseline in most zero-shot scenarios' for aggregated labels is not supported by the full 12-cell comparison across the four models and three datasets. For example, HS-Brexit/Gemma has Baseline_aggr_0S JSD 0.26 vs. MultiP_aggr_hard_0S 0.30; MD-Agreement/Llama3 has baseline 0.16 vs. MP 0.45; ConvAbuse/Olmo has baseline 0.21 vs. MP 0.42. Only on HS-Brexit does the best MP model beat the best baseline, and on MD-Agreement the best baseline (Llama3, JSD 0.16) is substantially better than the best MP model (Gemma, JSD 0.25). Please report the full aggregated-label zero-shot comparison, provide a paired summary (e.g., a sign test or effect size across the 12 model-dataset cells), and revise the abstract and Section 1 claims accordingly.
  2. [Table 2, ConvAbuse rows] The ConvAbuse rows for Olmo-7b-Instruct in the few-shot portion of Table 2 are nearly identical to the DeepSeek-7b-chat zero-shot rows (e.g., MultiP_aggr_FS: 46.83/45.83/0.24/0.38 vs. MultiP_aggr_0S: 46.83/45.83/0.24/0.38; Baseline_aggr_FS_RL: 50.73/44.95/0.14/0.29 vs. Baseline_aggr_0S_RL: 52.71/51.95/0.14/0.29). This suggests a copy/paste or labeling error that makes the headline few-shot results for ConvAbuse unreliable. Please verify the table entries and correct the values or the row labels.
  3. [§3.3.1, threshold selection] Section 3.3.1 states that the similarity threshold of 0.7 was selected after 'empirically testing various thresholds ranging from 0.5 to 0.8,' with no indication of whether this tuning was performed on the dev set or on the test set. Since the threshold is used to produce the main few-shot demonstration-selection results, tuning it on the evaluation data may inflate reported performance. Please clarify whether the threshold was chosen using a held-out split; if not, provide a sensitivity analysis varying the threshold on the dev set.
  4. [§5.2, Tables 7–9] The statement in Section 5.2 that the multi-perspective approach 'slightly outperforms the baseline in JSD and CE scores across all datasets' for aggregated labels in few-shot settings is too strong. The full tables show ties and counterexamples, e.g., HS-Brexit/Olmo Baseline_aggr_FS JSD 0.34 vs. MultiP_aggr_hard_FS 0.34; ConvAbuse/Gemma Baseline_aggr_FS 0.15 vs. MultiP_aggr_hard_FS 0.15; MD-Agreement/Llama3 Baseline_aggr_FS 0.28 vs. MultiP_aggr_hard_FS 0.28. Please report the complete comparison and replace 'across all datasets' with a quantitative summary that accounts for ties and direction.
  5. [§4.3, §5.1, Limitations] The evaluation of disaggregated labels compares model outputs to human distributions estimated from very small annotator pools (3–8 annotators per item, Table 1). The strong negative result — that LLMs collapse to monolithic distributions such as [1,0,0,0] — may partly reflect mismatch to these specific, noisy annotation targets rather than a general inability to model disagreement. The paper should either add an analysis of how sensitive the JSD/CE conclusions are to the small annotator-pool sizes (e.g., by bootstrapping or by focusing on items with more annotators) or explicitly reframe the claim in the abstract and Section 5.1 as applying to these LeWiDi datasets and their small annotation pools.
minor comments (6)
  1. [Table 1] The dataset statistics in Table 1 are not formatted as separate columns; for example, 'HS-Brexit784 168 168 2 6 69% Hate speech' runs together. Please fix the table layout.
  2. [§3.1] The references to 'Box 3.1 ↑' and 'Box 3.1 ↓' are confusing because the boxes are not numbered in the displayed text. Please label the prompt boxes explicitly.
  3. [§3.2.1] The notation 'A= a1, ..., an' lacks the set-braces or vector formatting used elsewhere; this makes the formal definition harder to read.
  4. [Appendix B, Table 6 caption] The caption explains that Llama3-8B-Instruct refuses to predict soft labels with RL on MD-Agreement, but the table also contains a fully empty row for that condition; please state whether the dashes in that row represent refusal or missing data.
  5. [§5.2, footnote 13] The sentence 'Detailed results will be shared upon acceptance due to space constraints' is not appropriate for a journal submission; the full results should be included in the appendix or a supplementary file.
  6. [References] Some references are incomplete or contain placeholder author lists (e.g., 'and 1 others'), which should be expanded with the full author lists or formatted according to the journal style.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims rest on external human label distributions and measured LLM outputs; the self-citation is background, and the threshold tuning is hyperparameter selection, not a fitted input renamed as a prediction.

full rationale

The paper's derivation chain is an empirical evaluation: LLM outputs are elicited under baseline and multi-perspective prompts and compared against external human disagreement distributions from the LeWiDi datasets using JSD, CE, and F1. These metrics are defined from the human annotations, not from the model's own outputs, so the main claims ('multi-perspective generation is viable in zero-shot settings, while few-shot setups often fail to capture the full spectrum of human judgments') are measured against outside data rather than being equivalent to the inputs by construction. The only self-citation of Muscato et al. (2024) appears in a background sentence listing prior work on learning from disagreement ('Casola et al., 2023; Davani et al., 2022; Sandri et al., 2023; Muscato et al., 2024') and is not load-bearing for any specific result. The retrieval threshold selection ('We empirically tested various thresholds ranging from 0.5 to 0.8 and found that a threshold of 0.7 produced the best performance in terms of cosine similarity') is a hyperparameter choice, potentially a soundness or leakage concern if tuned on evaluation data, but it does not make any reported JSD/CE value a fitted parameter renamed as a prediction. No equation in the paper defines one claimed quantity in terms of another claimed quantity, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. The skeptics' point about selective reporting in Section 5.1 versus the full tables is an internal-consistency and presentation issue, not a circularity issue. Therefore the paper is not circular in the sense defined here.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical evaluation and introduces no new free parameters other than the demonstration-retrieval threshold, which is tuned on the evaluation data. The key hidden premises are domain assumptions about the validity of entropy as a difficulty signal and the reliability of small-pool human distributions as ground truth.

free parameters (1)
  • BM25/PLM similarity threshold = 0.7
    Threshold for retrieving top-k similar demonstrations; authors state they tested 0.5-0.8 and chose 0.7 based on best performance (Section 3.3.1).
assumptions (3)
  • domain assumption Annotator disagreement entropy is a valid proxy for example difficulty and perspectivist informativeness.
    Used to select and order demonstration examples (Sections 3.3.1 and 3.3.2); if high-entropy examples are not actually informative, the selection and curriculum ordering strategies lose their motivation.
  • domain assumption The human label distribution in each dataset is the correct target distribution for evaluating multi-perspective models.
    All JSD/CE scores are computed against these distributions (Section 4.3), and conclusions about 'capturing disagreement' depend on this.
  • domain assumption JSD and CE are appropriate soft evaluation metrics for disagreement modeling.
    Stated in Section 4.3; if the metric favors certain output shapes (e.g., low variance distributions), the ranking of methods could change.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging the Gap: In-Context Learning for Modeling Human Disagreement." pith.science (2026). https://pith.science/paper/K3BWZCM4

@misc{pith2026250606113,
  author       = {Pith},
  title        = {Pith review of: Bridging the Gap: In-Context Learning for Modeling Human Disagreement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3BWZCM4}},
  note         = {Machine review of arXiv:2506.06113}
}
read the original abstract

Large Language Models (LLMs) have shown strong performance on NLP classification tasks. However, they typically rely on aggregated labels-often via majority voting-which can obscure the human disagreement inherent in subjective annotations. This study examines whether LLMs can capture multiple perspectives and reflect annotator disagreement in subjective tasks such as hate speech and offensive language detection. We use in-context learning (ICL) in zero-shot and few-shot settings, evaluating four open-source LLMs across three label modeling strategies: aggregated hard labels, and disaggregated hard and soft labels. In few-shot prompting, we assess demonstration selection methods based on textual similarity (BM25, PLM-based), annotation disagreement (entropy), a combined ranking, and example ordering strategies (random vs. curriculum-based). Results show that multi-perspective generation is viable in zero-shot settings, while few-shot setups often fail to capture the full spectrum of human judgments. Prompt design and demonstration selection notably affect performance, though example ordering has limited impact. These findings highlight the challenges of modeling subjectivity with LLMs and the importance of building more perspective-aware, socially intelligent models.

Figures

Figures reproduced from arXiv: 2506.06113 by the authors.

Figure 1
Figure 1. Overview of our findings. Our analysis shows [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. We evaluate multi-perspective ICL by selecting demonstration examples through three strategies: (1) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 35 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.15896

    arXiv 2021

  4. [4]

    Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Leonardo Neves, Kiamehr Rezaee, Luis Espinosa-Anke, Jiaxin Pei, and Jose Camacho-Collados. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.838 S uper T weet E val: A challenging, unified and heterogeneous benchmark for social media NLP research . In Findings of the Association for Computational L...

    work page doi:10.18653/v1/2023.findings-emnlp.838 2023

  5. [5]

    Ron Artstein. 2017. Inter-annotator agreement. Handbook of linguistic annotation, pages 297--313

    work page 2017

  6. [6]

    Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. https://doi.org/10.18653/v1/2021.bppf-1.3 We need to consider disagreement in evaluation . In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15--21, Online. Association for Computational Linguistics

    doi:10.18653/v1/2021.bppf-1.3 2021

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  8. [8]

    Silvia Casola, Soda Lo, Valerio Basile, Simona Frenda, Alessandra Cignarella, Viviana Patti, and Cristina Bosco. 2023. Confidence-based ensembling of perspective-aware models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3496--3507

    work page 2023

Show all 70 references
  1. [9]

    Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.587 C onv A buse: Data, analysis, and benchmarks for nuanced abuse detection in conversational AI . In Proceedings of the 2021 Conference on Empirical Methods in Natura...

    2021 doi

  2. [10]

    Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.842 seeing the big through the small : Can LLM s approximate human judgment distributions on NLI from a few explanations? In Findings...

    2024 doi

  3. [11]

    Van Dang, Michael Bendersky, and W Bruce Croft. 2013. Two-stage learning to rank for information retrieval. In Advances in Information Retrieval: 35th European Conference on IR Research, ECIR 2013, Moscow, Russia, March 24-27, 2013. Proceedings 35, pages 423--434. Springer

    2013

  4. [12]

    Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2021. https://arxiv.org/abs/2110.14839 Hate speech classifiers learn human-like social stereotypes . Preprint, arXiv:2110.14839

    2021 arXiv

  5. [13]

    Aida Mostafazadeh Davani, Mark D \' az, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92--110

    2022

  6. [14]

    Flor Miriam Plaza Del Arco, Arturo Montejo-R \'a ez, L Alfonso Urena Lopez, and Mar \' a-Teresa Mart \' n-Valdivia. 2021. Offendes: A new corpus in spanish for offensive language research. In Proceedings of the international conference on recent advances in natural language pr...

    2021

  7. [15]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.64 A survey on in-context learning . In Proceedings of the 2024 Conference o...

    2024 doi

  8. [16]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, and 1 others. 2024 b . A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107--1128

    2024

  9. [17]

    Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. Modular pluralism: Pluralistic alignment via multi-llm collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pa...

    2024

  10. [18]

    Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6715--6726

    2023

  11. [19]

    Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. 2024. The perspectivist paradigm shift: Assumptions and challenges of capturing human labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

    2024

  12. [20]

    Simona Frenda, Gavin Abercrombie, Valerio Basile, Alessandro Pedrani, Raffaella Panizzon, Alessandra Teresa Cignarella, Cristina Marco, and Davide Bernardi. 2024. Perspectivist approaches to natural language processing: a survey. Language Resources and Evaluation, pages 1--28

    2024

  13. [21]

    Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol...

    2021

  14. [22]

    Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.679 Demystifying prompts in language models via perplexity estimation . In Findings of the Association for Computational Linguistics: EMNLP 2023, page...

    2023 doi

  15. [23]

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, and 1 others. 2024. Olmo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association f...

    2024

  16. [24]

    Fatemah Husain and Ozlem Uzuner. 2021. A survey of offensive language detection for the arabic language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 20(1):1--44

    2021

  17. [25]

    Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. 2023. Lambada: Backward chaining for automated reasoning in natural language. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6...

    2023

  18. [26]

    Hannah Rose Kirk, Iason Gabriel, Chris Summerfield, Bertie Vidgen, and Scott A Hale. 2025. Why human-ai relationships need socioaffective alignment. arXiv preprint arXiv:2502.02528

    2025 arXiv

  19. [27]

    Hannah Rose Kirk, Bertie Vidgen, Paul R \"o ttger, and Scott A Hale. 2024. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pages 1--10

    2024

  20. [28]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Jiaming Zhou, and Haoqin Sun. 2024. Self-prompt tuning: Enable autonomous role-playing in llms. arXiv preprint arXiv:2407.08995

    2024 arXiv

  21. [29]

    Grgur Kova c , Masataka Sawayama, R \'e my Portelas, C \'e dric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870

    2023 arXiv

  22. [30]

    Xiaochong Lan, Chen Gao, Depeng Jin, and Yong Li. 2024. Stance detection with collaborative role-infused llm-based agents. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages 891--903

    2024

  23. [31]

    Noah Lee, Na Min An, and James Thorne. 2023. Can large language models capture dissenting human voices? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4569--4585

    2023

  24. [32]

    Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma, and Massimo Poesio. 2023. Semeval-2023 task 11: Learning with disagreements (lewidi). In Proceedings of the 17th International Workshop on Seman...

    2023

  25. [33]

    Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.822 Agreeing to disagree: Annotating offensive language datasets with annotators' disagreement . In Proceedings of the 2021 Conference...

    2021 doi

  26. [34]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022 a . Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950--1965

    2022

  27. [35]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022 b . What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learni...

    2022

  28. [36]

    Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. 2023. Training socially aligned language models on simulated social interactions. arXiv preprint arXiv:2305.16960

    2023 arXiv

  29. [37]

    Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, Yong Huang, and Wei Lu. 2024. Let's learn step by step: Enhancing in-context learning ability with curriculum learning. arXiv preprint arXiv:2402.10738

    2024 arXiv

  30. [38]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of th...

    2022 doi

  31. [39]

    Leena Mathur, Paul Pu Liang, and Louis-Philippe Morency. 2024. Advancing social intelligence in ai agents: Technical challenges and open questions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20541--20560

    2024

  32. [40]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing...

    2022

  33. [41]

    Justin M Mittelst \"a dt, Julia Maier, Panja Goerke, Frank Zinn, and Michael Hermes. 2024. Large language models can outperform humans in social situational judgments. Scientific Reports, 14(1):27449

    2024

  34. [42]

    Benedetta Muscato, Praveen Bushipaka, Gizem Gezici, Lucia Passaro, Fosca Giannotti, and 1 others. 2024. Multi-perspective stance detection. In CEUR WORKSHOP PROCEEDINGS, volume 3825, pages 208--214. CEUR-WS

    2024

  35. [43]

    Maja Pavlovic and Massimo Poesio. 2024. https://aclanthology.org/2024.nlperspectives-1.11/ The effectiveness of LLM s as annotators: A comparative overview and empirical analysis of direct representation . In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (...

    2024

  36. [44]

    Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. https://doi.org/10.18653/v1/2024.acl-long.492 Revisiting demonstration selection strategies in in-context learning . In Proceedings of the 62nd Annual Meeting of the Association...

    2024 doi

  37. [45]

    Gon c alo Pereira, Rui Prada, and Pedro A Santos. 2016. Integrating social power into the decision-making of cognitive agents. Artificial Intelligence, 241:1--44

    2016

  38. [46]

    Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. 2019. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9617--9626

    2019

  39. [47]

    Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682

    2022

  40. [48]

    Barbara Plank, Dirk Hovy, and Anders S gaard. 2014. https://doi.org/10.3115/v1/P14-2083 Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507--511, Baltimore,...

    2014 doi

  41. [49]

    Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. 2024. In-context learning with iterative demonstration selection. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7441--7455

    2024

  42. [50]

    Liang Qiu, Yizhou Zhao, Yuan Liang, Pan Lu, Weiyan Shi, Zhou Yu, and Song-Chun Zhu. 2022. https://doi.org/10.18653/v1/2022.sigdial-1.16 Towards socially intelligent agents with mental state transition and human value . In Proceedings of the 23rd Annual Meeting of the Special I...

    2022 doi

  43. [51]

    Giulia Rizzi, Elisa Leonardelli, Massimo Poesio, Alexandra Uma, Maja Pavlovic, Silviu Paun, Paolo Rosso, and Elisabetta Fersini. 2024. https://aclanthology.org/2024.nlperspectives-1.9/ Soft metrics for evaluation with disagreements: an assessment . In Proceedings of the 3rd Wo...

    2024

  44. [52]

    Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  45. [53]

    Paul R \"o ttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective nlp tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T...

    2022

  46. [54]

    Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elisabetta Je z ek. 2023. Why don’t you do it right? analysing annotators’ disagreement in subjective tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages...

    2023

  47. [55]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971--30004. PMLR

    2023

  48. [56]

    Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large lms. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762--3780

    2022

  49. [57]

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, and 1 others. 2024. Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Confere...

    2024

  50. [58]

    Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An information-theoretic approach to prompt engineering without ground truth labels. In Proceedings of the 60th Annual Meet...

    2022

  51. [59]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, and 1 others. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ...

    2022 arXiv

  52. [60]

    Mohammad Tahaei, Marios Constantinides, Daniele Quercia, Sean Kennedy, Michael Muller, Simone Stumpf, Q Vera Liao, Ricardo Baeza-Yates, Lora Aroyo, Jess Holbrook, and 1 others. 2023. Human-centered responsible artificial intelligence: Current & future trends. In Extended Abstr...

    2023

  53. [61]

    Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385--1470

    2021

  54. [62]

    Michiel Van Der Meer, Neele Falk, Pradeep Murukannaiah, and Enrico Liscio. 2024. Annotator-centric active learning for subjective nlp tasks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18537--18555

    2024

  55. [63]

    Nikolas Vitsakis, Amit Parekh, Tanvi Dinkar, Gavin Abercrombie, Ioannis Konstas, and Verena Rieser. 2023. ilab at semeval-2023 task 11 le-wi-di: Modelling disagreement or modelling perspectives? In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-...

    2023

  56. [64]

    Angelina Wang, Jamie Morgenstern, and John P Dickerson. 2025. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, pages 1--12

    2025

  57. [65]

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36

    2024

  58. [66]

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzni...

    2022 doi

  59. [67]

    Yuxia Wang, Minghan Wang, Yimeng Chen, Shimin Tao, Jiaxin Guo, Chang Su, Min Zhang, and Hao Yang. 2022 b . Capture human disagreement distributions by calibrated networks for natural language inference. In Findings of the Association for Computational Linguistics: ACL 2022, pa...

    2022

  60. [68]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  61. [69]

    Patrick H Winston. 1980. Learning and reasoning by analogy. Communications of the ACM, 23(12):689--703

    1980

  62. [70]

    Chen Zhang, Xiao Liu, Jiuheng Lin, and Yansong Feng. 2024. https://doi.org/10.18653/v1/2024.findings-acl.519 Teaching large language models an unseen language on the fly . In Findings of the Association for Computational Linguistics: ACL 2024, pages 8783--8800, Bangkok, Thaila...

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.