Understanding The Effect Of Temperature On Alignment With Human Opinions

Maja Pavlovic; Massimo Poesio

REVIEW 3 major objections 5 minor 1 cited by

Understanding The Effect Of Temperature On Alignment With Human Opinions

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sampling at higher temperature aligns LLMs with human opinions

desk verdict A useful method comparison with an overstated temperature story: sampling/log-prob beat direct prompting, but the paper's own table does not support higher temperature improving alignment. read the letter →

arxiv 2411.10080 v1 pith:ED5ESXTJ submitted 2024-11-15 cs.CL cs.CY

Maja Pavlovic , Massimo Poesio This is my paper · ORCID

classification cs.CLcs.CY

keywords opinionalignmenttemperaturesamplingMonteCarloestimationlog-probabilityhumandisagreementsubjectiveclassificationLLMuncertaintyoverconfidence

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates whether simple changes to how an LLM is queried can make its outputs better reflect the distribution of human opinions in subjective classification. On three binary tasks from the SemEval 2023 'Learning with Disagreements' datasets, it compares directly prompting gpt-3.5-turbo for an opinion distribution with two alternatives: Monte Carlo sampling of repeated responses and repeated log-probability extraction. The central finding is that both sampling-based methods, particularly with higher temperature settings, produce distributions closer to aggregated human annotator votes than direct prompting does, mainly because higher temperatures reduce the model's overconfident zero-entropy predictions on items where humans mostly agree. The paper argues this matters because LLMs are increasingly used where people legitimately disagree, and temperature is a parameter that is often unreported yet strongly affects how pluralistic model outputs appear.

What carries the argument

The load-bearing mechanism is the temperature parameter of the LLM's sampling distribution, which controls how much the model deviates from always choosing the maximum-likelihood token. At temperatures near zero the model becomes deterministic and overconfident; as temperature rises, sampled responses spread out across label categories. The paper estimates opinion distributions through Monte Carlo estimation, the frequency of valid responses over ten repeated samples, and log-probability estimation, averaging normalized exponentiated log-probabilities over samples. The comparison is measured with instance-level metrics: cross-entropy, Jensen-Shannon divergence, L1 distance, DistCE, EntCE, and expected calibration error.

What would settle it

Run the same three methods on a fresh random sample of items from each SemEval-2023 dataset, with ten samples and temperatures near zero, 0.8, and 2.0; if direct prompting matches or beats Monte Carlo sampling on cross-entropy and L1 distance in a dataset, the claimed advantage is not general.

Watch

Extended reading notes

Core claim

The paper's claim is that temperature-adjusted sampling is a viable, parameter-light way to align LLM distributions with human opinions on subjective tasks. The authors implement Monte Carlo estimation, sampling ten responses and counting valid category labels, and log-probability estimation, averaging normalized exponentiated log-probabilities of the top ten tokens across repeated samples, each at temperatures 2.0, 0.8, and near zero, and compare them against direct prompting. Across the three datasets, the sampling-based methods beat direct prompting on cross-entropy, Jensen-Shannon divergence, L1 distance, and the DistCE and EntCE calibration metrics; Monte Carlo estimation had the lowest scores overall, though the gap with log-probability estimation was small. At higher temperatures the model produced fewer zero-entropy distributions, and model entropy tracked human entropy more closely, while expected calibration error results were consistent with prior work showing that temperature can improve uncertainty calibration. The paper therefore proposes that temperature should be reported and considered in opinion-alignment studies, while cautioning that the evaluation target is the aggregated annotator distribution.

Load-bearing premise

The evaluation treats the aggregated human-annotator vote distribution as the true distribution of human opinions, even though some items were annotated by only two or three people and the 101-item subset was asserted to be representative without a documented selection procedure.

Editorial extensions

If this is right

Higher-temperature Monte Carlo sampling can replace direct prompting as a simple baseline for extracting opinion distributions in subjective classification.
Temperature should be reported alongside results in any study that extracts opinion distributions from LLMs, since it materially changes apparent alignment.
Entropy-based metrics should not be used alone: the paper shows that two very different distributions can share the same entropy, so L1 or distributional distances are also needed.
The method works without extra human explanations or auxiliary models, making it cheap to apply to new subjective tasks.
The same temperature mechanism that improves alignment also improves expected calibration error, linking opinion pluralism with uncertainty calibration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If temperature is a stand-in for the spread of opinions a model can express, the optimal setting may scale with the amount of annotator disagreement in the target data; a natural extension is to tune temperature per item or per dataset against annotator entropy.
Because the evaluation target is an aggregated annotator distribution, these results speak to matching crowd averages rather than representing any particular perspective; future work could weight annotators or model subpopulations separately.
The same mechanism suggests temperature-based sampling could improve calibration in other tasks where human label variation is treated as uncertainty, such as natural language inference or safety classification.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look if you work on opinion extraction or temperature reporting. The paper does one thing well: it tests three methods (direct prompting, Monte Carlo sampling, log-probability averaging) at three temperatures on three subjective classification benchmarks, and shows that sampling and log-probability clearly beat direct prompting. That finding is consistent across all metrics and datasets, with cross-entropy dropping from roughly 4 to below 1.4. It also correctly notes that prior work rarely reports temperature, which is a real methodological gap.

Soft spots: the title and conclusion claim that higher temperature improves alignment with human opinions. Table 2 does not support that cleanly. On cross-entropy, T=2 is the worst of the three sampling temperatures on all three datasets; T=0.8 is best on ConvAbuse and MD-Agree, while T≈0 is best on HS-Brexit. Higher temperature only wins on some secondary metrics (EntCE, DistCE, L1) for MD-Agree. So the temperature narrative is an over-reading of mixed, metric-dependent results. There are no error bars or significance tests, and with n=101 per dataset, the differences could easily be noise. The subset selection is described as "carefully chosen" but not disclosed, which is a reproducibility issue. The paper acknowledges the Pann≈P assumption in the Limitations, which is good, but sparse annotators in ConvAbuse make that evaluation target shaky.

Who it is for: people doing extraction of opinion distributions from LLMs and researchers thinking about evaluation methodology. It is a workshop-grade empirical note, not a definitive study. I would send it to review rather than desk reject, because the method comparison is useful and the temperature-reporting point is worth raising, but I would ask for major revisions: report standard errors, run paired significance tests, describe subset selection, and tone down the temperature claim to what the data actually show.

Referee Report

3 major / 5 minor

Summary. The paper compares three ways of extracting opinion distributions from GPT-3.5-turbo on three binary subjective classification tasks from SemEval-2023: direct prompting for a distribution, Monte Carlo estimation by sampling repeated responses, and log-probability estimation by averaging normalized top-k token probabilities. Each method is evaluated under three temperature settings (T=2, T=0.8, T≈0) using cross-entropy, Jensen-Shannon divergence, L1 distance, distribution calibration error, entropy calibration error, and expected calibration error. The main empirical claim is that sampling and log-probability approaches with simple parameter adjustments align better with human opinion distributions than direct prompting; the discussion and conclusion additionally claim that higher temperatures reduce overconfidence and yield distributions more closely calibrated with human opinions.

Significance. The comparison of extraction methods is a useful practical contribution: the improvements over direct prompting are large and consistent across all three datasets (cross-entropy drops from roughly 3.45-4.65 to at most 1.35), so the central methodological recommendation is credible and likely to be useful to practitioners. The paper also draws attention to the importance of reporting temperature settings, which is a timely point. However, the temperature-specific narrative is not supported by the reported numbers: the key alignment metric (cross-entropy) favours lower temperatures, while the metrics that favour higher temperature differ by very small margins, and no uncertainty quantification or significance testing is provided. The paper therefore needs to be revised to either establish the temperature effect rigorously or substantially tone down that claim.

major comments (3)

[§5, Table 2] The claim that 'sampling with higher temperatures reduced such overconfident predictions, resulting in distributions that are more closely calibrated with human opinions' is contradicted by the paper's own cross-entropy results. Cross-entropy is lower-is-better, and in Table 2 MC T=2 is worse than MC T=0.8 on every dataset (HS-Brexit 0.52 vs 0.34, ConvAbuse 1.25 vs 0.68, MD-Agree 1.35 vs 1.19) and worse than MC T≈0 on HS-Brexit and MD-Agree. Only a subset of metrics (EntCE, DistCE, L1) favours T=2, and those gains are small (0.01-0.07) and not consistent across datasets. With no significance tests or error bars, the paper should not state the high-temperature conclusion as a finding; please report paired comparisons or confidence intervals and revise the conclusion accordingly.
[§3, Methodology; §7, Limitations] The selection of the 'representative subset of 101 examples' is not described, despite the paper later noting that the sample size was 'carefully chosen to represent the overall distribution'. Because all results rest on n=101 per dataset and the temperature-related differences are small, the lack of a documented sampling procedure leaves open the possibility that the temperature effects are artefacts of the particular subset. Please specify exactly how the 101 examples were selected and provide bootstrap confidence intervals or permutation tests for the reported differences.
[§5, Table 3] The ECE results are used to support the temperature narrative, but ECE measures confidence calibration, not alignment with human opinions, which is the paper's stated focus. Moreover, the ECE values in Table 3 are extremely high (0.92-0.99), indicating poor calibration in absolute terms, and no test is reported for the differences between temperature conditions. Saying that the results are 'consistent with [12]' is not sufficient; if ECE is to be part of the argument, it needs a proper analysis and an explicit caveat that it is not an alignment metric.

minor comments (5)

[§1] There is a typo: 'allowing models to chose beyond' should be 'allowing models to choose beyond'.
[Table 2] The column header 'J SD' has an unnecessary space, and the dataset name 'MD-Agree.' is inconsistently punctuated across tables and text.
[§3.2] The notation 'H(.)' is introduced in running text but not defined as a function symbol; please define it explicitly when first used.
[Figure 2] The twelve panels in Figure 2 are too small to read the histograms; please enlarge the panels or split the figure by dataset so that entropy distributions are visually comparable.
[General] The manuscript does not state whether code, the exact 101-example subset, or raw predictions will be released; for an empirical paper whose contribution is the comparison, this limits reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is self-contained against external human annotations and fixed temperature conditions.

full rationale

The paper's central claims compare LLM-derived opinion distributions against human annotator distributions from the SemEval-2023 shared task, an external benchmark. The three methods (direct prompting, Monte Carlo estimation, and log-probability estimation) are defined from model outputs alone, and the temperature values (2.0, 0.8, and approximately 0) are chosen as treatment conditions before evaluation, not fitted to the outcome data. Self-citations appear where the authors reuse their own prior direct-prompting baseline [7] and a soft-metric evaluation paper [9], but neither citation is load-bearing in a way that defines the present result: the direct baseline is merely one of three compared methods, and the L1 metric is justified by an external reference to prior evaluation practice. The stated assumption that P_ann approximates P is an acknowledged limitation about data quality, not a circular step, because the human distributions are measured independently of the model outputs. The skeptical objection that the temperature-specific conclusion is not well supported by Table 2 is a correctness or statistical-evidence concern, not a circularity concern: unsupported does not mean the derivation reduces to its own inputs. No fitted parameter is renamed as a prediction, no uniqueness theorem is invoked, and no ansatz is smuggled in through citation. The derivation chain is therefore self-contained with respect to the evaluation target, and the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper relies on external human annotations as ground truth and on several methodological choices (temperature, M, k, top-p) that are not derived from the data. It introduces no new entities or fitted parameters, but its conclusions depend on the representativeness and stability of the evaluation setup.

free parameters (4)

temperature levels = 2.0, 0.8, ~0
The three temperature settings are chosen by hand to probe the effect of temperature. They are treatment conditions, not fitted to the target data, but the paper's temperature conclusion depends on these specific levels.
sampling iterations M = 10
The Monte Carlo and log-probability estimations use M=10 repeated samples. The choice is arbitrary and not varied, yet the central claim about sampling methods depends on this implementation detail.
top-k tokens k = 10
The log-probability estimation uses the top 10 token candidates. This is a hand-chosen parameter that could affect the resulting distributions.
top-p = 1.0
Top-p is fixed at 1.0 across all runs, removing nucleus sampling truncation. This is a methodological choice that affects the sampled outputs.

assumptions (3)

domain assumption The aggregated annotator distribution P_ann approximates the true conditional distribution P.
Stated in the Limitations section. This is load-bearing because all metrics compare LLM outputs to P_ann as ground truth.
domain assumption The 101-example subset is representative of each full dataset.
Stated in Section 3 as 'carefully chosen', but no selection procedure is given. This affects the validity of all aggregate scores.
domain assumption GPT-3.5-turbo's stochastic outputs are stable enough to measure small differences between temperature settings.
The experiments use M=10 samples per instance without measuring variance or using seeds, so the reported differences between temperatures could be noise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Understanding The Effect Of Temperature On Alignment With Human Opinions." pith.science (2026). https://pith.science/paper/ED5ESXTJ

@misc{pith2026241110080,
  author       = {Pith},
  title        = {Pith review of: Understanding The Effect Of Temperature On Alignment With Human Opinions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ED5ESXTJ}},
  note         = {Machine review of arXiv:2411.10080}
}

read the original abstract

With the increasing capabilities of LLMs, recent studies focus on understanding whose opinions are represented by them and how to effectively extract aligned opinion distributions. We conducted an empirical analysis of three straightforward methods for obtaining distributions and evaluated the results across a variety of metrics. Our findings suggest that sampling and log-probability approaches with simple parameter adjustments can return better aligned outputs in subjective tasks compared to direct prompting. Yet, assuming models reflect human opinions may be limiting, highlighting the need for further research on how human subjectivity affects model uncertainty.

Figures

Figures reproduced from arXiv: 2411.10080 by the authors.

**Figure 1.** MD-Agree. T=2.0; ideally, if entropies align, a model should display a diagonal line from bottom left to top right; lower L1 is better 5 Discussion The results indicate that sampling produced more closely aligned distributions than direct extraction of opinion distributions, primarily due to improved alignment in cases with full label agreement among humans (0 entropy), as visible in [PITH_FULL_IMAGE:figures/full_f… view at source ↗

**Figure 2.** Histograms of human and GPT entropy levels for SemEval2023 datasets: directly generated GPT distributions vs. those generated with the MCE method. 6 Conclusion We assessed three techniques for deriving opinion distributions from GPT in this empirical study and find that response sampling (MC) with higher temperatures produced distributions more closely reflecting human distributions of abusiveness and offensiveness.… view at source ↗

**Figure 3.** Ideally, if entropies align, a model should display a diagonal line from bottom left to top [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Average L1-distance manhattan-distance from model distributions in relation to entropy values of human opinion distributions: The sampling approach (MCE) performs more in line with expectations across all datasets, showing that this method exhibits greater confidence o…

**Figure 5.** Figure 5: Distribution for each run (direct, MC and LP). Human Labels have more variation in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
cs.CL 2024-12 conditional novelty 6.0 of 10

LLM-generated explanations, paired with a few human labels, produce model judgment distributions as close to human judgment distributions as human explanations do on NLI.

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages · cited by 1 Pith paper

[12]

SPUQ: Perturbation-based uncertainty quantification for large language models

Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. SPUQ: Perturbation-based uncertainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (V olume 1: Long Papers), pages 2336–2346, St. Julian’s, Malta, March 2024. Association for Computational L...

work page 2024
[1]

Position: A roadmap to pluralistic alignment

Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. Position: A roadmap to pluralistic alignment. In Proceedings of the 41st Inter- national Conference on Machine Learning , volume 235 of Proceedings of Machine Lear...

work page 2024
[2]

Modular pluralism: Pluralistic alignment via multi-LLM collaboration

Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-LLM collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 4151–4171, Miami, Florida, USA, November 2024. Association for Computational Linguistics

work page 2024
[3]

seeing the big through the small

Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. "seeing the big through the small": Can llms approximate human judgment distributions on nli from a few explanations?, 2024

work page 2024
[4]

Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language mod...

2024
[5]

Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971–30004

Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971–30004. PMLR, 2023

2023
[6]

Noah Lee, Na Min An, and James Thorne. Can large language models capture dissenting human voices? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4569–4585, Singapore, December 2023. Association for Computational Linguistics

work page 2023
[7]

The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation

Maja Pavlovic and Massimo Poesio. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024 , pages 100–110, Torino, Italia, May 2024. ELRA and ICCL

work page 2024

Show all 17 references

[8]

SemEval-2023 task 11: Learning with disagreements (LeWiDi)

Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma, and Massimo Poesio. SemEval-2023 task 11: Learning with disagreements (LeWiDi). In Proceedings of the 17th International Workshop on Semantic Ev...

2023
[9]

Soft metrics for evaluation with disagreements: an assessment

Giulia Rizzi, Elisa Leonardelli, Massimo Poesio, Alexandra Uma, Maja Pavlovic, Silviu Paun, Paolo Rosso, and Elisabetta Fersini. Soft metrics for evaluation with disagreements: an assessment. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives...

2024
[10]

Stop measuring calibration when humans disagree

Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892–1915, Abu Dhabi, United Arab Emirates, December

2022
[11]

Baan, Raquel Fernandez, Barbara Plank, and Wilker Aziz. Interpreting predictive probabilities: Model confidence or human label variation? In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (V olume 2: Short Papers) , ...

2024
[13]

Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection, 2021

Sohail Akhtar, Valerio Basile, and Viviana Patti. Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection, 2021

2021
[14]

Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement

Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , ...

2021
[15]

ConvAbuse: Data, analysis, and benchmarks for nuanced abuse detection in conversational AI

Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. ConvAbuse: Data, analysis, and benchmarks for nuanced abuse detection in conversational AI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7388–7403, Online and Punta ...

2021
[16]

You are an expert annotator in

Dina Almanea and Massimo Poesio. ArMIS - the Arabic misogyny and sexism corpus with annotator subjective disagreements. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2282–2291, Marseille, France, June 2022. European Language Resources Ass...

2022
[2022]

Association for Computational Linguistics. 5

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [12]

SPUQ: Perturbation-based uncertainty quantification for large language models

Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. SPUQ: Perturbation-based uncertainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (V olume 1: Long Papers), pages 2336–2346, St. Julian’s, Malta, March 2024. Association for Computational L...

work page 2024

[2] [1]

Position: A roadmap to pluralistic alignment

Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. Position: A roadmap to pluralistic alignment. In Proceedings of the 41st Inter- national Conference on Machine Learning , volume 235 of Proceedings of Machine Lear...

work page 2024

[3] [2]

Modular pluralism: Pluralistic alignment via multi-LLM collaboration

Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-LLM collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 4151–4171, Miami, Florida, USA, November 2024. Association for Computational Linguistics

work page 2024

[4] [3]

seeing the big through the small

Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. "seeing the big through the small": Can llms approximate human judgment distributions on nli from a few explanations?, 2024

work page 2024

[5] [4]

Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language mod...

2024

[6] [5]

Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971–30004

Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971–30004. PMLR, 2023

2023

[7] [6]

Noah Lee, Na Min An, and James Thorne. Can large language models capture dissenting human voices? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4569–4585, Singapore, December 2023. Association for Computational Linguistics

work page 2023

[8] [7]

The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation

Maja Pavlovic and Massimo Poesio. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024 , pages 100–110, Torino, Italia, May 2024. ELRA and ICCL

work page 2024