Pith. sign in

REVIEW 3 major objections 3 minor 57 references

A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI

T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-generated explanations, paired with a few human labels, match human-written explanations when estimating human judgment distributions on natural language inference.

desk verdict LLM-generated explanations guided by a few human labels look like a real substitute for human explanations in HJD estimation, but the free softmax temperature needs a sensitivity check before the comparability claim is fully trusted. read the letter →

arxiv 2412.13942 v2 pith:7BHBPEPZ submitted 2024-12-18 cs.CL

Beiduo Chen , Siyao Peng , Anna Korhonen , Barbara Plank This is my paper · ORCID
classification cs.CL
keywords humanjudgmentdistributionsnaturallanguageinferencelabelvariationLLM-generatedexplanationsmodeldistributionexplanationselectionannotationcostout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human annotators often disagree on natural language inference (NLI) labels, and capturing that disagreement as a human judgment distribution is costly. This paper tries to establish that the expensive human-written explanations used to elicit such distributions can be replaced by LLM-generated explanations, provided a few human labels guide which machine explanations to select. Across three LLMs and two NLI datasets, the resulting model judgment distributions come close to those produced with human explanations and to the gold ChaosNLI human judgment distribution. If true, this would let researchers estimate label disagreement on datasets that have labels but no explanations, at much lower annotation cost.

What carries the argument

The load-bearing mechanism is the Model Judgment Distribution (MJD) Estimator: an LLM answers a multiple-choice question about whether a premise entails, contradicts, or is neutral toward a hypothesis, and its judgment distribution is read from the first-token probabilities of the answer choices, with option, sequence, and length biases removed by averaging over permutations and 'n in one' combinations. Model explanations are generated by prompting the LLM to list all possible explanations for a given label, then selecting explanations either label-free (one per label) or label-guided (matching the label counts of a few human annotations, such as three entailment, one neutral, one contradiction), using the longest or first explanations. The decisive comparison is between human explanations and model explanations under identical label guidance, which isolates whether explanation content or the human labels drive the approximation.

What would settle it

Collect the same 341 VariErr/ChaosNLI instances and compare label-guided model-explanation MJDs against MJDs obtained by sampling many complete answers at nonzero temperature, say 100 samples per instance; if the sampled label frequencies diverge systematically from the first-token-probability distributions, the estimator's central measurement is invalid and the human-versus-model comparison would need to be redone. Alternatively, an instance set where human explanations encode idiosyncratic world knowledge absent from the premise should show a clear gap whenever that knowledge drives the human label.

Watch

Extended reading notes

Core claim

The central claim is that LLM-generated explanations are good proxies for human explanations inside the MJD Estimator pipeline for NLI: when the same few human labels guide explanation selection, swapping human-written explanations for machine-written ones yields model judgment distributions with comparable KL divergence, Jensen-Shannon distance, and total variation distance to the gold human judgment distribution, and classifiers fine-tuned on those distributions perform similarly. The paper further claims this transfers from datasets with human explanations to explanation-free datasets such as MNLI, and that classifiers trained on the resulting distributions improve out-of-domain ANLI performance. The title's rose metaphor states the point: the explanation's origin matters less than the label guidance it carries.

Load-bearing premise

The whole comparison rests on treating the first-token probabilities the LLM assigns to the three answer choices, after permutation averaging and temperature scaling, as a faithful opinion distribution; if that measurement does not reflect the LLM's genuine judgment, then both the human-explanation and model-explanation estimates are unreliable even if they look similar.

Editorial extensions

If this is right

  • Estimating human judgment distributions no longer requires collecting human-written explanations, so NLI datasets with only a few labels per instance can be turned into distribution-labeled training data.
  • The label-guided strategy transfers to datasets without any explanations, such as MNLI, with results comparable to the human-explanation condition.
  • Classifiers fine-tuned on model judgment distributions trained with label-guided explanations outperform models trained without distribution information on adversarial ANLI test sets.
  • Replacing human explanations with model explanations one by one leaves performance nearly flat, while replacing them with off-topic noise explanations clearly degrades results, showing that the relevance of explanation content matters.
  • Explanation variability across generated explanations may serve as an indicator of how well a set of explanations captures human label variation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the same label-guided explanation pipeline could be tested on other disagreement-rich annotation tasks, such as toxicity or relevance judgment, to see whether the proxy result is specific to NLI or general to LLM explanations.
  • Because the MJD is read from one LLM's first-token probabilities, the approach inherits that model's priors; ensembling explanations or MJDs from multiple LLMs is a natural extension the paper leaves open and might close more of the gap to human distributions.
  • The observation that explanations a single annotator rejected sometimes outperform preferred ones suggests diversity, not human-likeness, may be the active ingredient; a controlled study that varies explanation diversity while holding label guidance fixed could separate these factors.
  • A practical caution that follows from the noise-replacement ablation: the method's success depends on generated explanations staying on-topic for the correct label, so real applications should monitor explanation relevance rather than assume any generated text helps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This paper asks whether LLM-generated explanations can substitute for human-written explanations in the MJD Estimator of Chen et al. (2024), which approximates human judgment distributions (HJDs) on NLI from a few labels and explanations. The authors prompt Llama3-70B, Mixtral-8x7B, and GPT-4o to generate multiple explanations per label, then select explanations either label-free (one per class) or label-guided (matching a few human labels). They evaluate the resulting MJDs against ChaosNLI HJDs using KL, JSD, TVD, distance correlation, and downstream fine-tuning of BERT/RoBERTa, and test generalization to MNLI (which lacks human explanations) and to the ANLI benchmark. Ablations replace human explanations with model or noise explanations, and a single-annotator study compares preferred versus unpreferred model explanations. The central claim is that label-guided model explanations yield MJDs comparable to human-explanation MJDs when the same few human labels are used.

Significance. If the claim holds, the paper offers a practical reduction in annotation cost for human label variation estimation and extends HJD approximation to datasets without human explanations. The study has several genuine strengths: multiple LLMs, a noise-replacement control showing that explanation content matters, permutation averaging over label/option/order biases, a public release of the generated explanations, and an OOD evaluation with a candid contamination discussion. The main caveat is that the headline 'comparable' rests on a small number of point estimates and on model-specific softmax temperatures that are not swept, so the central result needs additional robustness evidence before it is fully convincing.

major comments (3)
  1. [Appendix B.1; Table 1] The statement that setting softmax temperature to τ=10 for GPT-4o and τ=20 for Mixtral is done 'without loss of generality' is not supported. Temperature controls the flatness of the first-token probability distributions: as τ grows, every MJD converges toward the uniform distribution, which mechanically shrinks KL/JSD/TVD differences between any two conditions. The near-equality of GPT-4o human (KL 0.187) versus VariErr label-guided (KL 0.192) and Mixtral human (KL 0.239) versus label-guided (KL 0.238) could therefore be an artifact of the chosen temperatures rather than evidence about explanation content. Llama3's parameter-free normalization provides an important counterexample, but the paper makes the claim across models. Please add a temperature sweep for GPT-4o and Mixtral and show that the human-vs-model explanation gap remains small and stable, or qualify the claim to the specific estimator configuration used.
  2. [Table 1; §3.2] The central comparison reports only point estimates. The differences between human and label-guided model explanations are small (e.g., Llama3 KL 0.238 vs. 0.234; GPT-4o KL 0.187 vs. 0.192), but no confidence intervals, standard errors, paired significance tests, or bootstrap resampling over the 341 instances are reported. The fine-tuning comparisons also appear to be single runs without seed variation. Without uncertainty quantification, 'comparable' is not distinguishable from 'not distinguishable at this sample size.' Please report paired bootstrap confidence intervals or an equivalent paired test on the distribution metrics, and report variance over multiple fine-tuning seeds.
  3. [§5; Table 1] The noise-replacement ablation, which is the main evidence that explanation content rather than the human labels drives the result, is run only with Llama3. The analogous content control is missing for GPT-4o and Mixtral, even though the paper's headline claim is made for all three LLMs. For GPT-4o the label-free condition is fairly close to the label-guided condition on several metrics (e.g., KL 0.252 vs. 0.192), so without a noise control it is unclear how much of the label-guided improvement is due to explanation relevance. Please extend the noise replacement to at least one additional model, or explicitly restrict the content-matters conclusion to Llama3.
minor comments (3)
  1. [Appendix B.1; Appendix B.3] There are several typos in the appendix: 'promo' should be 'prompt', 'Protocals' should be 'Protocols', and 'Labal-Free' should be 'Label-Free'.
  2. [Table 17 caption] The caption contains 'folloing', which should be 'following'.
  3. [§6] The human preference study relies on a single annotator. The paper acknowledges this, but the conclusion that 'unpreferred explanations are more diverse' should be presented as anecdotal evidence from one annotator rather than a general property of model explanations.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the human-versus-model explanation comparison is label-controlled and benchmarked against external HJDs; the main caveats are same-group method reuse and a free softmax temperature.

full rationale

The central comparison is not circular: both the human-explanation and model-explanation MJDs are conditioned on the same few human labels, so the label signal is held fixed and the only manipulated variable is the explanation source. The paper states this explicitly: 'the only difference is whether the explanations are human-annotated or LLM-generated' (Section 3.2). The evaluation target is the external ChaosNLI HJD, not a quantity constructed from the model explanations, and the label-guided model explanations are compared against this external benchmark rather than being fitted to it. The noise-replacement ablation further shows that explanation content, not just the label multinomial, drives the result: 'noise replacements deteriorate performances significantly' (Section 5). The MJD Estimator is inherited from Chen et al. (2024), which overlaps with the present authors, but it is shared by both comparison arms and is itself evaluated against external HJDs, so this is method reuse rather than a self-citation chain that forces the conclusion. The main caveat is the free softmax temperature (tau = 10 for GPT-4o and tau = 20 for Mixtral) in Appendix B.1, which the paper declares 'without loss of generality'; this is a robustness limitation and a possible confound for the absolute KL values, but there is no evidence or equation showing that the temperature was fitted to make the human/model comparison, and Llama3, which uses a parameter-free normalization, shows the same pattern. No self-definitional equation, fitted-input-as-prediction, imported uniqueness claim, or ansatz-via-citation reduction was found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The main free parameters are the per-model softmax temperatures. The axioms are the validity of the first-token probability method and the representativeness of the source datasets. No new entities are introduced.

free parameters (2)
  • Softmax temperature tau (GPT-4o) = 10
    Chosen per model for GPT-4o logits, no justification. Affects the MJD probabilities.
  • Softmax temperature tau (Mixtral) = 20
    Chosen per model for Mixtral logits, no justification.
assumptions (4)
  • domain assumption First-token probabilities from MCQA prompts reflect the LLM's judgment distribution after permutation averaging.
    Appendix B.1 describes the MJD Estimator; this is the core measurement assumption.
  • domain assumption ChaosNLI HJD is the gold standard for human judgment distributions.
    Section 3.1; the evaluation target. If ChaosNLI HJD is not representative, the comparisons lose meaning.
  • domain assumption The few human labels in VariErr and MNLI are drawn from the same distribution as the HJD being approximated.
    Section 3.2 uses these labels as guidance; if they are biased, the label-guided MJDs inherit the bias.
  • domain assumption Averaging over permutations and 'n in one' settings removes prompt bias.
    Appendix B.1; this is a common but unproven heuristic for debiasing LLM prompts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI." pith.science (2026). https://pith.science/paper/7BHBPEPZ

@misc{pith2026241213942,
  author       = {Pith},
  title        = {Pith review of: A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BHBPEPZ}},
  note         = {Machine review of arXiv:2412.13942}
}
read the original abstract

Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs). Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) and large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs. However, collecting explanations for every label is still time-consuming. This paper examines whether LLMs can be used to replace humans in generating explanations for approximating HJD. Specifically, we use LLMs as annotators to generate model explanations for a few given human labels. We test ways to obtain and combine these label-explanations with the goal to approximate human judgment distributions. We further compare the resulting human with model-generated explanations, and test automatic and human explanation selection. Our experiments show that LLM explanations are promising for NLI: to estimate HJDs, generated explanations yield comparable results to human's when provided with human labels. Importantly, our results generalize from datasets with human explanations to i) datasets where they are not available and ii) challenging out-of-distribution test sets.

Figures

Figures reproduced from arXiv: 2412.13942 by the authors.

Figure 1
Figure 1. Recent research has shown that LLMs can approximate a human judgment distribution (HJD) in natural language inference (NLI) with the help of human explanations, as shown in the upper part. While human explanations are still relatively expensive and scarce in most datasets, we ask in the lower part: Can LLMs provide reasonable generated explanations for different NLI labels to approximate HJD? Recent research propose… view at source ↗
Figure 2
Figure 2. Illustration of the process of generating model [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Llama3 Visualization using a ternary plot ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Visualization of MJDs from GPT-4o, with hu [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Results for ablation study. The green bars represent the performance of MJDs when replaced by model [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of gradually replacing human [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Llama3 with human explanations. 0 10 20 30 40 50 60 70 80 90 100 100 90 80 70 60 50 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 Entailment Neutral Contradiction Chaos 0 10 20 30 40 50 60 70 80 90 100 100 90 80 70 60 50 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 …
Figure 8
Figure 8. Figure 8: Llama3 with Lable-Free first model explanations. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Llama3 with Lable-Free longest model explanations. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Llama3 with VariErr Label-Guided first model explanations. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Llama3 with VariErr Label-Guided longest model explanations. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Llama3 with MNLI Label-Guided first model explanations. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Llama3 with MNLI Label-Guided longest model explanations. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Mixtral Visualization. 25% 50% 75%100% Proportion 0.23 0.24 0.25 0.26 0.27 0.28 0.29 Value human-ex Llama3 KL model-ex noise 25% 50% 75%100% Proportion 0.245 0.250 0.255 0.260 0.265 0.270 0.275 Value human-ex Llama3 JSD model-ex noise 25% 50% 75%100% Proportion 0.26 0…
Figure 15
Figure 15. Figure 15: Results for ablation study (Llama3) on gradually replacing first model/noise explanations. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 17
Figure 17. Figure 17: Visualization gradually replacing first noise [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Visualization for gradually replacing first [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Visualization gradually replacing first noise [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Visualization for gradually replacing longest [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 16
Figure 16. Figure 16: Visualization for gradually replacing first [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 16 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hasan Abu - Rasheed, Christian Weber, and Madjid Fathi. 2024. https://doi.org/10.1109/EDUCON60312.2024.10578654 Knowledge graphs as context sources for llm-based explanations of learning recommendations . In IEEE Global Engineering Education Conference, EDUCON 2024, Kos Island, Greece, May 8-11, 2024 , pages 1--5. IEEE

    arXiv 2024

  4. [4]

    Lora Aroyo and Chris Welty. 2015. https://doi.org/10.1609/AIMAG.V36I1.2564 Truth is a lie: Crowd truth and the seven myths of human annotation . AI Mag. , 36(1):15--24

    doi:10.1609/aimag.v36i1.2564 2015

  5. [5]

    Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.124 Stop measuring calibration when humans disagree . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892--1915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    doi:10.18653/v1/2022.emnlp-main.124 2022

  6. [6]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://doi.org/10.18653/V1/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015 , pages 632--642. The Ass...

    doi:10.18653/v1/d15-1075 2015

  7. [7]

    seeing the big through the small

    Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. 2024. https://aclanthology.org/2024.findings-emnlp.842 "seeing the big through the small": Can llms approximate human judgment distributions on NLI from a few explanations? In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA,...

    work page 2024

  8. [8]

    Cheng-Han Chiang and Hung-yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607--15631, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.870 2023

Show all 57 references
  1. [9]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. https://doi.org/10.1007/11736790\_9 The PASCAL recognising textual entailment challenge . In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, F...

    2005 doi

  2. [10]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/V1/N19-1423 BERT: pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Associat...

    2019 doi

  3. [11]

    Luc Devroye and G \' a bor Lugosi. 2001. Combinatorial methods in density estimation. Springer series in statistics. Springer

    2001

  4. [12]

    Chiara Di Bonaventura , Lucia Siciliani, Pierpaolo Basile, Albert Merono Penuela , and Barbara McGillivray. 2024. Is Explanation All You Need? An Expert Survey on LLM-generated Explanations for Abusive Language Detection

    2024

  5. [13]

    Ricardo Dominguez - Olmedo, Moritz Hardt, and Celestine Mendler - D \" u nner. 2023. https://doi.org/10.48550/ARXIV.2306.07951 Questioning the Survey Responses of Large Language Models . CoRR, abs/2306.07951

    2023 doi arXiv

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 doi arXiv

  7. [15]

    Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield - Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. ...

    2023 doi arXiv

  8. [16]

    Schindelin

    Dominik Maria Endres and Johannes E. Schindelin. 2003. https://doi.org/10.1109/TIT.2003.813506 A new metric for probability distributions . IEEE Trans. Inf. Theory , 49(7):1858--1860

    2003

  9. [17]

    Dirk Fahland, Fabiana Fournier, Lior Limonad, Inna Skarbovsky, and Ava J. E. Swevels. 2024. https://doi.org/10.48550/ARXIV.2401.12846 How well can large language models explain business processes? CoRR, abs/2401.12846

    2024 doi arXiv

  10. [18]

    Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fern \'a ndez, and Barbara Plank. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.887 What comes next? evaluating uncertainty in neural text generators against human production variability . In Proceedings of the 2023 Confer...

    2023 doi

  11. [19]

    Cornelia Gruber, Katharina Hechinger, Matthias Assenmacher, G \"o ran Kauermann, and Barbara Plank. 2024. https://aclanthology.org/2024.unimplicit-1.2 More labels or cases? assessing label variation in natural language inference . In Proceedings of the Third Workshop on Unders...

    2024

  12. [20]

    Xingwei He, Zhenghao Lin, Yeyun Gong, A - Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. https://doi.org/10.18653/V1/2024.NAACL-INDUSTRY.15 Annollm: Making large language models to be better crowdsourced annotators . In Proceedings of...

    2024 doi

  13. [21]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In 9th International Conference on Learning Representations, ICLR 2021,...

    2021

  14. [22]

    Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H. Gilpin. 2023. https://doi.org/10.48550/ARXIV.2310.11207 Can large language models explain themselves? A study of llm-generated self-explanations . CoRR, abs/2310.11207

    2023 doi arXiv

  15. [23]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L \' e lio Renard Lavaud, Lucile Saulnier, Marie...

    2024 doi arXiv

  16. [24]

    Nan - Jiang Jiang, Chenhao Tan, and Marie - Catherine de Marneffe. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.712 Ecologically valid explanations for label variation in NLI . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Decemb...

    2023 doi

  17. [25]

    Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79--86

    1951

  18. [26]

    Jenny Kunz and Marco Kuhlmann. 2024. https://doi.org/10.48550/ARXIV.2402.10532 Properties and challenges of llm-generated explanations . CoRR, abs/2402.10532

    2024 doi arXiv

  19. [27]

    Noah Lee, Na An, and James Thorne. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.278 Can large language models capture dissenting human voices? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, ...

    2023 doi

  20. [28]

    Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. 2022. https://doi.org/10.48550/ARXIV.2210.06726 Explanations from large language models make small reasoners better . CoRR, abs/2210.06726

    2022 doi arXiv

  21. [29]

    u ksekg \

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R \' e , Diana Acosta ...

    2023

  22. [30]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  23. [31]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 RoBERTa: A Robustly Optimized BERT Pretraining Approach . CoRR, abs/1907.11692

    2019 arXiv

  24. [32]

    Sebastian Lubos, Thi Ngoc Trang Tran, Alexander Felfernig, Seda Polat Erdeniz, and Viet - Man Le. 2024. https://doi.org/10.1145/3631700.3665185 Llm-generated explanations for recommender systems . In Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation a...

    2024

  25. [33]

    Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, and Dieuwke Hupkes. 2024. Lost in inference: Rediscovering the role of natural language inference for large language models. arXiv preprint arXiv:2411.14103

    2024 arXiv

  26. [34]

    Christopher D. Manning. 2006. https://api.semanticscholar.org/CorpusID:60340094 Local textual inference : It's hard to circumscribe , but you know it when you see it - and nlp needs it

    2006

  27. [35]

    Aditi Mishra, Sajjadur Rahman, Kushan Mitra, Hannah Kim, and Estevam Hruschka. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.484 Characterizing large language models as rationalizers of knowledge-intensive tasks . In Findings of the Association for Computational Linguist...

    2024 doi

  28. [36]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020 a . https://doi.org/10.18653/V1/2020.ACL-MAIN.441 Adversarial NLI: A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for Co...

    2020 doi

  29. [37]

    Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020 b . https://doi.org/10.18653/V1/2020.EMNLP-MAIN.734 What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMN...

    2020 doi

  30. [38]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  31. [40]

    Maja Pavlovic and Massimo Poesio. 2024 a . https://aclanthology.org/2024.nlperspectives-1.11 The effectiveness of LLM s as annotators: A comparative overview and empirical analysis of direct representation . In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP...

    2024

  32. [41]

    Maja Pavlovic and Massimo Poesio. 2024 b . Understanding the effect of temperature on alignment with human opinions. arXiv preprint arXiv:2411.10080

    2024 arXiv

  33. [42]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The `` problem '' of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Ab...

    2022 doi

  34. [43]

    Barbara Plank, Dirk Hovy, and Anders S gaard. 2014. https://doi.org/10.3115/v1/E14-1078 Learning part-of-speech taggers with inter-annotator agreement loss . In Proceedings of the 14th Conference of the E uropean Chapter of the Association for Computational Linguistics , pages...

    2014 doi

  35. [44]

    Oscar Sainz, Iker Garc \'i a-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D ' Amico-Wong, Melissa Dell, Run-Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Sury...

    2024 doi

  36. [45]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. https://proceedings.mlr.press/v202/santurkar23a.html Whose opinions do language models reflect? In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Ho...

    2023

  37. [46]

    Brown, Adam Santoro, Aditya Gupta, Adri \` a Garriga - Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adri \` a Garriga - Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocure...

    2023

  38. [47]

    Székely, Maria L

    Gábor J. Székely, Maria L. Rizzo, and Nail K. Bakirov. 2007. http://www.jstor.org/stable/25464608 Measuring and testing dependence by correlation of distances . The Annals of Statistics, 35(6):2769--2794

    2007

  39. [48]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  40. [49]

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. https://aclanthology.org/2024.emnlp-main.54 Large language models for data annotation and synthesis: A survey . In Proceedings of th...

    2024

  41. [50]

    Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. https://doi.org/10.1162/TACL\_A\_00685 Do llms exhibit human-like response biases? A case study in survey design . Trans. Assoc. Comput. Linguistics, 12:1011--1026

    2024 doi

  42. [51]

    Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. https://doi.org/10.1613/JAIR.1.12752 Learning from disagreement: A survey . J. Artif. Intell. Res., 72:1385--1470

    2021 doi

  43. [52]

    Manya Wadhwa, Jifan Chen, Junyi Jessy Li, and Greg Durrett. 2023. https://doi.org/10.48550/ARXIV.2305.14770 Using Natural Language Explanations to Rescale Human Judgments . CoRR, abs/2305.14770

    2023 doi arXiv

  44. [53]

    Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. https://doi.org/10.1145/3613904.3641960 Human-llm collaborative annotation through effective verification of LLM labels . In Proceedings of the CHI Conference on Human Factors in Computing Systems,...

    2024

  45. [54]

    Leon Weber - Genzel, Siyao Peng, Marie - Catherine de Marneffe, and Barbara Plank. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.123 Varierr NLI: separating annotation error from human label variation . In Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024 doi

  46. [55]

    Riedl, and Yejin Choi

    Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark O. Riedl, and Yejin Choi. 2022. https://doi.org/10.18653/V1/2022.NAACL-MAIN.47 Reframing human-ai collaboration for generating free-text explanations . In Proceedings of the 2022 Conference of the North American Chapter of...

    2022 doi

  47. [56]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. https://doi.org/10.18653/V1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computati...

    2018 doi

  48. [57]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/f5708199bdc013c5b56406db305b991e-Abstract-Conference.html Text alignment is an efficient unified model for massive NLP tasks . In Advances in Neural Information Process...

    2023

  49. [58]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. https://openreview.net/forum?id=shr9PXz7T0 Large language models are not robust multiple choice selectors . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, ...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.