{"id":"b0f64118-5221-4e77-9b6a-fa548714859d","arxiv_id":"2412.13942","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-generated explanations, paired with a few human labels, produce model judgment distributions as close to human judgment distributions as human explanations do on NLI.","lead":"This paper tests whether ChatGPT-style explanations written by a language model can stand in for human-written explanations when estimating how human annotators would distribute labels on natural language inference tasks. If it works, collecting the full distribution of human opinions on a sentence pair would no longer require paying people to explain every label they chose.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temperature choice in the MJD estimator is a free parameter; the reported human-vs-model explanation comparability for GPT-4o/Mixtral may be an artifact of flattening and needs a sensitivity check.","rationale":"Good-faith reading: the paper's practical claim is that when a few human labels are available, LLM-generated explanations can stand in for human explanations in the MJD estimator. The noise-replacement ablation (Section 5) and the OOD ANLI results provide real independent support: replacing human explanations with off-topic content degrades performance, and classifiers trained on MJDs generalize beyond the training distribution. I do not object to the overall direction.\n\nThe load-bearing weak point is the estimator's probability conversion. The comparison between conditions is only meaningful if the first-token probabilities represent something like the model's belief distribution. The temperatures τ=10 and τ=20 are not derived from data or theory, and the paper calls the choice 'without loss of generality,' which is mathematically inaccurate. High temperatures flatten distributions and can make any two explanation conditions look similar; the claimed equivalence could then be an artifact. This is a correctness risk rather than a novelty or scope dispute.\n\nWhy this over missing error bars: error bars would tell us whether the small gaps are reliable, but they would not tell us whether the gaps are small because the explanations are functionally equivalent or because the metric has been smoothed. The temperature sweep is a direct, cheap test of the latter. If the gap survives the sweep, the conditional accept is justified; if not, the claim must be restricted. I therefore keep the reader's conditional verdict unchanged.","tokens_in":45120,"tokens_out":9368,"duration_ms":91006,"concrete_test":"Recompute all Table 1 rows for GPT-4o and Mixtral (and, as a control, Llama3 using softmax on the same logits) with τ ∈ {0.1, 0.5, 1, 2, 5, 10, 20, 50, 100}, keeping permutation averaging and all other pipeline steps fixed. Record the human-ex vs. label-guided model-ex gap in KL, JSD, TVD, and fine-tuning F1 for each τ. If the gap stays below a pre-specified equivalence margin across the whole grid, the comparable-results claim is robust; if the gap or ranking changes materially (e.g., KL gap >0.02 or sign flips), the headline claim is temperature-dependent and should be re-scoped to specific temperatures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1's central comparison—human explanations vs. label-guided model explanations—is made with first-token probabilities derived from softmax at τ=10 (GPT-4o) and τ=20 (Mixtral), described in Appendix B.1 as 'without loss of generality.' This is not WLOG: temperature controls how flat the predicted distributions are, and as τ grows all MJDs approach the uniform distribution, shrinking any apparent difference between explanation conditions. The paper's own Limitations section concedes that this logit-to-probability transformation 'may not be universally suitable for all LLMs.' If the closeness in Table 1 (e.g., GPT-4o KL 0.187 human vs 0.192 model; Mixtral 0.239 vs 0.238) is robust only at these two temperatures, the headline claim that LLM explanations are comparable to human explanations is not established. Llama3 uses a parameter-free normalization and also shows comparable results, which mitigates the concern, but the claim is made across models. A temperature sweep is needed to determine whether the comparison is a statement about explanation content or about the estimator's flatness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper asks whether LLM-generated explanations can substitute for human-written explanations in the MJD Estimator of Chen et al. (2024), which approximates human judgment distributions (HJDs) on NLI from a few labels and explanations. The authors prompt Llama3-70B, Mixtral-8x7B, and GPT-4o to generate multiple explanations per label, then select explanations either label-free (one per class) or label-guided (matching a few human labels). They evaluate the resulting MJDs against ChaosNLI HJDs using KL, JSD, TVD, distance correlation, and downstream fine-tuning of BERT/RoBERTa, and test generalization to MNLI (which lacks human explanations) and to the ANLI benchmark. Ablations replace human explanations with model or noise explanations, and a single-annotator study compares preferred versus unpreferred model explanations. The central claim is that label-guided model explanations yield MJDs comparable to human-explanation MJDs when the same few human labels are used.","tokens_in":45371,"tokens_out":3979,"duration_ms":34952,"significance":"If the claim holds, the paper offers a practical reduction in annotation cost for human label variation estimation and extends HJD approximation to datasets without human explanations. The study has several genuine strengths: multiple LLMs, a noise-replacement control showing that explanation content matters, permutation averaging over label/option/order biases, a public release of the generated explanations, and an OOD evaluation with a candid contamination discussion. The main caveat is that the headline 'comparable' rests on a small number of point estimates and on model-specific softmax temperatures that are not swept, so the central result needs additional robustness evidence before it is fully convincing.","major_comments":[{"comment":"The statement that setting softmax temperature to τ=10 for GPT-4o and τ=20 for Mixtral is done 'without loss of generality' is not supported. Temperature controls the flatness of the first-token probability distributions: as τ grows, every MJD converges toward the uniform distribution, which mechanically shrinks KL/JSD/TVD differences between any two conditions. The near-equality of GPT-4o human (KL 0.187) versus VariErr label-guided (KL 0.192) and Mixtral human (KL 0.239) versus label-guided (KL 0.238) could therefore be an artifact of the chosen temperatures rather than evidence about explanation content. Llama3's parameter-free normalization provides an important counterexample, but the paper makes the claim across models. Please add a temperature sweep for GPT-4o and Mixtral and show that the human-vs-model explanation gap remains small and stable, or qualify the claim to the specific estimator configuration used.","section":"Appendix B.1; Table 1"},{"comment":"The central comparison reports only point estimates. The differences between human and label-guided model explanations are small (e.g., Llama3 KL 0.238 vs. 0.234; GPT-4o KL 0.187 vs. 0.192), but no confidence intervals, standard errors, paired significance tests, or bootstrap resampling over the 341 instances are reported. The fine-tuning comparisons also appear to be single runs without seed variation. Without uncertainty quantification, 'comparable' is not distinguishable from 'not distinguishable at this sample size.' Please report paired bootstrap confidence intervals or an equivalent paired test on the distribution metrics, and report variance over multiple fine-tuning seeds.","section":"Table 1; §3.2"},{"comment":"The noise-replacement ablation, which is the main evidence that explanation content rather than the human labels drives the result, is run only with Llama3. The analogous content control is missing for GPT-4o and Mixtral, even though the paper's headline claim is made for all three LLMs. For GPT-4o the label-free condition is fairly close to the label-guided condition on several metrics (e.g., KL 0.252 vs. 0.192), so without a noise control it is unclear how much of the label-guided improvement is due to explanation relevance. Please extend the noise replacement to at least one additional model, or explicitly restrict the content-matters conclusion to Llama3.","section":"§5; Table 1"}],"minor_comments":[{"comment":"There are several typos in the appendix: 'promo' should be 'prompt', 'Protocals' should be 'Protocols', and 'Labal-Free' should be 'Label-Free'.","section":"Appendix B.1; Appendix B.3"},{"comment":"The caption contains 'folloing', which should be 'following'.","section":"Table 17 caption"},{"comment":"The human preference study relies on a single annotator. The paper acknowledges this, but the conclusion that 'unpreferred explanations are more diverse' should be presented as anecdotal evidence from one annotator rather than a general property of model explanations.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"No concerns about attribution or novelty disclosure; the method is built transparently on Chen et al. (2024) and the paper is within scope. The main risk is the temperature sensitivity of the MJD estimator for the closed and mixture-of-experts models. If the authors provide a temperature sweep and uncertainty measures, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's core claim, that LLM-generated explanations, when selected using a few human labels, approximate the value of human explanations for HJD estimation, is new relative to Chen et al. (2024) and mostly holds up. The noise-replacement control and the OOD ANLI experiments are the strongest evidence. The main weakness is the free temperature parameter in the MJD estimator for GPT-4o and Mixtral; a sensitivity sweep is needed before I'd trust the 'comparable' wording across models. Llama3 uses a parameter-free normalization and shows the same pattern, which mitigates but does not remove the concern.\n\nWhat's actually new: Chen et al. needed human explanations; this paper shows you can prompt an LLM to generate explanations for each label, select them either label-free or guided by a few human labels, and get MJDs close to those from human explanations. The design isolates the explanation source well, since both conditions use the same few human labels. The paper ships code and data, which is reproducible evidence. The noise-replacement ablation is a good control: replacing human explanations with irrelevant model explanations hurts, so the result isn't just the labels doing all the work. Generalization to MNLI (no explanations) and ANLI (OOD) is a real practical benefit.\n\nSoft spots, in order. First, the temperature. Appendix B.1 says 'without loss of generality' for tau=10 (GPT-4o) and tau=20 (Mixtral), but that's not WLOG: softmax temperature controls the flatness of the predicted distribution, and as tau grows, the comparison between explanation conditions can be artificially compressed toward uniform. The paper's own Limitations section concedes the logit-to-probability transformation 'may not be universally suitable for all LLMs.' That should be flagged in review. Because Llama3 uses a parameter-free normalization and reproduces the comparable result, I don't think this sinks the paper, but a temperature sweep (or a justification of the chosen values) is required. Second, the 'comparable' claim is made without confidence intervals or significance tests; the differences between human and model explanation conditions are small but the reader can't tell if they're within noise. Third, the human preference study uses a single annotator; the finding that 'unpreferred' explanations work better is interesting but anecdotal. Minor: the label-free condition is weaker, as expected, which is fine.\n\nBottom line: this is a solid empirical contribution to the HLV/NLI community. It deserves a serious referee and likely acceptance after adding error bars and the temperature sensitivity check. I'd bring it to reading group and would cite it if I worked on label distributions.","headline":"LLM-generated explanations guided by a few human labels look like a real substitute for human explanations in HJD estimation, but the free softmax temperature needs a sensitivity check before the comparability claim is fully trusted.","tokens_in":45842,"tokens_out":2728,"would_cite":true,"duration_ms":25857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated explanations, paired with a few human labels, match human-written explanations when estimating human judgment distributions on natural language inference.","keywords":["human judgment distributions","natural language inference","label variation","LLM-generated explanations","model judgment distribution","explanation selection","annotation cost","out-of-distribution generalization"],"falsifier":"Collect the same 341 VariErr/ChaosNLI instances and compare label-guided model-explanation MJDs against MJDs obtained by sampling many complete answers at nonzero temperature, say 100 samples per instance; if the sampled label frequencies diverge systematically from the first-token-probability distributions, the estimator's central measurement is invalid and the human-versus-model comparison would need to be redone. Alternatively, an instance set where human explanations encode idiosyncratic world knowledge absent from the premise should show a clear gap whenever that knowledge drives the human label.","tokens_in":44974,"feed_emoji":"🤖","tokens_out":5255,"duration_ms":44624,"temperature":0.7,"pith_summary":"Human annotators often disagree on natural language inference (NLI) labels, and capturing that disagreement as a human judgment distribution is costly. This paper tries to establish that the expensive human-written explanations used to elicit such distributions can be replaced by LLM-generated explanations, provided a few human labels guide which machine explanations to select. Across three LLMs and two NLI datasets, the resulting model judgment distributions come close to those produced with human explanations and to the gold ChaosNLI human judgment distribution. If true, this would let researchers estimate label disagreement on datasets that have labels but no explanations, at much lower annotation cost.","feed_headline":"LLM explanations match human ones for labeling NLI","feed_subtitle":"Pairing a few human labels with machine-written reasons recovers crowd disagreement on NLI at a fraction of the cost.","key_machinery":"The load-bearing mechanism is the Model Judgment Distribution (MJD) Estimator: an LLM answers a multiple-choice question about whether a premise entails, contradicts, or is neutral toward a hypothesis, and its judgment distribution is read from the first-token probabilities of the answer choices, with option, sequence, and length biases removed by averaging over permutations and 'n in one' combinations. Model explanations are generated by prompting the LLM to list all possible explanations for a given label, then selecting explanations either label-free (one per label) or label-guided (matching the label counts of a few human annotations, such as three entailment, one neutral, one contradiction), using the longest or first explanations. The decisive comparison is between human explanations and model explanations under identical label guidance, which isolates whether explanation content or the human labels drive the approximation.","core_discovery":"The central claim is that LLM-generated explanations are good proxies for human explanations inside the MJD Estimator pipeline for NLI: when the same few human labels guide explanation selection, swapping human-written explanations for machine-written ones yields model judgment distributions with comparable KL divergence, Jensen-Shannon distance, and total variation distance to the gold human judgment distribution, and classifiers fine-tuned on those distributions perform similarly. The paper further claims this transfers from datasets with human explanations to explanation-free datasets such as MNLI, and that classifiers trained on the resulting distributions improve out-of-domain ANLI performance. The title's rose metaphor states the point: the explanation's origin matters less than the label guidance it carries.","pith_inferences":["Going beyond the paper, the same label-guided explanation pipeline could be tested on other disagreement-rich annotation tasks, such as toxicity or relevance judgment, to see whether the proxy result is specific to NLI or general to LLM explanations.","Because the MJD is read from one LLM's first-token probabilities, the approach inherits that model's priors; ensembling explanations or MJDs from multiple LLMs is a natural extension the paper leaves open and might close more of the gap to human distributions.","The observation that explanations a single annotator rejected sometimes outperform preferred ones suggests diversity, not human-likeness, may be the active ingredient; a controlled study that varies explanation diversity while holding label guidance fixed could separate these factors.","A practical caution that follows from the noise-replacement ablation: the method's success depends on generated explanations staying on-topic for the correct label, so real applications should monitor explanation relevance rather than assume any generated text helps."],"forward_implications":["Estimating human judgment distributions no longer requires collecting human-written explanations, so NLI datasets with only a few labels per instance can be turned into distribution-labeled training data.","The label-guided strategy transfers to datasets without any explanations, such as MNLI, with results comparable to the human-explanation condition.","Classifiers fine-tuned on model judgment distributions trained with label-guided explanations outperform models trained without distribution information on adversarial ANLI test sets.","Replacing human explanations with model explanations one by one leaves performance nearly flat, while replacing them with off-topic noise explanations clearly degrades results, showing that the relevance of explanation content matters.","Explanation variability across generated explanations may serve as an indicator of how well a set of explanations captures human label variation."],"supporting_citations":[{"why":"Introduces the MJD Estimator and the human-explanation baseline that this paper extends and compares against.","marker":"Chen et al. (2024)"},{"why":"Supplies VariErr NLI, whose expert label-explanation pairs form the human-explanation condition and the label guidance.","marker":"Weber-Genzel et al. (2024)"},{"why":"Provides ChaosNLI's 100-annotator human judgment distributions used as the gold target.","marker":"Nie et al. (2020b)"},{"why":"Supplies MNLI's five labels per instance, the explanation-free dataset used for label-guided transfer.","marker":"Williams et al. (2018)"},{"why":"Provides Llama3, one of the LLMs used for explanation generation and MJD estimation.","marker":"Dubey et al. (2024)"},{"why":"Provides Mixtral, the second open LLM tested in the experiments.","marker":"Jiang et al. (2024)"},{"why":"Provides GPT-4o, the closed LLM tested in the experiments.","marker":"OpenAI (2023)"},{"why":"Supplies the first-token probability method used to read the MJD from LLM answer logits.","marker":"Santurkar et al. (2023)"},{"why":"Supplies the lexical, syntactic, and semantic similarity measures used to compare explanation variability.","marker":"Giulianelli et al. (2023)"}],"fun_headline_variants":["LLM reasons rival human ones for NLI label distributions","Machine-written explanations stand in for human ones on NLI","With human labels, LLM explanations predict NLI disagreement","NLI label distributions: LLM explanations are human-like","When labels guide them, LLM reasons rival humans on NLI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on treating the first-token probabilities the LLM assigns to the three answer choices, after permutation averaging and temperature scaling, as a faithful opinion distribution; if that measurement does not reflect the LLM's genuine judgment, then both the human-explanation and model-explanation estimates are unreliable even if they look similar.","fun_headline_variants_meta":{"raw":{"variants":["LLM reasons rival human ones for NLI label distributions","Machine-written explanations stand in for human ones on NLI","With human labels, LLM explanations predict NLI disagreement","NLI label distributions: LLM explanations are human-like","When labels guide them, LLM reasons rival humans on NLI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000943,"raw_usage":{"total_tokens":3997,"prompt_tokens":883,"completion_tokens":3114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":3031}},"tokens_in":499,"tokens_out":3114,"duration_ms":22794,"temperature":1.0,"reasoning_tokens":3031,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:37:23.882583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect the same 341 VariErr/ChaosNLI instances and compare label-guided model-explanation MJDs against MJDs obtained by sampling many complete answers at nonzero temperature, say 100 samples per instance; if the sampled label frequencies diverge systematically from the first-token-probability distributions, the estimator's central measurement is invalid and the human-versus-model comparison would need to be redone. Alternatively, an instance set where human explanations encode idiosyncratic world knowledge absent from the premise should show a clear gap whenever that knowledge drives the human label.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Mixtral, the second open LLM tested in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the first-token probability method used to read the MJD from LLM answer logits."}],"review_version":1}