Pith. sign in

REVIEW 4 major objections 5 minor 15 references

Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that fine-tuning a language model on just a few dozen explanation-annotated examples from one source dataset transfers almost as well to 19 unseen NLI-related datasets as fine-tuning on the full source, and that the…

desk verdict Solid large-scale OOD self-rationalization study, but Finding 3's accuracy–explanation link is partly an artifact of scoring wrong predictions against the gold label. read the letter →

arxiv 2502.04797 v1 pith:Q42Z4DJD submitted 2025-02-07 cs.CL

Jing Yang , Max Glockner , Anderson Rocha , Iryna Gurevych This is my paper · ORCID
classification cs.CL
keywords self-rationalizationfree-textexplanationsout-of-distributiongeneralizationnaturallanguageinferencefact-checkinghallucinationdetectionexplanationevaluationfew-shotfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that self-rationalization—training a model to output a label together with a free-text explanation—can be learned from a small number of examples from one explanation-annotated dataset and still transfer to a broad range of out-of-distribution NLI-related tasks. It fine-tunes T5-Large and OLMo-7B on e-SNLI or e-FEVER with 1 to 128 shots and various selection strategies, then evaluates on 19 datasets spanning NLI, fact-checking, and hallucination detection. The central positive finding is that few-shot fine-tuning performs comparably to full-dataset fine-tuning, that the choice of source dataset matters more than sample selection, and that the T5-11B Acceptability score correlates best with human judgments of explanation quality among the reference-free metrics tested.

What carries the argument

The carrying mechanism is the Acceptability score: a T5 model trained to judge whether a free-text explanation is acceptable for a given premise–hypothesis–label triple, used in two sizes. A T5-Large version serves as a filter during sample selection, removing fine-tuning examples whose explanations score below 0.3, and an 11B version serves as the reference-free evaluation metric. The other machinery is the prompt-based fine-tuning template that makes a generative model emit both label and explanation, plus the 19-dataset OOD evaluation harness that measures macro F1 for labels and Acceptability for explanations.

What would settle it

Train the Acceptability scorer on a corpus that excludes SNLI and e-SNLI entirely, re-score the explanations of the 13 human-evaluated models on SICK, VitaminC, and XSUM Hallucination, and check whether e-SNLI-trained models still receive the highest Acceptability scores and whether the ranking still matches human judgments.

Watch

Extended reading notes

Core claim

The paper's central claim is that a model fine-tuned on a few dozen explanation-annotated examples from one source distribution generalizes out-of-distribution almost as well as a model fine-tuned on the entire source, while producing explanations rated nearly as highly by humans. In the human study of 13 selected models on three datasets, the Acceptability score (T5-11B) showed the strongest Spearman correlation with human scores (0.484 combined), ahead of Themis, Auto-J, and TigerScore. The paper also reports that e-FEVER-trained models achieve higher average OOD label F1, especially on fact-checking and hallucination-detection tasks, whereas e-SNLI-trained models produce higher Acceptability scores, and that label accuracy and Acceptability are positively associated across models.

Load-bearing premise

The explanation quality results lean on a scorer (the Acceptability model) that was itself trained in part on SNLI data, the same source family as the e-SNLI fine-tuning data, so its judgments may be skewed toward e-SNLI-trained models.

Editorial extensions

If this is right

  • Practitioners can build self-rationalizing classifiers for new NLI-like tasks with tens of explanation-annotated examples instead of thousands.
  • The Acceptability score can serve as a cheap, reference-free substitute for human annotation when comparing explanation quality across models.
  • Choosing the source explanation dataset is a bigger lever than sophisticated sample selection, so data-source choice should be the first design decision.
  • Label accuracy and explanation acceptability move together, so improving one tends not to sacrifice the other in these settings.
  • Long-document tasks (HDAS) are the hardest for explanation quality, indicating a practical boundary for current self-rationalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because part of SNLI was used to train the Acceptability scorer, the e-SNLI-trained models' Acceptability advantage may partly reflect scorer overlap; a scorer trained on disjoint data would test whether the advantage is a genuine explanation-quality gap.
  • Editorial inference: The few-shot success opens a cheap recipe for new tasks—draft candidate explanations with a general LLM, filter them with Acceptability, fine-tune—that the paper does not test directly.
  • Editorial inference: The positive label-accuracy/Acceptability link suggests Acceptability could double as an OOD confidence signal, but the low score range on HDAS shows the signal is task-dependent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies out-of-distribution self-rationalization by fine-tuning T5-Large and OLMo-7B on e-SNLI or e-FEVER with varying numbers of shots and several sample-selection strategies, then evaluating the 704 resulting models on 19 NLI-related datasets across three tasks. It introduces an acceptability-filtered sample-selection method, measures label prediction with Macro F1, and evaluates generated explanations via a human study on 13 selected models plus four reference-free automatic metrics. The main claims are that few-shot fine-tuning approaches full-shot OOD performance, that the fine-tuning data source matters more than sample selection, and that Acceptability (T5-11B) is the most human-correlated automatic metric and is positively related to label accuracy.

Significance. If the findings hold, the paper provides a useful large-scale empirical resource: transparent models, 19 diverse OOD datasets with contamination-aware exclusions, 704 trained models, code release, and a human-evaluation anchor for reference-free explanation metrics. The comparison of T5 and OLMo under identical fine-tuning protocols is valuable, and the paper is transparent about several circularity risks, including a control experiment in Section 6.2. However, the central claims are currently weakened by a contamination issue (WNLI) and by a mechanical confound between F1 and Acceptability, so the present version overstates the strength of its evidence.

major comments (4)
  1. [Section 4.1, Table 1] The paper states that datasets used for supervised fine-tuning of T5 were excluded to avoid contamination, yet WNLI is included in the OOD evaluation. WNLI is part of GLUE, and T5-Large was fine-tuned on all GLUE tasks in Raffel et al. (2020). WNLI is therefore in-distribution for T5-Large, and its inclusion inflates the average OOD F1 and Acceptability scores for T5 models in Figure 2 and Table 5. Please remove WNLI from the aggregate OOD results or analyze it separately and re-report the affected averages and conclusions.
  2. [Section 5.2.2, Appendix A.5] The Acceptability score is computed with the gold label as the answer for all samples, including those where the model's predicted label is wrong. Because a self-rationalizing model generates an explanation for its predicted label, a wrong prediction is scored against a conflicting answer and becomes unacceptable by construction, independent of explanation quality. This makes the positive F1-Acceptability relationship in Figure 3 and Finding 3 (Section 5.3.1) partly mechanical: lower-F1 models have more instances where the gold label conflicts with the generated label. The human study restricts to correctly predicted instances (Section 5.2.1), but the automatic Acceptability scores do not. Please recompute Acceptability on correctly predicted instances only, or otherwise condition on the predicted label, and re-examine whether the F1-Acceptability trend and the Pareto-front selection survive.
  3. [Section 5.2.1, Table 3] The human evaluation covers only 13 of 704 models and 3 of 19 datasets, with instances restricted to those correctly predicted by the model and models selected for high and comparable F1. The abstract and Section 5.2.3 claim that Acceptability is the most human-correlated reference-free metric for NLI-related explanation evaluation, but this is an extrapolation from a narrow, F1-biased sample. The paper should temper the claim to the evaluated setting, report per-model correlation ranges, and discuss how the correctness restriction and the high-F1 model selection affect the correlation estimates.
  4. [Section 5.2.4, Table 4, Section 6.2] The paper concludes that the acceptability filter improves explanation quality, but the evidence is based on the Acceptability metric itself, which is a larger version of the filter model (T5-11B vs T5-Large). The Section 6.2 control with a Themis filter covers only one OLMo model and only tests Acceptability scores, not human judgments. In Table 4, human scores for T5 e-FEVER 128-shot do not show a consistent improvement from the acceptability filter (accept-FastVote-k: 0.611, FastVote-k: 0.589, random: 0.623). The claim that data filtering improves explanation quality is therefore not supported by the human data and should be scaled back or supported by a human-annotated control experiment.
minor comments (5)
  1. [Section 5.1] 'This is further indicted by the results' should read 'This is further indicated by the results'.
  2. [Table 3] The text 'significant with ρ < 0.001' should read 'significant with p < 0.001', since ρ denotes the Spearman correlation coefficient rather than the p-value.
  3. [Appendix A.2, Eqs. (1)-(2)] The definition of pmean as the midpoint between pmax and pmin is unusual; please clarify whether the ambiguity score is the distance to this global midpoint rather than to the per-instance mean, and justify why this approximates the ambiguity notion of Swayamdipta et al. (2020).
  4. [Figure 3] The 'Trend' lines are shown without confidence intervals or a stated regression method; please add the regression equation, error bars, and significance level, especially because the correlation is central to Finding 3.
  5. [Section 5.2.3] Reporting only the point Spearman correlations without confidence intervals makes it hard to assess the stability of the ranking across the three small evaluation sets; please include confidence intervals or per-dataset variability.

Circularity Check

1 steps flagged · score 6.0 of 10

Finding 3's F1–Acceptability correlation is partly built into the metric: Acceptability is scored against the gold label even when the model predicted a different label, so wrong predictions are penalized by construction.

  1. self definitional [Section 5.2.2 and Appendix A.5; used in Section 5.3.1 (Figure 3) and Section 6.3]
    "For all reference-free metrics, we calculate the scores for all samples in the datasets, given ground truth inputs (hypothesis, premise, and gold label). ... Acceptability score premise: [premise] hypothesis: [hypothesis] answer: [gold label] explanation: [explanation] ... higher Acceptability scores are usually associated with better F1 scores."

    The Acceptability model is asked whether the explanation justifies the supplied answer, and the paper supplies the gold label as that answer for every instance, including instances where the model predicted a different label. Since the explanation was generated for the predicted label, any wrong prediction is scored as unacceptable by construction, regardless of how coherent the explanation is. Aggregating these scores over all instances therefore builds label accuracy into the explanation-quality measure. The positive F1–Acceptability trend in Figure 3, Table 5, and Section 6.3 is thus partly mechanical: low-F1 models have more instances in which the gold answer conflicts with the label the explanation defends, and those instances receive low Acceptability by definition.

full rationale

The paper's main transfer and few-shot results are self-contained and do not depend on a self-citation chain: the Acceptability filter and metric come from Wiegreffe et al. (2022), with no author overlap, and the paper explicitly tests the filter/metric lineage with a Themis-filter control (Table 13), mitigating that concern. The remaining circularity is narrower but real: the Acceptability score is computed with the gold label as the answer for all samples, so for wrong predictions the metric is definitionally low. This makes the third finding—that higher label accuracy produces better explanations—partially an artifact of the evaluation setup rather than an independent empirical discovery. The paper is transparent about the SNLI lineage of the Acceptability model, but it does not restrict its automatic Acceptability analysis to correctly predicted instances, so the F1–Acceptability correlation in Figure 3 and the binning analysis in Figure 4 remain confounded. Score 6 reflects one central claim that is partly circular by construction, while the rest of the evaluation stands on independent human judgments and external benchmarks.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its contribution is purely empirical, so the ledger consists of hand-chosen thresholds, borrowed model hyperparameters, and assumptions about data contamination and evaluation validity.

free parameters (4)
  • acceptability threshold = 0.3
    Samples with acceptability scores below 0.3 are removed from fine-tuning data. The threshold is chosen by the authors, not derived, and its value affects which training samples are used.
  • FastVote-k k = 150
    The number of selected samples for FastVote-k is set to the recommended k=150 from Su et al. (2022), not tuned here.
  • T5-Large learning rate = 3e-7 to 3e-4
    Learning rates are selected per source dataset and shot count based on the in-distribution validation set, which includes explanation BERTScore, so the choice is fit to ID data and may affect OOD explanation quality.
  • OLMo LoRA hyperparameters = r=16, alpha=16, lr=2e-4
    These are taken from Zarharan et al. (2024) rather than tuned here, but they are still hand-chosen and could affect OOD results.
assumptions (5)
  • domain assumption The Acceptability model from Wiegreffe et al. (2022) provides a valid reference-free measure of explanation quality for NLI-related tasks.
    Invoked in Sections 3.2 and 5.2: filtering and evaluation both rely on this model. Its training data (SNLI) overlaps with the e-SNLI source distribution.
  • domain assumption Macro F1 averaged over 19 datasets is a meaningful measure of OOD label prediction performance.
    Used in Figure 2 and Table 5. Datasets vary greatly in size and difficulty, so the average may not reflect performance on any specific dataset.
  • domain assumption The 13 selected models for human evaluation are representative of the 704 fine-tuned models.
    Section 5.2.1: models are selected by high and comparable F1 on three datasets. This selection may bias the human evaluation results toward better-performing models.
  • domain assumption T5-Large and OLMo-7B have transparent pretraining data that does not contain the OOD evaluation datasets.
    Section 4.1: the paper excludes datasets used in T5 supervised fine-tuning and relies on transparent OLMo pretraining. This is an assumption about data contamination, acknowledged but not fully verified for all 19 datasets.
  • domain assumption Human judgments from Prolific workers, with three workers per instance, are a reliable ground truth for explanation quality.
    Section 5.2.1: this is the benchmark for all automatic metrics. The paper does not report inter-annotator agreement for its own annotation study, only for the original datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks." pith.science (2026). https://pith.science/paper/Q42Z4DJD

@misc{pith2026250204797,
  author       = {Pith},
  title        = {Pith review of: Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q42Z4DJD}},
  note         = {Machine review of arXiv:2502.04797}
}
read the original abstract

Free-text explanations are expressive and easy to understand, but many datasets lack annotated explanation data, making it challenging to train models for explainable predictions. To address this, we investigate how to use existing explanation datasets for self-rationalization and evaluate models' out-of-distribution (OOD) performance. We fine-tune T5-Large and OLMo-7B models and assess the impact of fine-tuning data quality, the number of fine-tuning samples, and few-shot selection methods. The models are evaluated on 19 diverse OOD datasets across three tasks: natural language inference (NLI), fact-checking, and hallucination detection in abstractive summarization. For the generated explanation evaluation, we conduct a human study on 13 selected models and study its correlation with the Acceptability score (T5-11B) and three other LLM-based reference-free metrics. Human evaluation shows that the Acceptability score correlates most strongly with human judgments, demonstrating its effectiveness in evaluating free-text explanations. Our findings reveal: 1) few annotated examples effectively adapt models for OOD explanation generation; 2) compared to sample selection strategies, fine-tuning data source has a larger impact on OOD performance; and 3) models with higher label prediction accuracy tend to produce better explanations, as reflected by higher Acceptability scores.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 12 canonical work pages

  1. [4]

    babies” are “little

    arXiv preprint arXiv:2304.03439. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language mod- els during continual fine-tuning. arXiv preprint arXiv:2308.08747. Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew Peters. 2022. Few-Shot Self- Rationalization with Natural Lang...

    arXiv 2023

  2. [6]

    Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets

    ConjNLI: Natural Language Inference Over Conjunctive Sentences. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8240–8252, Online. Association for Computa- tional Linguistics. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation i...

    work page Pith review arXiv 2020

  3. [7]

    In Proceed- ings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computa- tional Linguistics: Human Language Technolo- gies, pages 624–643, Online

    Get your vitamin C! robust fact verifi- cation with contrastive evidence. In Proceed- ings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computa- tional Linguistics: Human Language Technolo- gies, pages 624–643, Online. Association for Computational Linguistics. Luca Soldaini, Rodney Kinney, Akshita Bha- gia, Dustin Schwenk...

    work page 2021

  4. [10]

    In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online

    Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Com- putational Linguistics. Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R Bowman. 2019a. GLUE: A Multi-Tas...

    work page 2019

  5. [11]

    Reframing Human-AI Collaboration for Generating Free-Text Explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pages 632–658, Seattle, United States. Association for Computational Linguistics. Sarah Wiegreffe, Ana Marasovi ´c, and Noah A. Smith. 2...

    work page 2022

  6. [12]

    The relevant information about the claim is lacking in the context

    Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3486–3501. Majid Zarharan, Pascal Wullschleger, Babak Behkam Kia, Mohammad Taher Pilehvar, and Jennifer Foster. 2024. Tell me why: Explainable public health fact-checking with ...

    work page 2022

  7. [14]

    Ordinal Common-sense Inference (JOCI) (Zhang et al., 2017) We follow Karimi Ma- habadi et al

    and not_entailment (when the score is no greater than 3, anything between 3 and 4 are removed), fol- lowing the literature convention (Karimi Mahabadi et al., 2020). Ordinal Common-sense Inference (JOCI) (Zhang et al., 2017) We follow Karimi Ma- habadi et al. (2020) by mapping the labels very likely to entailment; likely, plausible and technically possibl...

    work page 2020

  8. [15]

    explanation:

    We concatenate the premise sentences to- gether to form one premise paragraph. SciFact (Wadden et al., 2020) The dataset does not have public available labels for test set, thus we use the dev set. We do not perform evidence retrieval and use the cited document abstracts as evidence. Climate FEVER (Diggelmann et al., 2020) We use the paragraph-level evide...

    work page 2024

Show all 15 references
  1. [2016]

    We convert the mean human scores into two classes entailed (when the score is no less than

  2. [2017]

    In Proceedings of the Eighth Inter- national Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 100– 109, Taipei, Taiwan

    Natural Language Inference from Multi- ple Premises. In Proceedings of the Eighth Inter- national Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 100– 109, Taipei, Taiwan. Asian Federation of Natural Language Processing. Mike Lewis, Yinhan Liu, N...

  3. [2018]

    In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 738–744, Brussels, Belgium

    BLEU is not suitable for the evaluation of text simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 738–744, Brussels, Belgium. Association for Computational Lin- guistics. Swabha Swayamdipta, Roy Schwartz, Nicholas ...

    2018 arXiv

  4. [2020]

    In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online

    BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguis- tics. ...

    2023

  5. [2021]

    In Pro- ceedings of *SEM 2021: The Tenth Joint Confer- ence on Lexical and Computational Semantics, pages 78–88, Online

    NeuralLog: Natural language inference with joint neural and logical reasoning. In Pro- ceedings of *SEM 2021: The Tenth Joint Confer- ence on Lexical and Computational Semantics, pages 78–88, Online. Association for Computa- tional Linguistics. Jay DeYoung, Sarthak Jain, Nazne...

    2021 arXiv

  6. [2022]

    Association for Computational Linguistics

    Does self-rationalization improve robust- ness to spurious correlations? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7403–7416, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Arkadiy Saakyan, Tuh...

    2022

  7. [2023]

    In Findings of the Association for Computational Linguistics: ACL 2023, pages 4825–4838, Toronto, Canada

    Few shot rationale generation using self- training with dual teachers. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4825–4838, Toronto, Canada. Asso- ciation for Computational Linguistics. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Mad...

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.