Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rationale-augmented finetuning is not universally beneficial: across 18 tasks it improved accuracy only on the hardest tasks, lowered it on 11, and usually improved calibration, with the two effects linked by a linear difficulty-driven…

desk verdict Useful empirical study showing rationales can hurt accuracy while improving calibration, but the paper's difficulty-driven linear law is not established and Eq. 10 is internally inconsistent as printed. read the letter →

arxiv 2505.24147 v1 pith:HLWCKC7L submitted 2025-05-30 cs.CL

Chiwei Zhu , Benfeng Xu , An Yang , Junyang Lin , Quan Wang , Chang Zhou , Zhendong Mao This is my paper
classification cs.CL
keywords rationale-augmentedfinetuningchain-of-thoughtexpectedcalibrationerrortaskdifficultymodelreliabilitylanguageself-consistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rationale-augmented finetuning (RAFT) is widely treated as a default way to make smaller language models reason better, but this paper argues the benefit is conditional. Across 18 tasks, training with GPT-3.5-generated rationales improved accuracy on only the hardest tasks — chiefly math and ARC — and actually lowered accuracy on 11 of 18 datasets. From the complementary angle of reliability, rationales usually improved calibration, lowering expected calibration error (ECE) in 12 of 18 tasks and even beating the untrained base model in 3. The paper reports a tight linear relation between the accuracy gain and the calibration gain, $\Delta\text{ECE} \approx 0.75\,\Delta\text{Acc} + 0.05$ with Pearson $r \approx 0.97$, and attributes both gains to the intrinsic difficulty of the task. If correct, this turns rationale use from an all-or-nothing choice into a predictable data-curation decision.

What carries the argument

The central objects are the two difference metrics that define the paper's measurements: $\Delta\text{Acc} = \text{Acc}_{\text{RAFT}} - \text{Acc}_{\text{FT}}$ and $\Delta\text{ECE} = -(\text{ECE}_{\text{RAFT}} - \text{ECE}_{\text{FT}})$, where ECE is the binned expected calibration error. Their linear relationship, $\Delta\text{ECE} = 0.7479\,\Delta\text{Acc} + 0.0456$, is the empirical law that carries the argument from performance to reliability. The explanatory engine is task difficulty, operationalized through three metrics — GPT-3.5 rationale length, LLaMA-2-base rationale length, and label-only finetuning accuracy — with the latter two producing the predictive fits. A blank-rationale ablation, replacing real rationales with equal-length placeholder tokens, isolates the cause by showing that content, not length or format, drives both the harm and the calibration repair.

What would settle it

Run the same RAFT-versus-label-only comparison on new tasks that match existing label-only accuracies but carry deliberately corrupted rationales (e.g., off-topic reasoning or shuffled steps); if the difficulty-only account is right, the new points should fall on the fitted $\Delta\text{ECE}$–$\Delta\text{Acc}$ line, whereas corrupted rationales displacing points off the line would show that rationale content is an independent cause.

Watch

Extended reading notes

Core claim

The central discovery is that the value of rationale-augmented finetuning is task-dependent and predictable, not universal. In a controlled comparison on LLaMA-2-7B, insertion of rationales between input and answer improved accuracy on math reasoning and both ARC variants but degraded accuracy on 11 of the 18 tasks; the harmful cases are mostly easy classification, paraphrase, NLI, and coreference tasks. In parallel, RAFT usually improved calibration — lower ECE in 12 of 18 tasks, with 3 tasks falling below the untrained base model — and the two effects are linearly coupled: $\Delta\text{ECE} = 0.7479\,\Delta\text{Acc} + 0.0456$ ($r = 0.9681$, $p = 2.462\times10^{-10}$). The paper attributes the coupling to task difficulty, supports it with three difficulty metrics, and provides linear fits that let a user predict whether (and how much) RAFT will help from the label-only finetuning accuracy or from the length of rationales produced by an unaligned base model.

Load-bearing premise

The explanation stands or falls on the premise that intrinsic task difficulty is the common driver of both accuracy and calibration changes; the strongest difficulty measure is label-only finetuning accuracy, the same quantity used to define the accuracy gain, so the correlation may be partly mechanical.

Editorial extensions

If this is right

  • Rationale-augmented finetuning should not be applied as a default: on easy or medium tasks it can lower accuracy, so practitioners should first estimate task difficulty and add rationales only where the model struggles.
  • Calibration is a genuine benefit of rationales: because finetuning normally makes models overconfident, inserting rationales can bring ECE below both the label-only finetuned model and, in some tasks, the untrained base model.
  • The linear $\Delta\text{ECE}$–$\Delta\text{Acc}$ relation means accuracy and reliability move together: a task that gains from rationales will almost always also become better calibrated, and a task that loses accuracy tends to become worse calibrated.
  • Task difficulty measured by label-only finetuning accuracy or by LLaMA-2-generated rationale length can predict whether RAFT helps, giving a data-curation rule: allocate detailed rationales to hard examples and trim them from easy ones.
  • The same qualitative pattern appears under rationale-augmented prompting and across other base models, so the conclusion is not an artifact of one finetuning recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the difficulty law suggests a concrete allocation policy — add rationales only to examples whose label-only accuracy is below a threshold, and trim them from easy examples — which can be tested by training on such a mixture and comparing accuracy and ECE against both the all-rationale and no-rationale extremes.
  • Editorial inference: since the GPT-3.5 rationale-length metric fails to track the trend while LLaMA-2-base lengths work, difficulty probes should be computed with an unaligned base model; an aligned generator's rationale length is too entangled with its own training distribution to be a portable difficulty measure.
  • Editorial inference: the linear $\Delta\text{ECE}$–$\Delta\text{Acc}$ coupling may be a general calibration phenomenon rather than a rationale-specific one — any intervention that selectively improves learnability on hard tasks might produce the same line — which could be tested with interventions other than rationales, such as extra data on hard slices or test-time compute.
  • Editorial inference: the blank-rationale ablation implies the content of rationales, not their length or the training format, causes both harm and calibration repair; a direct extension is to filter or rewrite rationales by error type (off-topic, trivial, unreasonable) and check whether the harmful tasks move toward the helpful side of the line.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper empirically studies Rationale-Augmented Fine-Tuning (RAFT) across 18 tasks using LLaMA-2-7B, measuring both accuracy and calibration (ECE). It reports three findings: (1) rationales can hurt performance (11/18 tasks), (2) rationales often improve calibration (12/18 tasks), and (3) the accuracy improvement and ECE improvement are linearly related (Section 3.3, Eq. 7), with both attributed to intrinsic task difficulty (Section 4). The authors propose difficulty metrics based on rationale length and label-only fine-tuning accuracy, and provide an actionable regression for predicting RAFT gains (Eqs. 9-10). Robustness checks include different models, hyper-parameters, prompts, multi-task training, and out-of-sample validation on SUBJ and CoinFlip.

Significance. If the findings hold, the paper provides a valuable correction to the prevailing assumption that rationales are uniformly beneficial in fine-tuning, and it introduces calibration as a complementary evaluation axis for RAFT. The empirical coverage is broad (18 tasks, 7 categories), and the paper includes useful ablations across models, prompt annotators, and multi-task settings. The out-of-sample validation on two additional datasets and the public code release are concrete strengths. However, the central 'difficulty-driven linear law' rests on statistical fits that currently have unresolved correctness and robustness issues, so the paper's main quantitative claim is not yet established at the level the text presents.

major comments (4)
  1. [§4.3, Eq. (10)] Eq. (10) is arithmetically implausible as printed: with Acc_FT in [0,1], the regression ΔAcc = 0.8031·Acc_FT + 0.6730 predicts ΔAcc between 0.673 and 1.476, which for Acc_FT > 0.18 implies Acc_RAFT = Acc_FT + ΔAcc > 1. Since accuracy cannot exceed 1, either the sign of the slope is wrong (e.g., it should be negative) or the regressor is a reversed measure such as 1 − Acc_FT. This equation is load-bearing for the paper's actionable-insight claim and must be corrected; the corrected version should be re-validated on the two held-out datasets.
  2. [§3.3, Fig. 2 and Table 5] The headline linear correspondence between ΔAcc and ΔECE is quantified from 18 points, but no error bars or seed variance are reported for the individual ΔAcc/ΔECE values, and the fit may be dominated by the four math tasks with ΔAcc > 0.4. Please provide a jackknife/leave-one-out analysis or explicitly report the fit after excluding the math cluster. Without such analysis, the universal 'linear correspondence' claim is not robustly supported.
  3. [§4.2, Table 10] The difficulty-driven explanation is weakened by the failure of the GPT-3.5 rationale-length metric: it shows no significant correlation with ΔAcc (Pearson 0.0010, p=0.4984) in the RAFT setting, and the paper attributes this to unspecified GPT-3.5 bias. Since this is the only difficulty metric not derived from Acc_FT, the reader needs a concrete, testable justification for why the LLaMA2-length metric is reliable while the GPT-3.5 metric is not, rather than a post-hoc explanation.
  4. [§2.2, §3.2] ECE is computed from the 10-level self-consistency confidence (n/10), which is a very coarse confidence measure and may understate calibration error. The paper should either report a sensitivity check using softmax-based confidence from the model's final answer distribution, or justify why the coarse 10-level confidence is adequate for the conclusions about calibration improvement.
minor comments (4)
  1. [Appendix J, Table 9 title] The table title contains a typo: 'Improvment' should be 'Improvement'.
  2. [Appendix D.1] Prompt templates contain typos ('Follwing', 'is is', 'determine whether is is true') that should be fixed for clarity.
  3. [Fig. 3 caption] The note that 'y-axes are folded for better display' is unclear; please explain the folding or use standard axis scaling.
  4. [§3.3] Please clarify that ΔECE as defined in Eq. (6) is the negative ECE change, so positive values denote improvement; the current text is clear but the sign convention could be stated earlier.

Circularity Check

1 steps flagged · score 4.0 of 10

Eq. 10 regresses ΔAcc on Acc_FT, which is already a component of ΔAcc by Eq. 5; the 'difficulty-driven' quantitative law is partly built in, though independent metrics and out-of-sample checks keep the core findings non-circular.

  1. self definitional [Section 3 (Eq. 5) and Section 4.3 (Eq. 10); Table 10]
    "∆Acc = AccRAF T− AccF T, (5) ... ∆Acc = 0.8031 × AccF T+ 0.6730 (10)"

    Eq. 10 is presented as evidence that the accuracy gain from RAFT is driven by 'intrinsic task difficulty,' with difficulty Metric 3 defined as label-only finetuning accuracy Acc_FT. But Eq. 5 defines ΔAcc as Acc_RAFT − Acc_FT, so the regressor Acc_FT appears inside the regressand by construction. Even if RAFT added only a constant benefit, ΔAcc would be mechanically related to Acc_FT through the −Acc_FT term and through ceiling/floor effects; the regression therefore cannot by itself establish a difficulty-driven causal law. The issue is compounded because, as printed, Eq. 10 implies Acc_RAFT > 1 for any Acc_FT > 0.18, so the fitted equation is not an independently evaluable quantitative prediction.

full rationale

The paper's headline observational findings—RAFT harms performance on 11/18 tasks, improves calibration on many tasks, and the cross-task ΔECE–ΔAcc fit (Eq. 7) is strongly linear—are empirical measurements, not derivatives of the assumptions. I found no load-bearing self-citation: the authors' 2023 calibration paper is cited only as background alongside independent work, and no uniqueness theorem or ansatz is imported from the authors' prior papers. The one genuinely circular element is the use of label-only finetuning accuracy Acc_FT as difficulty Metric 3 and then regressing ΔAcc on it in Eq. 10. Since Eq. 5 defines ΔAcc = Acc_RAFT − Acc_FT, Acc_FT is a component of the regressand; the observed correlation is therefore partly mechanical and cannot by itself validate the claim that 'intrinsic difficulty' drives both accuracy and reliability gains. This is compounded by Eq. 10 being arithmetically inconsistent as printed, forcing Acc_RAFT > 1 for realistic Acc_FT values. However, the paper also provides an independent difficulty proxy (LLaMA-2-generated rationale length) that shows significant correlations with both ΔAcc and ΔECE, a non-mechanical ΔECE–Acc_FT correlation, and an out-of-sample check on SUBJ and CoinFlip. The central claim therefore retains independent content; the circularity is partial and localized to the Acc_FT-based difficulty regression, so a moderate score of 4 is appropriate.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper's contribution is empirical. Its fitted regression lines (Eqs. 7, 9, 10) are free parameters estimated from the same 18 datasets. Its causal difficulty story rests on domain assumptions about what counts as difficulty and about the validity of self-consistency-derived confidence. No invented entities are introduced.

free parameters (6)
  • linear fit slope α for ΔECE vs ΔAcc = 0.7479
    Equation 7, OLS fit over 18 datasets. Used to claim linear correspondence.
  • linear fit intercept β for ΔECE vs ΔAcc = 0.0456
    Equation 7, OLS fit over 18 datasets.
  • difficulty prediction slope for Eq 9 = 0.0029
    ΔAcc = 0.0029 × Len(R_LLaMA2) - 0.5567, fitted over 18 datasets.
  • difficulty prediction intercept for Eq 9 = -0.5567
    Equation 9, fitted over 18 datasets.
  • difficulty prediction slope for Eq 10 = 0.8031
    ΔAcc = 0.8031 × Acc_FT + 0.6730, fitted over 18 datasets.
  • difficulty prediction intercept for Eq 10 = 0.6730
    Equation 10, fitted over 18 datasets.
assumptions (4)
  • domain assumption The 18 selected tasks are representative of diverse NLP tasks and are treated as independent samples for regression.
    The paper draws universal conclusions from 18 datasets; the p-value assumes independence of task-level observations.
  • domain assumption Self-consistency voting frequency n/10 is a valid confidence estimate for ECE computation.
    Section 2.3: confidence is n/10 from 10 sampled reasoning paths; ECE is computed on this coarse frequency, not on model probabilities.
  • domain assumption Rationales generated by GPT-3.5 that lead to correct answers are of sufficient quality and neutrality for measuring RAFT effects.
    Section 2.3 and Appendix C. Filtering to correct-answer rationales may make rationales unrepresentative of natural reasoning, although it biases in favor of RAFT.
  • domain assumption Task difficulty can be measured by mean rationale length and by label-only finetuning accuracy.
    Section 4.1. One of three proposed metrics (GPT-3.5 rationale length) failed to show the expected correlation, weakening the general claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability." pith.science (2026). https://pith.science/paper/HLWCKC7L

@misc{pith2026250524147,
  author       = {Pith},
  title        = {Pith review of: Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HLWCKC7L}},
  note         = {Machine review of arXiv:2505.24147}
}
read the original abstract

Training language models with rationales augmentation has been shown to be beneficial in many existing works. In this paper, we identify that such a prevailing view does not hold consistently. We conduct comprehensive investigations to thoroughly inspect the impact of rationales on model performance as well as a novel perspective of model reliability. The results lead to several key findings that add new insights upon existing understandings: 1) Rationales can, at times, deteriorate model performance; 2) Rationales can, at times, improve model reliability, even outperforming their untrained counterparts; 3) A linear correspondence exists in between the performance and reliability improvements, while both are driven by the intrinsic difficulty of the task. These findings provide informative regulations on the broad utilization of rationales and raise critical implications on the procedure of explicitly aligning language models with implicit human thoughts. Codes can be found at https://github.com/Ignoramus0817/rationales.

Figures

Figures reproduced from arXiv: 2505.24147 by the authors.

Figure 1
Figure 1. Illustration of training LMs with rationale [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Improvement in Accuracy (x-axis) and Ex￾pected Calibration Error (ECE) (y-axis) under RAFT for different datasets. We empirically find that rationale’s impacts on performance and reliability are synchronized. We attribute this linear correlation to the intrinsic diffi￾culty of specific tasks. We further propose several difficulty metrics to validate this assumption and es￾tablish, for the first time, a quantitative … view at source ↗
Figure 3
Figure 3. Improvements in accuracy and ECE under RAFT. Datasets are re-ordered according to the improvements [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: ECE of finetuned and base models. ∆F T −Base means difference of finetuned and pretrained models. 3.1 Impacts on Model Performance As is seen in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Correspondence between model improvement and task difficulty across different metrics: (a), (b): Average [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Impacts of varied rationales. sue unbiased rationales that naturally arises from pre-trained LLMs, we design a Mix-of-Task In￾Context Prompting strategy. We first employ LLaMA2 as the generator, since it is not explic￾itly aligned to human instructions, we prepend 3 de…
Figure 7
Figure 7. Figure 7: Probability of generating the final answer given different lengths of rationale tokens. Tokens corresponds [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Improvement in Accuracy and ECE of different models. Significant linear correspondence can still be [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Improvement in Accuracy and ECE of models trained on rationales generated with LLaMA-3.1-70B [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: ECE of models trained with rationales produced by annotator B and C. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Experimental results of blank rationales. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Improvements in accuracy and ECE under rationale-augmented prompting setting. Datasets are re [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Improvement in Accuracy and ECE of gpt-3.5-turbo-0613 under rationale-augmented prompting setting. Each point is a datasets, and its x/y-coordinate represents the improvement in model accuracy/ECE respectively. Point labels are omitted as points are close to each othe…
Figure 14
Figure 14. Figure 14: Improvement in accuracy and ECE under rationale-augmented prompting setting when task difficulty is [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Improvement in Accuracy and ECE of models trained in multi-task settings. (a) Baseline: single task [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AI Reasoning for Wireless Communications and Networking: A Survey and Perspectives

    cs.NI 2025-09 conditional novelty 4.0 of 10

    A survey that organizes LLM and AI reasoning methods into a taxonomy and maps them onto the physical, link, network, transport, and application layers of wireless networks.

Reference graph

Works this paper leans on

63 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. https://arxiv.org/abs/1812.01193 e-snli: Natural language inference with natural language explanations . Preprint, arXiv:1812.01193

    arXiv 2018

  2. [2]

    Samuel Carton, Surya Kanoria, and Chenhao Tan. 2022. https://arxiv.org/abs/2112.00071 What to learn, and how: Toward effective learning from rationales . Preprint, arXiv:2112.00071

    arXiv 2022

  3. [3]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping H...

    arXiv 2022

  4. [4]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://api.semanticscholar.org/CorpusID:3922816 Think you have solved question answering? try arc, the ai2 reasoning challenge . ArXiv, abs/1803.05457

    arXiv 2018

  5. [5]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  6. [6]

    Morris H DeGroot and Stephen E Fienberg. 1983. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12--22

    work page 1983

  7. [7]

    Shrey Desai and Greg Durrett. 2020. https://arxiv.org/abs/2003.07892 Calibration of pre-trained transformers . Preprint, arXiv:2003.07892

    arXiv 2020

  8. [8]

    Xiaowen Ding, Bing Liu, and Philip S Yu. 2008. A holistic lexicon-based approach to opinion mining. In Proceedings of the 2008 international conference on web search and data mining, pages 231--240

    2008

Show all 63 references
  1. [9]

    DrewWham and Mauricio Nascimento. 2020. https://kaggle.com/competitions/coin-flips Coin flips . Kaggle

    2020

  2. [10]

    Gregory Druck, Burr Settles, and Andrew McCallum. 2009. https://aclanthology.org/D09-1009 Active learning by labeling features . In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 81--90, Singapore. Association for Computational Li...

    2009

  3. [11]

    Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2025. https://arxiv.org/abs/2110.08420 Understanding dataset difficulty with V -usable information . Preprint, arXiv:2110.08420

    2025 arXiv

  4. [12]

    Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. https://arxiv.org/abs/2301.12726 Specializing smaller language models towards multi-step reasoning . Preprint, arXiv:2301.12726

    2023 arXiv

  5. [13]

    Francis Galton. 1886. Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15:246--263

  6. [14]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. https://arxiv.org/abs/1706.04599 On calibration of modern neural networks . Preprint, arXiv:1706.04599

    2017 arXiv

  7. [15]

    Peter Hase and Mohit Bansal. 2022. https://doi.org/10.18653/v1/2022.lnls-1.4 When can models learn from explanations? a formal framework for understanding the roles of explanation data . In Proceedings of the First Workshop on Learning with Natural Language Supervision, pages ...

    2022 doi

  8. [16]

    Guande He, Jianfei Chen, and Jun Zhu. 2023. https://arxiv.org/abs/2305.19249 Preserving pre-trained features helps calibrate fine-tuned language models . Preprint, arXiv:2305.19249

    2023 arXiv

  9. [17]

    Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136

    2016 arXiv

  10. [18]

    Anubha Kabra, Sanketh Rangreji, Yash Mathur, Aman Madaan, Emmy Liu, and Graham Neubig. 2023. https://arxiv.org/abs/2311.09553 Program-aided reasoners (better) know what they know . Preprint, arXiv:2311.09553

    2023 arXiv

  11. [19]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 arXiv

  12. [20]

    Daniel Kahneman. 2011. Thinking, fast and slow. macmillan

    2011

  13. [21]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. https://arxiv.org/abs/2205.11916 Large language models are zero-shot reasoners . Preprint, arXiv:2205.11916

    2023 arXiv

  14. [22]

    K Kotovsky, J.R Hayes, and H.A Simon. 1985. https://doi.org/10.1016/0010-0285(85)90009-X Why are some problems hard? evidence from tower of hanoi . Cognitive Psychology, 17(2):248--294

    1985 doi

  15. [23]

    Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023. https://arxiv.org/abs/2306.14050 Symbolic chain-of-thought distillation: Small models can also "think" step-by-step . Preprint, arXiv:2306.14050

    2023 arXiv

  16. [24]

    Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. 2022. https://arxiv.org/abs/2210.06726 Explanations from large language models make small reasoners better . Preprint, arXiv:2210.06726

    2022 arXiv

  17. [25]

    Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics

    2002

  18. [26]

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. https://arxiv.org/abs/2212.08410 Teaching small language models to reason . Preprint, arXiv:2212.08410

    2023 arXiv

  19. [27]

    Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. https://doi.org/10.18653/v1/2020.acl-main.92 A diverse corpus for evaluating and developing E nglish math word problem solvers . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pa...

    2020 doi

  20. [28]

    Robert G Miller. 1962. Statistical prediction by discriminant analysis. In Statistical prediction by discriminant analysis, pages 1--54. Springer

    1962

  21. [29]

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. 2023. https://arxiv.org/abs/2311.110...

    2023 arXiv

  22. [30]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. https://arxiv.org/abs/2306.02707 Orca: Progressive learning from complex explanation traces of gpt-4 . Preprint, arXiv:2306.02707

    2023 arXiv

  23. [31]

    Allan H Murphy. 1973. A new vector partition of the probability score. Journal of Applied Meteorology and Climatology, 12(4):595--600

    1973

  24. [32]

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29

    2015

  25. [33]

    Khanh Nguyen and Brendan O'Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint arXiv:1508.05154

    2015 arXiv

  26. [34]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. https://arxiv.org/abs/1910.14599 Adversarial nli: A new benchmark for natural language understanding . Preprint, arXiv:1910.14599

    2020 arXiv

  27. [35]

    Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. 2019. Measuring calibration in deep learning. In CVPR workshops, volume 2

    2019

  28. [36]

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. https://arxiv.org/abs/2112.00114 Show your work: Scratchpads for intermediate c...

    2021 arXiv

  29. [37]

    Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, and Greg Durrett. 2021. https://arxiv.org/abs/2109.01653 Creak: A dataset for commonsense reasoning over entity knowledge . Preprint, arXiv:2109.01653

    2021 arXiv

  30. [38]

    OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Bel...

    2023 arXiv

  31. [39]

    TN Palmer, FJ Doblas-Reyes, Antje Weisheimer, and MJ Rodwell. 2008. Toward seamless prediction: Calibration of climate change projections using seasonal forecasts. Bulletin of the American Meteorological Society, 89(4):459--470

    2008

  32. [40]

    Bo Pang and Lillian Lee. 2004. https://doi.org/10.3115/1218955.1218990 A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts . In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics ( ACL -04) , p...

    2004

  33. [41]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. https://arxiv.org/abs/2103.07191 Are nlp models really able to solve simple math word problems? Preprint, arXiv:2103.07191

    2021 arXiv

  34. [42]

    Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. https://arxiv.org/abs/1808.09121 Wic: the word-in-context dataset for evaluating context-sensitive meaning representations . Preprint, arXiv:1808.09121

    2019 arXiv

  35. [43]

    Subhro Roy and Dan Roth. 2015. https://doi.org/10.18653/v1/D15-1202 Solving general arithmetic word problems . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743--1752, Lisbon, Portugal. Association for Computational Linguistics

    2015 doi

  36. [44]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. https://arxiv.org/abs/1907.10641 Winogrande: An adversarial winograd schema challenge at scale . Preprint, arXiv:1907.10641

    2019 arXiv

  37. [45]

    Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. https://arxiv.org/abs/2212.00193 Distilling reasoning capabilities into smaller language models . Preprint, arXiv:2212.00193

    2023 arXiv

  38. [46]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters . Preprint, arXiv:2408.03314

    2024 arXiv

  39. [47]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Emp...

    2013

  40. [48]

    Robyn Speer, Joshua Chin, and Catherine Havasi. 2018. https://arxiv.org/abs/1612.03975 Conceptnet 5.5: An open multilingual graph of general knowledge . Preprint, arXiv:1612.03975

    2018 arXiv

  41. [49]

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2024. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12183

    2024 arXiv

  42. [50]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://arxiv.org/abs/1811.00937 Commonsenseqa: A question answering challenge targeting commonsense knowledge . Preprint, arXiv:1811.00937

    2019 arXiv

  43. [51]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  44. [52]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. https://arxiv.org/abs/1905.00537 Superglue: A stickier benchmark for general-purpose language understanding systems . Preprint, arXiv:1905.00537

    2020 arXiv

  45. [53]

    Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2023 a . https://arxiv.org/abs/2211.01562 Pinto: Faithful language reasoning using prompt-generated rationales . Preprint, arXiv:2211.01562

    2023 arXiv

  46. [54]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 b . https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . Preprint, arXiv:2203.11171

    2023 arXiv

  47. [55]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  48. [56]

    Bingsheng Yao, Prithviraj Sen, Lucian Popa, James Hendler, and Dakuo Wang. 2023. https://arxiv.org/abs/2305.03117 Are human explanations always helpful? towards objective evaluation of human natural language explanations . Preprint, arXiv:2305.03117

    2023 arXiv

  49. [57]

    Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. https://aclanthology.org/N07-1033 Using `` annotator rationales '' to improve machine learning for text categorization . In Human Language Technologies 2007: The Conference of the North A merican Chapter of the Association...

    2007

  50. [58]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28

    2015

  51. [59]

    Ye Zhang, Iain Marshall, and Byron C. Wallace. 2016. https://doi.org/10.18653/v1/D16-1076 Rationale-augmented convolutional neural networks for text classification . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 795--804, Aust...

    2016 doi

  52. [60]

    Yuan Zhang, Jason Baldridge, and Luheng He. 2019. https://arxiv.org/abs/1904.01130 Paws: Paraphrase adversaries from word scrambling . Preprint, arXiv:1904.01130

    2019 arXiv

  53. [61]

    Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.654 On the calibration of large language models and alignment . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9778--9795...

    2023 doi

  54. [62]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  55. [63]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.