Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A distillation recipe that trains small language models to answer from correct rationales and correct erroneous ones raises average accuracy by 3.0 points over the strongest baseline on reasoning benchmarks.

desk verdict A useful and honestly reported distillation recipe whose stated correctness-perception mechanism is contradicted by the paper's own Appendix D—worth refereeing as a conditional. read the letter →

arxiv 2509.05602 v2 pith:BQMHEORR submitted 2025-09-06 cs.CL

Hongyan Xie , Yitong Yao , Yikun Ban , Zixuan Huang , Deqing Wang , Zhenhe Wu , Haoxiang Su , Chao Wang
show 1 more author
Shuangyong Song
This is my paper · ORCID
classification cs.CL
keywords chain-of-thoughtdistillationknowledgesmalllanguagemodelsspuriouscorrelationsrationalecorrectnessfaithfulnessout-of-distributiongeneralizationreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Small language models trained on teacher-generated chain-of-thought data can latch onto shortcuts between the question and the final answer, ignoring whether the reasoning actually supports the answer. This paper proposes CoPeD, a distillation recipe that gives the student two tasks: answer from a rationale marked correct, and rewrite a rationale marked wrong. A correctness-aware weighted loss then down-weights training examples whose rationale and answer are poorly aligned. On BIG-Bench Hard and out-of-distribution reasoning sets, the paper reports that a LLaMA2-7B student reaches 52.9% average accuracy, 3.0 points above the strongest baseline, with improved faithfulness and soundness of generated rationales. The practical stakes are that small models could inherit reasoning ability without inheriting the teacher's unreliable rationales.

What carries the argument

The machinery has two coupled parts. The first is a correctness-aware task setting: status strings 'the rationale is right' and 'the rationale is wrong' route the student either to answer prediction conditioned on the rationale or to rationale correction, thereby cutting the direct question–answer shortcut and adding a learn-from-mistakes objective. The second is the Correctness-Aware Weighted loss, whose per-sample confidence weight $w_i = \operatorname{softmax}_i\left(-\frac{L_r^{(i)} + L_a^{(i)} + |L_r^{(i)} - L_a^{(i)}|}{\tau}\right)$ is used to up-weight examples where the rationale and answer losses are both low and aligned, and to down-weight noisy or contradictory ones. The status-token gating is the nominal carrier of the spurious-correlation argument; the weighting term is what makes the supervision robust to the heuristic correctness labels.

What would settle it

Permuting the status tokens during training while holding everything else fixed would settle the claim: if accuracy stays near 52.9%, the correctness signal is not what drives the gain. A second comparison that removes the correction task while keeping the weighted loss would show whether the revision objective is the active ingredient.

Watch

Extended reading notes

Core claim

The central claim is that reasoning distillation fails when the student learns to predict answers from rationales without checking whether the rationale is right, so CoPeD inserts that check as a training signal. For each question the teacher supplies both a correct and an erroneous rationale, labeled by whether the teacher's predicted answer matches the ground truth. The student is trained, under separate status tokens, either to produce the answer from the correct rationale or to rewrite the erroneous rationale into the correct one. A second component, the Correctness-Aware Weighted loss, assigns each sample a softmax weight from the combined rationale and answer losses plus their discrepancy, so well-aligned examples dominate training. The paper reports that this dual-task, reweighted recipe outperforms prior distillation baselines on in-domain BBH and on out-of-distribution sets, raises faithfulness and soundness scores, and improves accuracy even at 1.1B scale and with 12.5% of the training data.

Load-bearing premise

The load-bearing premise is that the student can learn to distinguish correct from erroneous rationales and act on that distinction at inference time, but the paper's own Section 6 and Appendix D show that the status string has no significant effect whether used as input or output, so this premise is not met in the reported experiments.

Editorial extensions

If this is right

  • On LLaMA2-7B, CoPeD-TL averages 52.9% across BBH-test, BB-sub, AGIEval, ARC-E, and ARC-C, beating CasCoD by 3.0 points overall and by 9.6 points on the in-domain BBH-test.
  • Rationale faithfulness and soundness both improve: GPT-4-averaged faithfulness goes from 77.4% with CasCoD to 81.2%, and soundness from 64.9% to 70.2%.
  • Using only 12.5% of the BBH-train data, CoPeD-TL beats CasCoD by 16.8% on BBH-test and by 9.4% on BB-sub, indicating strong data efficiency.
  • At 1.1B scale, the distilled model reaches 113.1% of the teacher's in-domain accuracy, suggesting the recipe is most valuable in low-resource settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper's own Section 6 and Appendix D show that the student cannot reliably verify its rationale, the reported gains are unlikely to come from the claimed answer-from-correct-revise-wrong gating; they more likely come from the correction task and the sample reweighting.
  • A direct test would permute the status tokens during training: if accuracy holds near 52.9%, then correctness perception is not the active ingredient and CoPeD reduces to weighted multi-task distillation.
  • The gating mechanism might become operational in a larger student or one paired with an external verifier; the same recipe at 13B or with a verification model could show a larger effect than the 7B experiments do.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes CoPeD, a distillation method for small language models that trains the student on two tasks: answer prediction conditioned on a teacher-provided correct rationale, and correction of teacher-provided erroneous rationales. A second component, the Correctness-Aware Weighted Loss, reweights training examples using the student's own rationale and answer losses, with a temperature parameter. Experiments on BBH as the in-distribution set and BB-sub, AGIEval, and ARC as out-of-distribution sets compare CoPeD with several CoT distillation baselines on LLaMA2-7B, Mistral-7B, TinyLLaMA, and LLaMA2-13B. The paper reports that LLaMA2-7B with CoPeD-TL reaches 52.9% average accuracy, outperforming CasCoD by 3.0%, and that CoPeD improves faithfulness and soundness as judged by ChatGPT and GPT-4.

Significance. If the results are reproducible, CoPeD is a useful distillation recipe: the gains on LLaMA2-7B are consistent across IND and most OOD benchmarks, the model-size and data-size ablations are informative, and the use of two LLM evaluators for faithfulness and soundness is a strength. The paper also promises code and data release. At the same time, the paper's central mechanistic claim is not supported by its own evidence: the student model is shown to be almost incapable of verifying rationale correctness, so the claimed inference-time routing between 'answer from correct rationale' and 'revise erroneous rationale' cannot be operational. The practical value therefore rests on the auxiliary correction task and sample reweighting, not on the correctness-perception mechanism advertised in the title and abstract.

major comments (4)
  1. [§2.2, §3.3, Appendix D (Figure 7), Limitations] The inference-time mechanism that motivates the paper is not operational, as the authors' own analysis shows. Section 2.2 describes routing the student to answer from a correct rationale or to revise an erroneous rationale, but at inference the model is not given a correctness label. Appendix D (Figure 7) reports 'no significant performance difference' between using the rationale status string as input and as output, and the Limitations section states that the student model 'currently struggles to effectively validate whether the rationale derived from its reasoning is indeed correct.' Since the student cannot verify correctness, the claimed 'predict answers based on correct rationales and revise them when they are incorrect' mechanism cannot be executed at inference. The reported gains may come instead from the rationale-correction loss and the sample weighting, not from correctness perception. This is a load-bearing issue for the central claim and requires either an inference-time verification component or a substantial reframing of the contribution.
  2. [Appendix A (Tables 4 and 5) and Table 1] The in-distribution result is inflated by hyperparameter selection on the test set. Table 4 in Appendix A tunes the temperature τ on BBH-test, and Table 5 selects α by average performance that includes BBH-test; the same BBH-test is then reported as the IND result in Table 1, including the headline 69.8% versus CasCoD's 60.2%. Because both hyperparameters are chosen to maximize the reported test metric, the IND comparison is not a fair evaluation and the 9.6% IND improvement is not a valid out-of-sample claim. The authors should use a held-out validation split for selection or report the selection procedure and its effect on the comparison.
  3. [§2.3, Eq. (4), Algorithm 1] The correctness-aware weighting is self-referential and does not measure rationale-answer alignment. In Eq. (4), the weight w_i is computed from the student's own rationale loss and answer loss, and Algorithm 1 shows that these are cross-entropy losses on the student's generated outputs. A sample with low student loss therefore receives high weight, so the weight represents the student's current ease with the sample, not the degree to which the rationale supports the answer, and it is not an oracle correctness label. The claim that this weighting 'directs the model to focus more on samples where the rationale offers stronger support for the correct answer' is not supported by the definition of w_i. The authors should either provide evidence that low-loss samples correspond to genuinely faithful rationales or reframe the weighting as a self-paced curriculum rather than correctness awareness.
  4. [§3.3, Table 1] The cross-model generalization claim is weak without variance estimates. For Mistral-7B-v0.2, CoPeD-TL averages 61.2% versus CasCoD's 61.6%, and on ARC-C it is 68.6% versus 75.5%, a 6.9-point deficit. No error bars, multiple seeds, or significance tests are reported for any model. The paper's phrasing that Mistral 'delivers consistently competitive performance' is not enough to support a general claim of effectiveness, and the difference between LLaMA2-7B and Mistral-7B may reflect noise rather than method behavior. The authors should add seeded runs with standard deviations or otherwise establish that the main comparisons are stable.
minor comments (6)
  1. [§2.1] The text says 'we following Dai et al. (2024a) and use its provided prompt templates to guild the teacher'; 'guild' should be 'guide', and the sentence should be rewritten for grammatical correctness.
  2. [Figures 3, 6, 8 and text] The method name is written inconsistently as 'CoCPerD' in Figures 3, 6, and 8 and in the Appendix D text, while the rest of the paper uses 'CoPeD'. Please unify the terminology.
  3. [§3.2] The implementation section says 'We use LLaMA-7B (Touvron et al., 2023) as the base student model', but the tables and the rest of the paper refer to 'LLaMA2-7B'. Also, Table 1 header says 'LLaMA2-7B' while the student row in the table maybe refers to a different version; please confirm the exact model used.
  4. [Eq. (4)] The notation in Eq. (4) is underspecified: the softmax is taken over i, but no domain (batch, dataset) is stated, and the sample-wise losses L_r and L_a are not defined formally as functions of the model outputs and targets. Please clarify.
  5. [Table 1] The table caption contains a footnote marker '♠' and the text says 'the results borrowed from Dai et al. (2024b)', but no '♠' appears in the table body. Please either add the marker to the appropriate rows or remove the footnote.
  6. [References] Some references are incomplete: 'Hu et al.' for LoRA has no year, 'Guo et al.' for BIG-Bench has no full title, and 'Li et al.' for Chain-of-Knowledge has no year. Please provide complete reference entries.

Circularity Check

2 steps flagged · score 6.0 of 10

BBH-test result is selected by hyperparameter tuning on BBH-test, and Eq. 4-5 define 'reliability' from the student's own losses, making the headline IND gain partly in-sample and the weighting self-referential.

  1. fitted input called prediction [Appendix A.1 (Table 4), Appendix A.2 (Table 5), Section 3.3 (Table 1)]
    "we evaluate CoPeD-TL on the BBH-test set under various τ values. The results show that performance increases as τ grows from 0.5 to 2.5, peaking at 69.8%. ... Therefore, we adopt α=0.5 as the default setting in our final model. ... It outperforms the strongest baseline, CasCoD, by 3.0%. In particular, it surpasses CasCoD by 9.6% in the IND scenario."

    The hyperparameters τ and α are selected by maximizing accuracy on BBH-test (Table 4, Table 5), and the same BBH-test accuracy of 69.8% is then reported in Table 1 and Section 3.3 as the main in-distribution result. The headline 'surpasses CasCoD by 9.6% in the IND scenario' is therefore an in-sample tuning artifact: the reported IND number is the maximum of the searched configuration rather than an independent evaluation of a fixed method. This is a fitted parameter renamed as a prediction of IND performance.

  2. self definitional [Section 2.3, Eqs. (4)-(5)]
    "Based on these losses, we calculate a normalized confidence weight wi, reflecting the reliability of the sample. Specifically, wi is computed as follows: wi =softmax i(−L(i) r +L (i) a +|L (i) r −L (i) a | τ), ... The Correctness-Aware Weighted loss is then defined as a weighted summation over the sample-wise rationale and answer losses: Lra = Σ_i wi·(L(i) r +L(i) a)."

    Eq. (5) is a weighted sum of L_r and L_a, while Eq. (4) defines the weights as a softmax over these same L_r and L_a values plus their difference. The paper interprets wi as 'reliability' and low-loss samples as 'faithful and consistent rationale–answer pairs,' but no external correctness signal enters the definition: the student's current loss defines the quality of the very samples being optimized. By construction, the model up-weights samples it already fits well, so the claimed 'focus on high-quality, well-aligned rationales' reduces to self-paced easy-sample selection rather than independent evidence of rationale correctness.

full rationale

The clearest circular element is the hyperparameter selection on BBH-test in Appendix A, followed by reporting that same BBH-test number as the main IND result in Table 1 and Section 3.3. The 69.8% used for the headline 9.6% IND gain is the maximum of the Table 4 sweep over τ, and α=0.5 is chosen from Table 5, so the IND comparison is partly an in-sample optimization result. A second self-referential element appears in Eqs. (4)-(5): the 'correctness-aware' weight is a function of the student's own rationale and answer losses, and the same losses are then re-weighted, making the mechanism definitionally equivalent to self-paced easy-sample weighting. I did not count the Appendix D/Figure 7 admission that 'the student model is almost incapable of effectively verifying the correctness of the generated rationale' as a circularity; that is a validity and mechanism failure of the claimed inference-time routing, not a logical equivalence between inputs and outputs. The paper retains independent content in the OOD benchmarks, model-size ablations, and data-efficiency experiments, so the score is partial rather than total: one reported benchmark is fitted, and one component of the loss is self-referential, while the overall derivation is not wholly equivalent to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the heuristic correctness labels, the assumption that low student loss identifies trustworthy samples, and test-set-based hyperparameter selection. No new entities are introduced.

free parameters (4)
  • α (task balance) = 0.5
    Tuned on BBH-test in Appendix A.2 (Table 5); balances answer prediction and rationale correction losses.
  • τ (temperature) = 5.0 for 7B/13B/Mistral, 2.5 for TinyLLaMA
    Tuned on BBH-test in Appendix A.1 (Table 4); controls softmax sharpness of sample weights.
  • n (starting epoch for weighted loss) = 5 for 7B/13B/Mistral, 10 for TinyLLaMA
    Chosen per model size; determines when uniform weighting switches to correctness-aware weighting (Algorithm 1).
  • learning rate per model = 4e-4 (1.1B), 3e-4 (7B/Mistral), 1e-4 (13B)
    Selected per model size in Section B.1; the authors state optimal value varies with model size.
assumptions (4)
  • domain assumption Rationale correctness is determined solely by whether the teacher's predicted answer matches the ground truth
    Section 2.1: if the LLM's predicted answer matches ground truth, the rationale is considered likely correct; otherwise erroneous. The paper acknowledges this heuristic may be noisy.
  • domain assumption Low student loss indicates a high-quality rationale-answer pair
    Section 2.3: the confidence weight uses the student's own rationale and answer losses to identify trustworthy examples; no independent validation of this assumption.
  • domain assumption LLM evaluators (ChatGPT, GPT-4) can reliably assess whether a rationale supports an answer
    Section 3.4: faithfulness and soundness are scored by ChatGPT and GPT-4 binary judgments with no human validation.
  • ad hoc to paper BBH-test accuracy is a valid criterion for selecting hyperparameters
    Appendix A tunes α and τ to maximize BBH-test accuracy, then reports the same BBH-test accuracy as the main IND result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation." pith.science (2026). https://pith.science/paper/BQMHEORR

@misc{pith2026250905602,
  author       = {Pith},
  title        = {Pith review of: Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQMHEORR}},
  note         = {Machine review of arXiv:2509.05602}
}
read the original abstract

Large language models (LLMs) excel at reasoning tasks but are expensive to deploy. Thus small language models (SLMs) are fine-tuned on CoT data generated by LLMs to copy LLMs' abilities. However, these CoT data may include noisy rationales that either fail to substantiate the answers or contribute no additional information to support answer prediction, which leads SLMs to capture spurious correlations between questions and answers and compromise the quality of reasoning. In this work, we propose Chain-of-Thought Correctness Perception Distillation (CoPeD), which aims to improve the reasoning quality of the student model from the perspectives of task setting and data utilization. Firstly, we introduce a correctness-aware task setting that encourages the student model to predict answers based on correct rationales and revise them when they are incorrect. This setting improves the faithfulness of reasoning and allows the model to learn from its mistakes. Then, we propose a Correctness-Aware Weighted loss, which dynamically adjusts the contribution of each training instance based on the combined loss of the rationale and the answer. This strategy encourages the model to focus more on samples where the rationale offers stronger support for the correct answer. Experiments have shown that CoPeD is effective on both in-distribution (IND) and out-of-distribution (OOD) benchmark reasoning datasets.

Figures

Figures reproduced from arXiv: 2509.05602 by the authors.

Figure 1
Figure 1. During training, the student model may cap [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of Chain-of-Thought Correctness Perception Distillation(CoPeD). We use teacher and student models to generate correct and erroneous rationales for the entire training set. Then, we adopt a multi-task learning framework to leverage these rationales, where one task is trained to predict the answer based on correct rationales, and the other task is trained to correct erroneous rationales as additional supervis… view at source ↗
Figure 3
Figure 3. CoPeD adopts different strategies based on [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Ablation study on model size for IND and four OOD datasets. The dotted line indicates the performance [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Ablation study on training data size for IND and four OOD datasets. The dotted line indicates the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Compare training CoPeD with different target [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Comparison between using the rationale status [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Answer-conditioned chains of thought are harmful supervision for reasoning distillation—causing up to ~27-point accuracy drops—and the damage is invisible to correctness filters.

Reference graph

Works this paper leans on

48 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Shengnan An, Zexiong Ma, Siqi Cai, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2024. Can llms learn from mistakes? an empirical study on reasoning tasks. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 833--854

    work page 2024

  2. [2]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2023. Learning from mistakes makes llm better reasoner. arXiv preprint arXiv:2310.20689

    arXiv 2023

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  4. [4]

    Hyungjoo Chae, Yongho Song, Kai Ong, Taeyoon Kwon, Minjin Kim, Youngjae Yu, Dongha Lee, Dongyeop Kang, and Jinyoung Yeo. 2023. Dialogue chain-of-thought distillation for commonsense-aware conversational agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5606--5632

    work page 2023

  5. [5]

    Hongzhan Chen, Siyue Wu, Xiaojun Quan, Rui Wang, Ming Yan, and Ji Zhang. 2023 a . Mcc-kd: Multi-cot consistent knowledge distillation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6805--6820

    work page 2023

  6. [6]

    Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. 2023 b . Disco: Distilling counterfactuals with large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5514--5528

    work page 2023

  7. [7]

    Li Chenglin, Qianglong Chen, Liangyue Li, Caiyu Wang, Feng Tao, Yicheng Li, Zulong Chen, and Yin Zhang. 2024. Mixed distillation helps smaller language models reason better. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1673--1690

    2024

  8. [8]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

Show all 48 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://api.semanticscholar.org/CorpusID:3922816 Think you have solved question answering? try arc, the ai2 reasoning challenge . ArXiv, abs/1803.05457

    2018 arXiv

  2. [10]

    Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. 2024 a . Beyond imitation: Learning key reasoning steps from dual chain-of-thoughts in reasoning distillation. arXiv preprint arXiv:2405.19737

    2024 arXiv

  3. [11]

    Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. 2024 b . Improve student’s reasoning generalizability through cascading decomposed cots distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15623--15643

    2024

  4. [12]

    Tao Feng, Yicheng Li, Li Chenglin, Hao Chen, Fei Yu, and Yin Zhang. 2024. Teaching small language models reasoning through counterfactual distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5831--5842

    2024

  5. [13]

    Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In Proceedings of the 40th International Conference on Machine Learning, pages 10421--10430

    2023

  6. [14]

    Beyond imitation: Leveraging fine-grained quality signals for alignment

    Geyang Guo, Ranchi Zhao, Tianyi Tang, Xin Zhao, and Ji-Rong Wen. Beyond imitation: Leveraging fine-grained quality signals for alignment. In The Twelfth International Conference on Learning Representations

  7. [15]

    Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852--14882

    2023

  8. [16]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. In Proceedings of the 36th International Conf...

    2022

  9. [17]

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the As...

    2023

  10. [18]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

  11. [19]

    Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051--1068

    2023

  12. [20]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  13. [21]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  14. [22]

    Hojae Lee, Junho Kim, and SangKeun Lee. 2024. Mentor-kd: Making small language models better multi-step reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17643--17658

    2024

  15. [23]

    Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023. Symbolic chain-of-thought distillation: Small models can also “think” step-by-step. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2023

  16. [24]

    Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. 2022. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726

    2022 arXiv

  17. [25]

    Xiang Li, Shizhu He, Jiayu Wu, Zhao Yang, Yao Xu, Yang jun Jun, Haifeng Liu, Kang Liu, and Jun Zhao. 2024. Mode-cotd: Chain-of-thought distillation for complex reasoning tasks with mixture of decoupled lora-experts. In Proceedings of the 2024 Joint International Conference on ...

    2024

  18. [26]

    Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources

    Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In The Twelfth International Conference on Learning Representations

  19. [27]

    Weize Liu, Guocong Li, Kai Zhang, Bang Du, Qiyuan Chen, Xuming Hu, Hongxia Xu, Jintai Chen, and Jian Wu. 2024. Mind’s mirror: Distilling self-evaluation capability and comprehensive thinking from large language models. In Proceedings of the 2024 Conference of the North America...

    2024

  20. [28]

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773--1781

    2023

  21. [29]

    R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5)

    2023 arXiv

  22. [30]

    Jiawei Shao and Xuelong Li. 2025. https://doi.org/10.1109/MNET.2025.3541208 Ai flow at the network edge . IEEE Network, pages 1--1

    2025

  23. [31]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

    2024

  24. [32]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computati...

    2023

  25. [33]

    Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. 2024. Can llms learn from previous mistakes? investigating llms' errors to boost for reasoning. arXiv preprint arXiv:2403.20046

    2024 arXiv

  26. [34]

    Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M

    Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cant \'o n Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes,...

    2023 arXiv

  27. [35]

    Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2024. Investigating mysteries of cot-augmented distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6071--6086

    2024

  28. [36]

    Danqing Wang and Lei Li. 2023. Learning from mistakes via cooperative study assistant for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10667--10685

    2023

  29. [37]

    Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023 a . Scott: Self-consistent chain-of-thought distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5546--5558

    2023

  30. [38]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations

  31. [39]

    Zhaoyang Wang, Shaohan Huang, Yuxuan Liu, Jiahai Wang, Minghui Song, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, et al. 2023 b . Democratizing reasoning ability: Tailored learning from large language model. In Proceedings of the 2023 Conference on Empirical Me...

    2023

  32. [40]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  33. [41]

    Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North American...

    2022

  34. [42]

    Bohao Yang, Chen Tang, Kun Zhao, Chenghao Xiao, and Chenghua Lin. 2024. Effective distillation of table-based reasoning ability from llms. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 202...

    2024

  35. [43]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385

    2024 arXiv

  36. [44]

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. Agieval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299--2314

    2024

  37. [45]

    Yuhang Zhou and Wei Ai. 2024. Teaching-assistant-in-the-loop: Improving knowledge distillation from imperfect teacher models in low-budget scenarios. arXiv preprint arXiv:2406.05322

    2024 arXiv

  38. [46]

    Xuekai Zhu, Biqing Qi, Kaiyan Zhang, Xinwei Long, Zhouhan Lin, and Bowen Zhou. 2024. Pad: Program-aided distillation can teach small models reasoning better than chain-of-thought fine-tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Associatio...

    2024

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.