Pith. sign in

REVIEW 3 major objections 4 minor 19 references

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that self-distillation as a lone objective, with no reward term, does not improve task accuracy on difficult reasoning tasks, because a teacher conditioned on one privileged reference solution encodes that trajectory…

desk verdict Careful negative result for single-trajectory PI self-distillation, but the abstract's 'does not teach anything' overreaches the evidence. read the letter →

arxiv 2608.04794 v1 pith:CNVG4DOP submitted 2026-08-05 cs.AI cs.LG

Sarthak Harne , Chinmay Karkar , Yash Pandya , Ahmed Awadallah , Akshay Nambi This is my paper · ORCID
classification cs.AIcs.LG
keywords self-distillationprivilegedinformationPIbiasscoreper-tokendistillationlossreasoningdegradationlargelanguagemodelstrainingdynamicsreferencesolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether self-distillation (SD), in which a student is trained to match a teacher that is conditioned on privileged information such as a reference solution, teaches anything when it is the only training signal. The paper reproduces the gains reported for SD on an easy benchmark, then applies the same implementation to difficult question answering, mathematics, coding, and multi-turn agentic tool use, and finds that the per-token loss falls steadily while validation accuracy stays flat or degrades. The paper attributes this to a mechanism beginning with PI bias: a teacher that has seen one reference solution favors that trajectory over other correct solutions, so the dense per-token target is nearly blind to whether a rollout is correct. The conclusion is that as a lone objective, SD optimizes a signal decoupled from task success, and that a falling SD loss should not be read as evidence of learning.

What carries the argument

The load-bearing instrument is the PI Bias Score, $\text{PS}_t(w)=s^T_t(w)-s^S_t(w)$, the teacher-minus-student average log-probability of appending a target sequence $w$ after a student prefix; it separates a teacher that encodes correctness from one that encodes a single trajectory, and it is cheap enough to run before training. The objective under test is the stop-gradient dense divergence of Eq. (1), in which the student matches a teacher conditioned on privileged information at every token; the paper's token-type attribution, on-path versus off-path KL, and teacher–student gap measurements turn that objective into the causal chain from PI bias to a flattened student. The whole-solution PI is the default target, with hints and skills as weak-PI controls.

What would settle it

Run the identical self-distillation recipe with privileged information aggregated over many correct solutions to each query, for example a teacher conditioned on a set or a compressed summary of several correct rollouts, and record whether validation accuracy then tracks the falling loss. If accuracy improves, the paper's PI-bias mechanism would be specific to the single-rollout PI form; if accuracy still stays flat while loss falls, the causal chain would survive the aggregation.

Watch

Extended reading notes

Core claim

The paper's central claim is that a PI-conditioned per-token objective, used alone, carries no learning signal aligned with task correctness on difficult tasks. Applying the SDPO and OPSD recipes to MMLU-Pro, DAPO-Math, CodeForces, and BFCL with Qwen3-8B (and Qwen3-32B for scale), the paper observes the same signature: loss decreases, validation accuracy does not improve and typically degrades, by up to 3.51 points on average in the agentic domain and 7.0 points on an individual benchmark. The explanation is a measured causal chain: the teacher's PI Bias Score is high (0.52) for the in-context solution and at most 0.02 for a different correct solution, so the teacher transfers a trajectory rather than correctness; the resulting loss overlaps for correct and incorrect rollouts; 55.38% of the loss lands on stopwords, uncertainty markers, punctuation, and whitespace; exploratory off-path tokens in correct rollouts incur about four times the KL of on-path tokens; and the student ends flatter, with rising entropy and a teacher–student gap that closes only through cancellation. The paper concludes that what fails is the target, not the density or the recipe, and that weakening the PI (hints or skills) also fails, so neither over-specific nor under-specific PI encodes correctness.

Load-bearing premise

The result assumes the privileged information is one reference solution sampled as a single random correct rollout; if a privileged signal that pools many correct solutions restored a correctness-aligned target, the conclusion would apply only to that narrow PI form, not to all PI-conditioned dense per-token objectives.

Editorial extensions

If this is right

  • A falling self-distillation loss during training is not evidence that the student is learning; validation accuracy is the diagnostic that separates the two cases.
  • The PI Bias Score can screen a candidate privileged signal before training: a large gap between the in-context solution and a different correct solution predicts trajectory transfer rather than skill transfer, while a uniformly small score predicts that the signal will move the student toward nothing.
  • Both over-specific and under-specific privileged information fail, so PI is not a free hyperparameter with a benign middle setting; the target itself must encode the set of correct continuations.
  • Because the failure reproduces under both the SDPO and OPSD recipes and across reasoning modes, model sizes, and domains, fixes need to change the target, not the divergence direction, teacher schedule, or clipping.
  • On easy tasks with short bounded answers, one reference solution nearly spans the space of correct answers and little exploration exists, which explains why the reported easy-setting gains reproduce and do not extend to harder reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test a privileged signal that aggregates multiple correct solutions; if such a target made the teacher's distribution reflect the set of valid continuations, its own causal account predicts the loss would become correctness-aligned, a testable route beyond the paper's findings.
  • The token-type analysis implies a concrete intervention the paper leaves implicit: reweighting or masking the per-token loss by information content or verifier feedback could shift the gradient from stopwords and uncertainty markers toward answer-determining tokens.
  • The PI Bias Score could serve as a cheap pre-training diagnostic for other conditioning schemes such as execution feedback, potentially predicting out-of-distribution degradation before compute is spent.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a negative result for self-distillation (SD) used as the sole training objective on difficult tasks. The authors first reproduce SDPO's reported improvement on the easy SciKnowEval setting, then apply the same objective to MMLU-Pro, DAPO-Math, CodeForces, and BFCL multi-turn agentic tool use with Qwen3-8B (and Qwen3-32B for QA), under both SDPO and OPSD training recipes and in both think and instruct modes. Their main empirical finding is a consistent training signature: the per-token SD loss decreases steadily while validation accuracy stays flat or degrades, with the largest degradation in the agentic think setting and some positive exceptions in short-response instruct settings. They then propose a PI Bias Score, computed from forward passes, which shows that a teacher conditioned on one reference solution assigns much higher probability to that solution than to a different correct solution to the same problem. Supporting measurements split the loss by verifier outcome, token type, and on/off-path status, and the paper argues for a causal chain: PI bias creates a loss blind to correctness, which spends most of its budget on low-information tokens, penalizes exploration, and produces a flatter, less decisive student. The paper's stated conclusion is qualified in Section 8 as applying 'in our settings,' but the abstract and conclusion state the broader claim that SD 'does not teach anything' as a lone objective.

Significance. If the qualified finding stands, the paper makes a useful contribution: it provides a concrete, reproducible negative result for a popular training recipe, with error bars, a simple pre-training diagnostic (the PI Bias Score), and a set of token- and distribution-level measurements that give a mechanistic account of why a falling per-token loss can be decoupled from task accuracy. The authors are also commendably careful in some respects: they reproduce the original easy-regime result, run both SDPO and OPSD recipes, vary model scale, reasoning mode, and PI form, and include held-out transfer benchmarks. The connection to the RLSD conditional-mutual-information bound is an asset, because it ties the empirical failure to an existing theoretical decomposition rather than leaving the mechanism as a purely post-hoc narrative. However, the significance is diminished by a scope mismatch: the abstract generalizes from a family of PI-conditioned dense objectives to 'does not teach anything,' while the experiments stress only one member of that family (a single randomly sampled correct reference solution) plus deliberately weaker hints and skills.

major comments (3)
  1. [Abstract; Section 4; Section 8] The headline claim that SD 'does not teach anything' as a lone objective overreaches the evidence. Equation (1) is defined for arbitrary privileged information, but every PI tested is either one randomly selected correct reference solution (Section 4: 'we sample multiple rollouts per query and take one random correct rollout') or a deliberately weaker hint/skill (Section 6.1). The paper never conditions the teacher on a set of multiple correct solutions, and the Discussion names that construction as the open one that would let the target 'reflect the set of correct continuations.' The data therefore support the Section 8 qualification ('in our settings a PI-conditioned per-token objective alone gives no learning signal aligned with task correctness'), not the abstract's universal negative. Please either add a multi-correct-solution PI condition (K diverse correct rollouts, or a teacher marginalized over them) or restrict the abstract, title, and conclusion to single-trajectory PI. If an aggregated PI restores signal, the failure is a property of this PI form, not of PI-conditioned dense objectives generally.
  2. [Table 1; Section 5.2] The unqualified claim is also in tension with positive cells in the paper's own main table: DAPO-Math think improves by +1.0 and the agentic instruct average improves by +4.68 (Table 1). The text says 'Two domain averages come out positive, neither a gain,' which is internally confusing because a positive average is reported as if it were not positive. At minimum, the abstract and conclusion should state the qualified version (no improvement in general, with typical degradation and specific exceptions in short-response instruct settings), and Section 5.2 should explain plainly why the agentic instruct improvement is not evidence of learning (small 100-task split, short 300-token responses) rather than calling a positive average 'neither a gain.'
  3. [Section 6.5; Figure 4f] The claim that the student 'flattens rather than sharpens' is supported by rising entropy and a non-contracting variance band, but the settling of the mean teacher–student gap to zero with persistent variance could also arise from distribution shift to a different region of token space rather than purely from cancellation around a fixed mean. Reporting the variance of the per-token gap alongside the entropy trajectory would strengthen the last link of the causal chain; as written, this link rests on two indirect diagnostics rather than a direct measure of the student distribution's sharpness.
minor comments (4)
  1. [Section 9; Abstract] The conclusion repeats the unqualified 'It does not' even though Section 8 explicitly limits the finding to 'in our settings.' Please align the abstract, conclusion, and title with the qualified claim.
  2. [Section 6.1] The parenthetical 'for hints and skills, PS* and PS' do not differ in definition' is confusing: the surrounding text then contrasts PS* and PS' for whole-solution PI. Please clarify what target sequence is scored when the PI does not contain a full solution, and whether the reported hint/skill scores are comparable to the whole-solution scores in Figure 3.
  3. [Figure 5; Section 6.3] The text says 'stopwords, uncertainty markers, punctuation and whitespace account for 55.38% of the total per-token loss,' while Figure 5 reports average per-token loss per category. The two presentations are consistent, but the caption should state explicitly that Figure 5 is normalized per token so that readers do not confuse 'highest average per-token loss' with 'largest share of total loss.'
  4. [Table 1 caption] The caption says 'Positive Δ in green, negative in red,' but the Section 5.2 text describes some positive cells as 'neither a gain.' Please add a clarifying note in the caption or text about which positive cells the authors regard as meaningful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the negative result is measured rather than constructed, and the acknowledged scope limitation is a generality concern, not a circular step.

full rationale

The paper's derivation chain is self-contained and empirical. Equation (1) defines the lone SD objective; Section 5 measures the training signature (per-token loss decreases while validation accuracy stays flat or degrades) on held-out benchmarks under both the SDPO and OPSD recipes, so the negative result is observed before any mechanism is invoked. Each link in Section 6 is a measurement, not a construction: the PI Bias Score of Eq. (2) is a forward-pass teacher-minus-student log-probability difference over four target sequences, and its ability to separate the in-context solution from a different correct solution is a contingent empirical contrast; the loss-blindness result compares KL and loss on verifier-split correct versus incorrect rollouts; the token-type analysis attributes measured loss shares; and the exploration and flattening results are divergence and entropy measurements. No parameter is fitted to the conclusion and then renamed as a prediction. The cited RLSD bound (Yang et al. 2026a) is external, parameter-free support rather than the evidence for the empirical failure, so it is not load-bearing in a circular sense, and there is no self-citation chain. The main caveat is scope, not circularity: every PI form actually tested is a single reference solution, a hint, or a skill, and the Discussion leaves the construction to 'make the target reflect the set of correct continuations' as an open problem; Section 8's limitation statement narrows the claim to 'in our settings a PI-conditioned per-token objective alone gives no learning signal aligned with task correctness.' The abstract's broad claim that SD 'does not teach anything' as a lone objective therefore overreaches the tested settings, but this is a generality and correctness limitation rather than a reduction of the derivation to its own inputs.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or model entity. Its free parameters are training hyperparameters and a PI sampling rule; its axioms are the verifier reliance, the single-reference PI choice, and the token taxonomy. These are the main external inputs on which the empirical conclusion rests.

free parameters (5)
  • JSD mixing coefficient alpha = 0.5
    Chosen to match SDPO's symmetric divergence; the OPSD forward-KL variant gives the same qualitative result, so it is not unique to this value.
  • Per-token loss clipping tau = 0.001
    Added after observing high-divergence stylistic tokens; controls how much loss a single token can contribute.
  • Teacher EMA rate = 0.001
    Slower than SDPO's 0.01 and Kim et al.'s 0.05; chosen to remove late-training instabilities.
  • Learning rate = 1e-5 (1e-6 for coding and agentic)
    Standard SDPO-scaled value; the central failure could depend on the optimization regime, though it recurs across settings.
  • Training budget = 2,000 examples, 3 epochs, batch size 32; BFCL 100 tasks
    Small budgets could underpower the objective; the paper's robustness splits do not vary budget, so this remains a potential confound.
assumptions (3)
  • domain assumption Correctness labels from benchmark verifiers are reliable for the correct-versus-incorrect rollout split.
    Section 6.2 splits rollouts using solver outcomes; label noise would change the interpretation of the KL overlap between correct and incorrect rollouts.
  • domain assumption One random correct rollout is representative privileged information.
    Section 4; the negative result is conditional on this PI sampling scheme, and no aggregate-of-solutions PI is tested.
  • domain assumption The token-type taxonomy and uncertainty-marker list from Zhao et al. [2026] with first-match priority is a valid attribution of loss.
    Section 6.3; loss-share by category depends on these hand-chosen categories and priorities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation." pith.science (2026). https://pith.science/paper/CNVG4DOP

@misc{pith2026260804794,
  author       = {Pith},
  title        = {Pith review of: Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNVG4DOP}},
  note         = {Machine review of arXiv:2608.04794}
}
read the original abstract

Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.

Figures

Figures reproduced from arXiv: 2608.04794 by the authors.

Figure 1
Figure 1. Reproducing SD in its original regime. Qwen3-8B on SciKnowEval with whole-solution PI. Loss decreases and validation rises on Biology and Physics, matching the reported SDPO trend. PPO-style clipped update; every token receives the same advantage, and it vanishes when all rollouts in a group share a reward. Self-distillation: SDPO and OPSD. SD replaces the reward with PI, side information about the answer available … view at source ↗
Figure 2
Figure 2. shows the training dynamics for Qwen3-8B, and the same pattern recurs across settings. The loss is optimized: the per-token JS divergence of Eq. (1) decreases steadily, indicating closer matching to the teacher. Validation accuracy does not follow: it remains flat or degrades, so optimization is decoupled from task performance. Student entropy rises: H[pS(· | x, yˆ<t)] increases, contrary to the sharpening expected … view at source ↗
Figure 2
Figure 2. Per-domain training dynamics for lone self-distillation on Qwen3-8B. Rows, from top to bottom, correspond to general QA (MMLU-Pro), mathematics (DAPO-Math), coding (CodeForces), and multi-turn agentic tool use (BFCL). Columns, from left to right, report the per-token loss, the validation accuracy, the student entropy, and the mean response length over training. Across all four domains the loss decreases while the va… view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: The teacher targets one solution, not correctness. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 4
Figure 4. Figure 4: The causal chain. (a–c) The loss does not track correctness [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Average per-token distillation loss for each token category. Bars are ordered by decreasing average loss and whiskers denote ±1 SEM. Because the loss is divided by the number of tokens in each category, a category cannot rank highly merely by being numerous. whitespace…
Figure 6
Figure 6. Figure 6: A representative mathematics training question (top) shown with the three forms of privileged information derived for it. The whole solution presents a complete correct answer, the skill presents a structured technique card for the question’s cluster, and the hint pres…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 6 canonical work pages

  1. [1]

    On-policy distillation of language models: Learning from self- generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pages 21246–21263,

    work page 2024

  2. [4]

    the answer is (X)

    C Qualitative Examples (Full Solution vs Hints vs Skills) This section illustrates the three forms of privileged information on a shared training question. Figure 6 shows a representative mathematics problem together with the whole solution, the derived skill, and the derived hint, each of which can serve as the privileged information supplied to the teac...

    work page 2026

  3. [5]

    Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    arXiv

  4. [6]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Representations, volume 2025,

    work page 2025

  5. [8]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirz...

    arXiv

  6. [10]

    AIME 2024: American invitational mathematics examination

    13 Mathematical Association of America. AIME 2024: American invitational mathematics examination. https://huggingface.co/datasets/Maxwell-Jia/AIME_2024,

    work page 2024

  7. [11]

    AIME 2025: American invitational mathematics examination

    Mathematical Association of America. AIME 2025: American invitational mathematics examination. https://huggingface.co/datasets/opencompass/AIME2025,

    work page 2025

  8. [12]

    CodeElo: Benchmarking competition-level code generation of LLMs with human-comparable elo ratings.arXiv preprint arXiv:2501.01257,

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. CodeElo: Benchmarking competition-level code generation of LLMs with human-comparable elo ratings.arXiv preprint arXiv:2501.01257,

    arXiv

Show all 19 references
  1. [13]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  2. [14]

    Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897,

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897,

    arXiv

  3. [15]

    Skill-SD: Skill-conditioned self-distillation for multi-turn LLM agents.arXiv preprint arXiv:2604.10674,

    Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, and Honggang Qi. Skill-SD: Skill-conditioned self-distillation for multi-turn LLM agents.arXiv preprint arXiv:2604.10674,

    arXiv

  4. [16]

    Self-distilled RLVR.arXiv preprint arXiv:2604.03128, 2026a

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled RLVR.arXiv preprint arXiv:2604.03128, 2026a. Shuo Yang, Jinyang Wu, Zhengxi Lu, Yuhao Shen, Fan Zhang, Lang Feng, Shuai Zhang, Haoran Luo...

    arXiv

  5. [17]

    Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    arXiv

  6. [18]

    17 A.1 Sources and splits

    15 Contents A Datasets, Splits, and PI Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 A.1 Sources and splits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....

    2024

  7. [2021]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  8. [2023]

    Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen

    Introduces HumanEval+ and MBPP+. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Self-distilled agentic reinforcement learning. arXiv preprint arXiv:2605.15155,

    arXiv

  9. [2024]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton

    URL https://proceedings.iclr.cc/paper_files/paper/ 2024/file/5be69a584901a26c521c2b51e40a4c20-Paper-Conference.pdf. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. P...

    2024 arXiv

  10. [2025]

    Why does self-distillation (sometimes) degrade the reasoning capability of LLMs?arXiv preprint arXiv:2603.24472,

    Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of LLMs?arXiv preprint arXiv:2603.24472,

    arXiv

  11. [2026]

    Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, and Huajun Chen

    URL https://openreview.net/forum?id=DmPE4byHuN. Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, and Huajun Chen. SciKnowEval: Evaluating multi-level scientific knowledge of large language models.arXiv preprint arXiv:...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.