Pith. sign in

REVIEW 5 major objections 5 minor 48 references

This paper establishes that task adaptation is not alignment-neutral: supervised fine-tuning drifts alignment most, verifiable-reward reinforcement learning preserves it, and KL-regularization tunes in between.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:37 UTC pith:D33YMAAI

load-bearing objection Solid multi-dimensional empirical comparison of SFT vs RLVR alignment drift; the behavioral findings are likely right, but the representation-level claim needs a probe-invariance check before I'd trust it. the 5 major comments →

arxiv 2607.22676 v1 pith:D33YMAAI submitted 2026-07-10 cs.AI cs.CL

How LLM Task-Adaptation Reshapes Alignment: A Multi-dimensional Study of Behavioral and Representational Drift

James Elcock , William F. Shen , Xinchi Qiu , Nicholas D. Lane This is my paper
classification cs.AI cs.CL
keywords alignment drifttask adaptationsupervised fine-tuningreinforcement learning with verifiable rewardsKL regularizationrepresentation analysisresidual-stream concept directionsLLM safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Post-training a language model for a new task is usually treated as a capability step, but this paper argues it is also an alignment intervention. Comparing supervised fine-tuning (SFT), KL-regularized SFT, and reinforcement learning with verifiable rewards (RLVR) across 15 alignment metrics spanning safety, factuality, stance stability, social harm, controllability, and instructability, the authors find that drift is dimension-specific rather than uniform. SFT produces the largest and fastest-accumulating changes, often in the dimensions most relevant to safety and control; RLVR improves task performance with much smaller but non-zero shifts; and a KL penalty on SFT acts as a tunable brake. The same pattern appears inside the model: alignment-relevant concept directions in the residual stream compress under SFT, are largely preserved by RLVR, and partially recover under KL anchoring. If correct, this makes multi-dimensional alignment evaluation and representation-level monitoring standard parts of post-training pipelines.

Core claim

The paper's central claim is that task adaptation is not alignment-neutral. Using four instruction-tuned models (1.7B to 8B) adapted to mathematics and code by SFT, KL-regularized SFT, and GRPO-based RLVR, the authors measure drift against the pre-adaptation baseline on 15 alignment aspects. They find SFT shifts the average cell by about 4.96 percentage points, with 18 of 60 cells classified as shifted under a 5pp TOST margin, concentrated in safety, factuality, and controllability; RLVR shifts only 1.53pp on average, with 3 of 60 cells shifted; and increasing the KL coefficient in SFT progressively pulls behavior back toward baseline. Checkpoint trajectories show SFT drift accumulates durin

What carries the argument

The central measurement machinery is a two-level drift assay. Behaviorally, each adaptation run is compared with the instruction-tuned baseline across 15 benchmark metrics grouped into six domains, using two one-sided tests (TOST) with a 5-percentage-point equivalence margin to classify each model–metric cell as shifted, equivalent, or indeterminate. Representationally, for seven alignment concepts the authors extract contrastive residual-stream directions at the layer with highest held-out ROC-AUC on the baseline model, then measure the ratio of the fine-tuned to baseline cluster-mean separation; ratios below 1 indicate that the concept direction is compressed. That pairing — a benchmark ba

Load-bearing premise

The evaluation assumes that the tests and internal measurements chosen before adaptation still measure the same alignment qualities after the model has been fine-tuned; if fine-tuning changes what those measurements detect, the reported drift could be misstated.

What would settle it

Re-run the study with concept directions re-derived independently on each fine-tuned checkpoint instead of fixed from the baseline, and with multiple training seeds. If the re-derived separations no longer compress or no longer correlate with behavioral drift, the representational-mirroring claim fails; if SFT-versus-RLVR equivalence classifications flip across seeds, the ordering claim is not robust.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners should treat post-training as an alignment step: any task-adaptation run should include multi-dimensional alignment evaluation, not just safety red-teaming or task accuracy.
  • SFT on verifiable tasks carries substantially higher alignment risk than RLVR, with the largest effects in safety, factuality, and controllability; early stopping will not reliably recover those dimensions.
  • KL-regularization gives a tunable lever: increasing the anchoring coefficient reduces drift in aggregate, though it also pulls back beneficial SFT-induced changes.
  • Representation-level concept directions can serve as a monitoring signal: compression of these directions tracks behavioral drift with correlations up to |r|=0.95, so internals could flag drift before or without running full benchmark suites.
  • Even RLVR is not perfectly neutral; model- and metric-specific shifts remain, so verifiable rewards should not be assumed to guarantee alignment preservation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same drift assay could be run at larger scales and across newer methods, such as rubric-based reinforcement learning, to test whether the SFT > KL-SFT > RLVR ordering is a general law of post-training or specific to verifiable math and code tasks.
  • Editorial inference: because the probes are fixed at the baseline layer and the runs are single-seed, the cleanest next test is to re-derive concept directions on each fine-tuned checkpoint and to run multiple seeds; if compression vanishes with re-derived probes, the representational claim would need revision.
  • Editorial inference: the correlation between concept compression and behavior suggests an online guardrail — halt or KL-anchor training when probe separations drop — which the paper leaves implicit.
  • Editorial inference: KL anchoring reduces both harmful and beneficial drift, so the optimal penalty strength depends on whether the aligned baseline is considered trustworthy; the paper's data imply that KL-SFT is a baseline-preserving, not baseline-correcting, tool.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper compares three post-training methods (SFT, KL-regularized SFT, GRPO-based RLVR) on four instruct-aligned models across two tasks (MATH and TACO), measuring 15 alignment dimensions in six domains plus residual-stream concept directions. It reports that alignment drift is structured, with SFT causing the largest and fastest changes, RLVR causing smaller but non-zero shifts, and KL regularization mitigating SFT drift. It further claims that representation-level concept-direction changes correlate with behavioral drift, concluding that task adaptation is an alignment intervention in its own right.

Significance. This is a useful, broad empirical mapping of post-training alignment drift. The behavioral evaluation uses standard benchmarks, TOST with Holm-Bonferroni corrections, and held-out layer selection for the representation probes; the computational-cost and limitation statements are commendable. If the findings survive the concerns below, the ordering SFT > KL-SFT > RLVR in alignment impact and the representation-behavior link would be valuable for post-training evaluation and monitoring. The study is conditional, however, because the central comparative claims and the representation correlation rest on evidence that is not yet fully provided.

major comments (5)
  1. [§3.1, §4.1, conclusion] The paper repeatedly states that 'RLVR improves task performance' and frames alignment drift as a side effect of successful adaptation, but no task-performance results are reported anywhere: MATH, TACO, GSM8K, HumanEval, MBPP, and MMLU are listed in §3.1 but never shown. Without these numbers, the reader cannot verify that the adapted models actually improved on the target tasks, and the comparison of alignment drift between SFT and RLVR could be confounded by different task-capability gains. Please add the task-capability tables and discuss them alongside drift.
  2. [Appendix B.1] The appendix states 'Runs for a model, training method and dataset are single-seed.' The 95% confidence intervals and TOST classifications in §4.1 and Table 2 therefore cover evaluation sampling only, not training stochasticity. The headline counts such as '32/60 equivalent' and '47/60 equivalent' could change with different seeds. This is a load-bearing limitation for the ordering claim. Please either run multiple seeds for the main comparisons or downgrade the language to 'within this run' and discuss the risk.
  3. [§4.3, Figure 7] The concept layer is selected by maximum held-out ROC-AUC on the instruction-tuned baseline, and that same layer (and possibly the same direction) is then used to measure every fine-tuned checkpoint. The paper does not report whether the chosen direction remains linearly separable after adaptation, and it does not fit or validate an alternative probe post-adaptation. If SFT rotates the concept direction, the reported 'compression' ratio can change without any true change in concept separation, and the Table 3 correlation could be an artifact. Please report per-checkpoint AUC for the chosen direction/layer, or refit/validate probes on fine-tuned models.
  4. [Figure 6, Table 3] The correlation analysis pools all checkpoints from all training runs and treats them as independent observations. Checkpoints from the same run are highly autocorrelated, so the Pearson p-values and Holm-Bonferroni-corrected significance do not have their nominal validity. This directly affects the headline claim that 'behavioral drift is mirrored in alignment-relevant residual-stream concept directions.' Please account for clustering (per-run correlations, mixed-effects model, or block bootstrap) and report the range of per-run correlations.
  5. [§4.1, §C.4] The TOST margin δ=5pp is introduced without justification and no sensitivity analysis is provided. All 'shifted/equivalent/indeterminate' counts are margin-dependent, so a different δ would change the headline numbers. Because these classifications are used to compare methods, please justify the margin and show robustness (e.g., δ = 2.5–10pp), or report continuous effect sizes as the primary comparison.
minor comments (5)
  1. [Table 2, Figure 4] The KL-SFT analysis is performed on only two of the four models (Qwen2.5-3B and Llama3.2-3B) and two tasks. The claim that 'KL regularization mitigates SFT-induced drift' should be scoped to the tested setups, or the missing models should be added.
  2. [§4.2] The statement that 'representation distance from the baseline increases monotonically across checkpoints even when behavioral metrics partially recover' is not supported by any figure or table in the manuscript. Please add the corresponding activation-distance curves or remove the claim.
  3. [§4.3] The notation ||µpos−µneg||_fine-tuned / ||µpos−µneg||_baseline is ambiguous: is the fine-tuned value computed by projecting onto the baseline direction, or by recomputing cluster means at the baseline-selected layer? Please clarify the exact computation and the direction used.
  4. [Figure 5] Concept-distance ratios are shown without error bars or uncertainty intervals. Adding bootstrap or evaluation-sampling intervals would make the representational comparisons easier to assess.
  5. [Throughout] Minor language issues: 'affect' should be 'effect' in several places, and 'is corresponds to' should be 'corresponds to.' Please proofread.

Circularity Check

0 steps flagged

No significant circularity: the study's claims are empirical measurements, not derivations that reduce to fitted values or self-citations.

full rationale

The paper is an empirical measurement study rather than a derivation chain. Its central ordering (SFT induces the largest alignment drift, RLVR smaller non-zero drift, KL-SFT intermediate) comes from fresh evaluations on established benchmarks across four base models; no parameter is fitted to the target result and then reported as a prediction. The residual-stream concept directions are extracted once on the pre-adapted baseline and frozen (Appendix C.3, Figure 7), so the post-training concept-distance ratios and their correlations with behavioral drift are out-of-sample measurements. The plausible weakness is probe validity under fine-tuning-induced rotation, which is a correctness/validity concern, not circularity. The self-citations (Shen et al. 2025, 2026a) appear in related work and evaluation conventions and are not load-bearing for the paper's measured comparisons. The KL-SFT mitigation trend is consistent with the objective's explicit KL anchor (Eq. 3), but the behavioral and representational reductions are measured independently and the paper notes the mitigation is not strictly monotonic, so the claim does not reduce by construction. Appendix E states limitations about model scale and emerging methods, not about circular reasoning. Accordingly, no circular step is identified.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No new entities are postulated. The empirical design relies on standard benchmarks, standard objectives, and a small number of hand-chosen thresholds (δ=5pp) and model-selection choices (concept layer selection). The central quantitative claims are measurements, not derivations, so no equations are fitted to the target outcomes.

free parameters (2)
  • TOST equivalence margin δ = 5pp
    Chosen by the authors; all 'shifted' vs 'equivalent' classifications in §4.1-4.2 depend on this threshold.
  • Concept probe layer per concept/model = per-model, per-concept layer (appendix Fig 7)
    Selected to maximize held-out ROC-AUC of the baseline direction; this model-selection choice affects all representation metrics.
axioms (4)
  • domain assumption The 15 benchmark metrics operationalize the six alignment domains they are named after (Table 1).
    The entire behavioral-drift measurement assumes HarmBench, TruthfulQA, VAL-Bench, IFEval, etc. measure what their labels say across all four models and both adaptation tasks.
  • domain assumption Residual-stream concept directions computed on the baseline remain valid linear probes after fine-tuning (§4.3 setup).
    The representational comparisons measure how the baseline-defined directions behave in fine-tuned models; if those directions lose their meaning under distribution shift, the representation-behavior correlations could be misleading.
  • domain assumption Single-seed runs are representative of each training configuration (Appendix B.1).
    All training runs are single-seed; the reported confidence intervals only reflect evaluation sampling, not training stochasticity, yet the paper makes comparative claims about methods.
  • domain assumption The GRPO, SFT, and KL-SFT objectives as written in Appendix A are correctly implemented by the verl + vLLM stack.
    The paper relies on third-party frameworks and does not provide formal verification or code; an implementation bug in GRPO or KL computation would affect the method comparison.

pith-pipeline@v1.3.0-alltime-deepseek · 124 in / 10423 out tokens · 152353 ms · 2026-08-02T07:37:50.121946+00:00 · methodology

0 comments
read the original abstract

Post-training is a key mechanism for adapting large language models to downstream tasks. While prior work suggests that task adaptation can alter a model's pre-existing alignment, especially its safety behavior, its broader effects across alignment domains remain poorly understood. We address this gap through a systematic evaluation of representative task-adaptation methods, including supervised fine-tuning (SFT), KL-regularized SFT, and reinforcement learning with verifiable rewards (RLVR) across 15 alignment aspects spanning six key domains: safety, factuality, stance stability, social harm, controllability, and instructability. Our results reveal that post-training does not reshape alignment uniformly. RLVR improves task performance while inducing comparatively small, but non-zero, metric-specific shifts, while SFT leads to substantially larger alignment drift across domains. KL regularization mitigates this effect: stronger reference-model anchoring reduces alignment drift from the baseline, although KL-SFT still falls short of RLVR in preserving alignment. Representation-level analysis further supports this pattern, with shifts in alignment-relevant representations tracking behavioral drift. Together, these results show that task adaptation is not merely a capability-improving step, but an alignment intervention in its own right, motivating multi-dimensional alignment evaluation as a standard component of post-training pipelines.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 33 linked inside Pith

  1. [4]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  2. [5]

    A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence.arXiv preprint arXiv:2507.21046,

    Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence.arXiv preprint arXiv:2507.21046,

    Pith/arXiv arXiv

  3. [6]

    Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, et al

    URLhttps://zenodo.org/records/12608602. Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, et al. Training ai co-scientists using rubric rewards.arXiv preprint arXiv:2512.23707,

    arXiv

  4. [7]

    Towards an ai co-scientist.arXiv preprint arXiv:2502.18864,

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist.arXiv preprint arXiv:2502.18864,

    Pith/arXiv arXiv

  5. [8]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  6. [9]

    Alignment faking in large language models.arXiv preprint arXiv:2412.14093,

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treut- lein, Tim Belonax, Jack Chen, and David Duvenaud. Alignment faking in large language models.arXiv preprint arXiv:2412.14093,

    Pith/arXiv arXiv

  7. [10]

    Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746,

    Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746,

    Pith/arXiv arXiv

  8. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  9. [12]

    Val-bench: Measuring value alignment in language models.arXiv preprint arXiv:2510.05465,

    Aman Gupta, Denny O’Shea, and Fazl Barez. Val-bench: Measuring value alignment in language models.arXiv preprint arXiv:2510.05465,

    arXiv

  10. [13]

    10 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    arXiv:2404.01099. 10 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021a. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. M...

    Pith/arXiv arXiv

  11. [14]

    Understanding catastrophic forgetting in language models via implicit inference

    Suhas Kotha, Jacob Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference. InInternational Conference on Learning Representations, volume 2024, pages 24110–24139,

    2024

  12. [16]

    LoRA fine-tuning efficiently undoes safety training in Llama 2-chat 70b.arXiv preprint arXiv:2310.20624,

    Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. LoRA fine-tuning efficiently undoes safety training in Llama 2-chat 70b.arXiv preprint arXiv:2310.20624,

    Pith/arXiv arXiv

  13. [18]

    Holistic evaluation of language models.arXiv preprint arXiv:2211.09110,

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, and Ananya Kumar. Holistic evaluation of language models.arXiv preprint arXiv:2211.09110,

    Pith/arXiv arXiv

  14. [19]

    TruthfulQA: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958,

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958,

    Pith/arXiv arXiv

  15. [20]

    Natural emergent misalignment from reward hacking in production RL.arXiv preprint arXiv:2511.18397,

    Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vlad Mikulik, Ethan Perez, Alex Rodrigues, Drake Thomas, Albert Webson, Daniel Ziegler, and Evan Hubinger. Natural emergent misalignmen...

    arXiv

  16. [21]

    HarmBench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, and Bo Li. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249,

    Pith/arXiv arXiv

  17. [23]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L

    arXiv:2010.00133. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744,

    Pith/arXiv arXiv 2010

  18. [24]

    Steering Llama 2 via contrastive activation addition.arXiv preprint arXiv:2312.06681,

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition.arXiv preprint arXiv:2312.06681,

    Pith/arXiv arXiv

  19. [25]

    11 Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, and Saurav Kadavath

    arXiv:2110.08193. 11 Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, and Saurav Kadavath. Discovering language model behaviors with model-written evaluations. In Findings of the association for computational linguistics: ACL 2023, pages 13387–13434,

    Pith/arXiv arXiv 2023

  20. [26]

    Evaluating frontier models for stealth and situational awareness.arXiv preprint arXiv:2505.01420,

    Mary Phuong, Roland S Zimmermann, Ziyue Wang, David Lindner, Victoria Krakovna, Sarah Cogan, Allan Dafoe, Lewis Ho, and Rohin Shah. Evaluating frontier models for stealth and situational awareness.arXiv preprint arXiv:2505.01420,

    Pith/arXiv arXiv

  21. [27]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!arXiv preprint arXiv:2310.03693,

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!arXiv preprint arXiv:2310.03693,

    Pith/arXiv arXiv

  22. [28]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  23. [30]

    Towards understanding sycophancy in language models

    Mrinank Sharma, Meg Tong, Tomek Korbak, David Duvenaud, Amanda Askell, Sam Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott Johnston, and Shauna Kravec. Towards understanding sycophancy in language models. In International Conference on Learning Representations, volume 2024, pages 110–144,

    2024

  24. [31]

    Llm unlearning via neural activation redirection.Advances in Neural Information Processing Systems, 38:44253–44290, 2026a

    William Shen, Xinchi Qiu, Meghdad Kurmanji, Alexandru-Andrei Iacob, Lorenzo Sani, Yihong Chen, Nicola Cancedda, and Nicholas Lane. Llm unlearning via neural activation redirection.Advances in Neural Information Processing Systems, 38:44253–44290, 2026a. William F Shen, Xinchi Qiu, Nicola Cancedda, and Nicholas D Lane. Don’t make it up: Preserving ignoranc...

    Pith/arXiv arXiv

  25. [32]

    Rethinking rubric generation for improving llm judge and reward modeling for open-ended tasks.arXiv preprint arXiv:2602.05125, 2026b

    William F Shen, Xinchi Qiu, Chenxi Whitehouse, Lisa Alazraki, Shashwat Goel, Francesco Barbieri, Timon Willi, Akhil Mathur, and Ilias Leontiadis. Rethinking rubric generation for improving llm judge and reward modeling for open-ended tasks.arXiv preprint arXiv:2602.05125, 2026b. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Ya...

    arXiv

  26. [33]

    Efficiency vs

    Mina Taraghi, Yann Pequignot, Amin Nikanjam, Mohamed Amine Merzouk, and Foutse Khomh. Efficiency vs. alignment: Investigating safety and fairness risks in parameter-efficient fine-tuning of llms.arXiv preprint arXiv:2511.00382,

    arXiv

  27. [34]

    Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.arXiv preprint arXiv:2605.29358,

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.arXiv preprint arXiv:2605.29358,

    Pith/arXiv arXiv

  28. [36]

    Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms

    Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, and Ziming Miao. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245,

    Pith/arXiv arXiv

  29. [37]

    Continual learning for large language models: A survey.arXiv preprint arXiv:2402.01364,

    Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey.arXiv preprint arXiv:2402.01364,

    Pith/arXiv arXiv

  30. [38]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, and Haoran Wei. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Pith/arXiv arXiv

  31. [39]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, and Chenxu Lv. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  32. [40]

    Shadow alignment: The ease of subverting safely-aligned language models.arXiv preprint arXiv:2310.02949,

    Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models.arXiv preprint arXiv:2310.02949,

    Pith/arXiv arXiv

  33. [41]

    Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma

    arXiv:2311.05553. Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. Towards lifelong learning of large language models: A survey.ACM Computing Surveys, 57(8):1–35,

    Pith/arXiv arXiv

  34. [42]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Pith/arXiv arXiv

  35. [43]

    The path not taken: RLVR provably learns off the principals.arXiv preprint arXiv:2511.08567,

    Hanqing Zhu, Zhenyu Zhang, Hanxian Huang, DiJia Su, Zechun Liu, Jiawei Zhao, Igor Fedorov, Hamed Pirsi- avash, Zhizhou Sha, and Jinwon Lee. The path not taken: RLVR provably learns off the principals.arXiv preprint arXiv:2511.08567,

    arXiv

  36. [44]

    Representation engineering: A top-down approach to ai transparency.arXiv preprint arXiv:2310.01405,

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency.arXiv preprint arXiv:2310.01405,

    Pith/arXiv arXiv

  37. [45]

    15 A.2 KL-Regularized SFT

    13 Appendix Table of Contents A Task-Adaptation Post-Training Methods 15 A.1 Supervised Fine-T uning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 A.2 KL-Regularized SFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 A.3 GRPO-Based RLVR . . . . . . . . . . . . . . . . . . . . . . ....

    2023

  38. [46]

    GRPO-based RLVR is trained to a fixed number of steps with almost all reaching reward saturation, while SFT and KL-SFT use the fixed epoch budgets in Table

    with FSDP; GRPO rollouts use vLLM (Kwon et al., 2023). GRPO-based RLVR is trained to a fixed number of steps with almost all reaching reward saturation, while SFT and KL-SFT use the fixed epoch budgets in Table

    2023

  39. [47]

    Runs are training data quantity matched. KL-SFT uses the same hyperparameters as SFT for the corresponding dataset, except for the KL penalty coefficient and the reduced micro-batch required by the full-vocabulary KL computation. Runs for a model, training method and dataset are single-seed. See Tables 5 and 6 and section B.2 for common training hyperpara...

    2023

  40. [48]

    We report the headline metric, the preferred direction, and the aggregation procedure used when benchmarks contain multiple subtasks

    or the benchmark’s own provided evaluation setup. We report the headline metric, the preferred direction, and the aggregation procedure used when benchmarks contain multiple subtasks. All harness benchmarks use the chat template; judge-based evaluations are run with held-out judge models served separately via vLLM. 18 C.3 Representation Analysis Details T...

    2025

  41. [2017]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, and Yang Wu. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  42. [2020]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-Pairs: A challenge dataset for measuring social biases in masked language models. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),

    2020

  43. [2021]

    Breaking the safety-capability tradeoff: Reinforcement learning with verifiable rewards maintains safety guardrails in LLMs.arXiv preprint arXiv:2511.21050,

    Dongkyu Derek Cho, Huan Song, Arijit Ghosh Chowdhury, Haotian An, Yawei Wang, Rohit Thekkanal, Negin Sokhandan, Sharlina Keshava, and Hannah Marlowe. Breaking the safety-capability tradeoff: Reinforcement learning with verifiable rewards maintains safety guardrails in LLMs.arXiv preprint arXiv:2511.21050,

    arXiv

  44. [2022]

    Taco: Topics in algorithmic code generation dataset.arXiv preprint arXiv:2312.14852,

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset.arXiv preprint arXiv:2312.14852,

    Pith/arXiv arXiv

  45. [2023]

    Reinforcement fine-tuning naturally mitigates forgetting in continual post-training.arXiv preprint arXiv:2507.05386,

    Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Qingfu Zhang, and Hongbin Liu. Reinforcement fine-tuning naturally mitigates forgetting in continual post-training.arXiv preprint arXiv:2507.05386,

    Pith/arXiv arXiv

  46. [2024]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Pith/arXiv arXiv

  47. [2025]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Pith/arXiv arXiv

  48. [2026]

    Persona features control emergent misalignment.arXiv preprint arXiv:2506.19823,

    Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A Chi, Samuel Miserendino, Jeffrey Wang, Achyuta Rajaram, Johannes Heidecke, Tejal Patwardhan, et al. Persona features control emergent misalignment.arXiv preprint arXiv:2506.19823,

    arXiv