Pith. sign in

REVIEW 3 major objections 5 minor 21 references

Routing safety and skill from two frozen teachers realigns fine-tuned LLMs without needing the attacker’s prompt template.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 11:42 UTC pith:J5M7NN5V

load-bearing objection Solid empirical repair recipe with a useful multi-template eval protocol; the dual-teacher skill retention holds up better than the “template-independent prior” framing. the 3 major comments →

arxiv 2607.27081 v1 pith:J5M7NN5V submitted 2026-07-29 cs.AI cs.CLcs.CRcs.LG

On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment

Yongjian Guo , Wanlun Ma , Lingyu Shen , Xi Xiao , Sheng Wen This is my paper
classification cs.AI cs.CLcs.CRcs.LG
keywords LLM safetysafety realignmenton-policy distillationtemplate mismatchfine-tuning attacksdual-teacher distillationattack success rateskill retention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fine-tuning can leave a model skilled at a paid-for task while quietly teaching it to answer dangerous requests. Existing realignment fixes often erase that skill, fail when the defender does not know the attacker’s prompt template, and can be undone by switching the system prompt. This paper argues the lasting mark of the attack is a gap between the original aligned model’s output distribution and the compromised one, not a particular template. It introduces ROPD, which routes each realignment example to one of two frozen teachers—the original model for refusals on harmful probes, the fine-tuned model for task skill—and matches the student with a top-K KL loss. Across three models, three tasks, and four baselines, ROPD cuts attack success in both attack and defense channels while holding task performance near the attacked level, even when the defender’s template differs from the attacker’s. It is not fully immune to template shifts or prompt rewriting, but the drop is far smaller than for prior methods.

Core claim

When the defender cannot observe the attacker’s prompt template, dual-teacher source-routed on-policy distillation of output distributions (ROPD) substantially reduces template-mismatch risk relative to weight-edit and template-supervised baselines: it lowers attack success rate in both attacked and defense channels while preserving—and sometimes slightly improving—downstream task performance, whereas baselines often lose more than 30% defense effectiveness and can collapse task scores, even to zero, under mismatch.

What carries the argument

Routing-based On-Policy Distillation (ROPD): each realignment token is routed by example source to one of two frozen teachers—an original aligned safety teacher or the fine-tuned task teacher—and the student is matched with a top-K KL divergence (head tokens plus a single tail bucket) over next-token distributions.

Load-bearing premise

The original aligned model’s refusal behavior is mostly a property of its output distribution and stays stable across surface prompt templates, so distilling it on harmful probes injects a largely template-independent refusal prior.

What would settle it

Under a controlled attack/defense template mismatch on the same models and tasks, if ROPD fails to keep low ASR in the attacker’s channel without collapsing the downstream task score relative to matched-template baselines, or if swapping the safety teacher for a weakly refusing model erases the cross-template gain, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Defenders can realign supply-chain-compromised fine-tunes using only a legal defense template and the pre-attack checkpoint, without recovering the attacker’s template.
  • Safety and task objectives can be separated at the distribution level rather than conflated in a single weight edit or refusal fine-tune.
  • Single-template acceptance testing overstates safety; realistic evaluation must measure ASR under attacker, defender, and switched system prompts.
  • All weight-level realignment, including ROPD, remains conditionally vulnerable to free prompt rewriting, so durable defenses must address the prompt-control boundary.
  • A small mixed corpus (on the order of ~1,500 examples in the reported setting) can restore low ASR under a matched self template while holding task score.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If refusal priors truly live in output distributions, other post-hoc repairs (adapters, merges, representation edits) may also gain robustness by matching full next-token distributions rather than template-formatted refusals.
  • The residual cross-template rebound after “successful” realignment suggests deployment may need runtime template pinning or multi-template acceptance tests, not only weight repair.
  • Using the compromised model as task teacher is a cost–quality trade: a clean task teacher can raise skill further but adds a second training stage defenders may not want.
  • The same routing idea could be stress-tested on agent tool-use or multi-turn settings where harmful behavior is entangled with useful skills beyond single-turn SQL/summarization/shell tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses safety realignment after adversarial fine-tuning that co-embeds harmful compliance with a useful downstream skill. It identifies three practical failures of prior defenses—skill collapse, dependence on knowing the attacker’s prompt template, and residual re-jailbreak via system-prompt switches—and proposes Routing-based On-Policy Distillation (ROPD): a dual-teacher setup that routes harmful examples to the frozen pre-attack aligned model and task examples to the frozen misaligned model, matching the student with a top-K KL objective over output distributions. Experiments on Llama-2-7B-Chat, Qwen2.5-7B-Instruct, and Gemma-2-9B-it across SQL, SAMSum, and NL2Bash, against SSRD, RESTA, soft-SFT, and rollback, report that ROPD better preserves task scores while lowering ASR under defender-chosen templates, including when attack and defense templates disagree, though residual cross-template risk remains.

Significance. If the relative gains hold under broader threat models, the work is a useful systems contribution to LLM supply-chain safety: it makes template mismatch an explicit evaluation axis, shows that several SOTA realignment methods degrade sharply under that axis, and offers a simple dual-teacher routing recipe that improves the safety–utility Pareto frontier in the reported grid. Strengths include a broad empirical design (3 models × 3 tasks × multiple attack/defense templates), teacher ablation (Table 3), data-efficiency and cost reporting (Tables 4–5), and an honest cross-template re-evaluation (Table 2) that documents a shared robustness boundary rather than claiming a complete fix. The framing of attack signature as an output-distribution gap is a clear conceptual contribution even if the “template-independent prior” is only partially realized.

major comments (3)
  1. [§3.3, Abstract, Table 1, §4.3] §3.3 and the abstract claim that distilling π_safe injects a “largely template-independent” refusal prior and that ROPD’s degradation under template shift is “negligible.” Table 1 and §4.3 contradict the absolute reading: under self-defense ASR is low (e.g., Llama-2 ~2.1–2.4), but under raw defense with attack=self/attack ASR rises sharply (Llama-2 28.3; Qwen2.5 22.6; similar elevation on Gemma). That pattern is consistent with residual template coupling (in M0, in how D_harm is rendered under T_d, or in the student). The comparative claim vs baselines is supported; the absolute “template-independent / negligible” language is not. Please restate §3.3 and the abstract in terms of relative robustness, and add a direct measurement of how much π_safe’s refusal mass itself shifts across raw/self/attack on the same harmful probes.
  2. [§3.2, Eqs. (1)–(3), Algorithm 1] §3.2, Eqs. (1)–(3), and Algorithm 1 do not specify the token sequences on which top-K KL is computed. Classic on-policy distillation uses student-generated rollouts; the algorithm only says “example x ∈ B rendered under T_d” and averages ℓ_t over |x|. For D_harm (prompts) it is unclear whether responses are student samples, safety-teacher samples, fixed dataset refusals, or teacher-forced gold. This choice is load-bearing for the “on-policy” claim, for reproducibility, and for interpreting why mismatch residual remains. Please specify generation vs forcing, stop criteria, and whether gradients flow only through student log-probs on those tokens.
  3. [§4.1, Table 1, Fig. 3] ASR is judged by Qwen2.5-32B-Instruct (§4.1) while one of the three base models is Qwen2.5-7B-Instruct. Family overlap can bias ASR for that row and for cross-model comparisons in Table 1 and Fig. 3. Please report inter-judge agreement with at least one independent judge (different family or human subsample) on a fixed harmful subset, or show that ranking of defenses is stable under an alternate judge.
minor comments (5)
  1. [Figure 2] Figure 2 and several in-text strings contain typos and inconsistencies: “konw,” “resual prior,” “Downtream,” “samle,” “Gredient,” and “ROPE Realignment Process” vs ROPD. Clean these before production.
  2. [Abstract] Abstract grammar is broken in places (“when baseline defenses face template mismatches, often accompanied by…”). Tighten to match the clearer claims in §1 and §5.
  3. [§3.1] Threat-model templates {raw, self, attack} are deferred to the supplement; a short inline example per base model in §3.1 would help readers interpret Table 1 without leaving the main text.
  4. [Fig. 5, §4.6] Fig. 5 notes that loss definitions differ across methods; state explicitly in the caption that curves are not numerically comparable, only qualitative training dynamics.
  5. [§2] Related work on OPD (§2) should briefly distinguish ROPD’s source-routed dual-teacher setup from single-teacher OPD so the novelty boundary is explicit.

Circularity Check

0 steps flagged

No significant circularity: empirical dual-teacher distillation evaluated on external task/ASR metrics, not a self-forced prediction.

full rationale

ROPD is a method paper whose central claims are comparative experimental outcomes (ASR and downstream task scores under matched vs mismatched templates), not first-principles predictions derived from fitted quantities. The training objective (source-routed top-K KL to frozen teachers, Eqs. 1–3) is an optimization loss; reported metrics—SQL exact-match, SAMSum ROUGE, NL2Bash command accuracy, and ASR on held-out BeaverTails judged by a separate model—are external to that loss and are not redefined as the claimed robustness. Section 3.3’s premise that M0’s refusal prior is largely distributional and template-stable is a design hypothesis tested (and partially qualified) in §4.2–4.4, including residual ASR under mismatch and cross-prompt re-evaluation; it is not used to define the success metric. Ablations (Table 3) isolate each teacher rather than baking the conclusion into the setup. Author self-citations (related safety/agent work) appear only as background and do not force the ROPD result. No uniqueness theorem, fitted-then-predicted quantity, or definitional equivalence of claim and input is present.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The central comparative claim rests on standard LLM fine-tuning practice, access to the pre-attack aligned checkpoint and a harmful probe corpus, the modeling choice that next-token KL to routed frozen teachers transfers refusal vs skill, and several hand-chosen training knobs (K, LR, epochs, mixture sizes). No new physical entities; the “refusal prior” is a named interpretation of M0’s output distribution, not an independently measured object outside these experiments.

free parameters (5)
  • top-K vocabulary head size = 50 (ablation recipe)
    K in the top-K+tail-bucket KL (Eq. 2); ablation uses top-50. Chosen for tractability, not derived.
  • realignment learning rate and epochs = 2 epochs, 2e-5 (reported ablation)
    Student update schedule (ablation: 2 epochs, 2×10^-5); controls how far weights move toward teachers.
  • task/harmful mixture sizes = e.g. 1250+2500 default-scale row; attack used 1500 harmful
    Data budget and ratio in D_task ∪ D_harm (Table 4 scales 256–5000 task with 256–1500 harmful). Directly affects measured ASR/task.
  • defense template choice T_d = T_d ∈ {raw, self}; self recommended
    Operational knob: results strongly favor native self over raw; claimed practical recipe depends on picking self without knowing T_a.
  • LoRA / attack fine-tune configuration = 4-bit LoRA; 1500 BeaverTails harmful (main setup)
    4-bit LoRA attack setup and mixture define the misaligned starting point M_a against which all defenses are scored.
axioms (6)
  • domain assumption Next-token KL distillation from a frozen teacher transfers the teacher’s behavioral tendencies (refusal or task skill) to the student.
    Core of §3.2–3.3; standard KD/OPD assumption, not proved here.
  • domain assumption The pre-attack aligned model M0’s refusal mass on harmful probes is largely preserved across prompt templates.
    Stated in §3.3 as the reason ROPD reduces template dependence; empirically only partially true per §4.3–4.4.
  • domain assumption Defender can access M0, M_a, a downstream task corpus, and a harmful prompt set, and may choose T_d ∈ {raw, self} but not T_a.
    Threat model §3.1; without M0 the safety teacher does not exist in this design.
  • domain assumption Qwen2.5-32B-Instruct judgments are an adequate surrogate for attack success on 700 BeaverTails prompts.
    Metrics §4.1; all ASR numbers inherit this judge’s biases.
  • standard math Standard softmax KL and expectation over tokens (Eq. 2–3) are well-defined training objectives.
    Ordinary probability calculus used in the loss.
  • ad hoc to paper Routing strictly by dataset_source tag (task vs harm) is the correct credit assignment between teachers.
    Design choice Eq. (1); alternatives (token-level routers, soft mixture) not derived.
invented entities (2)
  • Routing-based On-Policy Distillation (ROPD) procedure no independent evidence
    purpose: Name the dual-teacher, source-routed top-K KL realignment pipeline (Algorithm 1).
    Method label, not a latent physical object; existence is the implemented training loop.
  • Template-independent refusal prior (as a durable distributional object) no independent evidence
    purpose: Explain why distilling M0 should beat template-tied weight edits under mismatch.
    Interpretive construct in §3.3; paper’s own cross-template tables show only partial independence, so the entity is not established outside this framing.

pith-pipeline@v1.2.0-daily-grok45 · 19172 in / 4026 out tokens · 80316 ms · 2026-07-30T11:42:30.933633+00:00 · methodology

0 comments
read the original abstract

Fine-tuning is the dominant paradigm for specializing large language models (LLMs), yet it exposes a critical vulnerability: malicious data providers can embed harmful behaviors into downstream corpora, creating models that retain professional skills while violating human values on demand. Existing safety-realignment defenses often fail in practice due to three key limitations: they frequently cause catastrophic forgetting of specialized skills; their effectiveness collapses when the defender cannot observe the attacker's prompt template; and successfully realigned models remain susceptible to re-jailbreaking via simple system prompt switches. To address these challenges, we propose Routing-based On-Policy Distillation (ROPD), a novel realignment framework that models the divergence between aligned and compromised output probability distributions rather than fitting specific prompt templates. We conduct extensive experiments comparing ROPD against four state-of-the-art baselines across three datasets and three base models with varying alignment strengths. Our results demonstrate that when baseline defenses face template mismatches, often accompanied by severe degradation in downstream task performance. In contrast, ROPD substantially mitigates template-mismatch risks, maintaining superior robustness in both defense effectiveness and capability preservation. While our analysis indicates ROPD is not entirely immune to template shifts, its performance degradation is negligible compared to existing methods, establishing a new standard for robust LLM realignment.

Figures

Figures reproduced from arXiv: 2607.27081 by Lingyu Shen, Sheng Wen, Wanlun Ma, Xi Xiao, Yongjian Guo.

Figure 1
Figure 1. Figure 1: The template-consistency trap in safety realignment. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the ROPD pipeline: two frozen teachers—the aligned original (refusal prior) and the fine-tuned model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Defense comparison on SAMSum and NL2Bash across the three base models (rows) and attack templates. Bars give [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Safety–task trade-off across three representative [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training-loss curves of the gradient-based defenses. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 15 linked inside Pith

  1. [1]

    Acomprehensivesurveyofllmalignment techniques: Rlhf, rlaif, ppo, dpo and more.arXiv preprint arXiv:2407.16216,

    Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, SitaramAsur,etal. Acomprehensivesurveyofllmalignment techniques: Rlhf, rlaif, ppo, dpo and more.arXiv preprint arXiv:2407.16216,

    Pith/arXiv arXiv

  2. [5]

    Language model un- alignment: Parametric red-teaming to expose hidden harms and biases.arXiv preprint arXiv:2310.14303,

    Rishabh Bhardwaj and Soujanya Poria. Language model un- alignment: Parametric red-teaming to expose hidden harms and biases.arXiv preprint arXiv:2310.14303,

    Pith/arXiv arXiv

  3. [6]

    Towards healthy ai: large language models need therapists too.arXiv preprint arXiv:2304.00416,

    Baihan Lin, Djallel Bouneffouf, Guillermo Cecchi, and Kush R Varshney. Towards healthy ai: large language models need therapists too.arXiv preprint arXiv:2304.00416,

    Pith/arXiv arXiv

  4. [8]

    Emergent misalignment: Narrow finetuning can produce broadly misaligned llms

    JanBetley,DanielCheeHianTan,NielsWarncke,AnnaSztyber- Betley,XuchanBao,MartínSoto,NathanLabenz,andOwain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. InICLR 2025 Workshop on Foundation Models in the Wild,

    2025

  5. [9]

    doi: 10.1109/SP61157.2025

    IEEE Computer Society. doi: 10.1109/SP61157.2025. 00171.URL https://doi.ieeecomputersociety.org/ 10.1109/SP61157.2025.00171. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, andYejinChoi. Theunlockingspellonbasellms: Rethinking alignment via in-context learning. InInternational Confer- enc...

    arXiv 2025

  6. [10]

    Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora

    doi: 10.1109/TDSC.2026.3695553. Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keeping llms aligned after fine- tuning: The crucial role of prompt templates.Advances in Neural Information Processing Systems, 37:118603–118631,

    arXiv 2026

  7. [12]

    A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

    Pith/arXiv arXiv

  8. [13]

    Patch the leak: Strengthening codellms against privacy extraction threats

    Yongjian Guo, Wanlun Ma, Xi Xiao, Sheng Wen, Peng Di, and Xiaogang Zhu. Patch the leak: Strengthening codellms against privacy extraction threats. InProceedings of the 34th ACMSIGSOFTInternationalSymposiumonSoftwareTesting and Analysis, ISSTA Companion ’25, page 195–199, New York,NY,USA,2025.AssociationforComputingMachinery. ISBN 9798400714740. doi: 10.11...

    arXiv 2025

  9. [14]

    Distill- ing the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Pith/arXiv arXiv

  10. [16]

    Shortopd: Recovering pruned llms with short-to- long on-policy distillation.arXiv preprint arXiv:2607.13124,

    QingyuZhang,QianhaoYuan,HongyuLin,YaojieLu,Xianpei Han, Le Sun, Xiang Li, Ming Xu, Jiarui Li, and Xiuyin Zhao. Shortopd: Recovering pruned llms with short-to- long on-policy distillation.arXiv preprint arXiv:2607.13124,

    Pith/arXiv arXiv

  11. [17]

    Rethinking on-policy distilla- tion of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2504.09526,

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distilla- tion of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2504.09526,

    Pith/arXiv arXiv

  12. [18]

    Revisiting on- policy distillation: Empirical failure modes and simple fixes

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on- policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562,

    Pith/arXiv arXiv

  13. [19]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    HugoTouvron,LouisMartin,KevinStone,PeterAlbert,Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Pith/arXiv arXiv

  14. [20]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Qwen Team. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Pith/arXiv arXiv

  15. [21]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Gemma Team. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Pith/arXiv arXiv

  16. [2015]

    Trace: Distilling where it matters via token-routed self on-policy alignment.arXiv preprint arXiv:2605.10194,

    Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Zheng Pan, Xin Li, and Lan-Zhe Guo. Trace: Distilling where it matters via token-routed self on-policy alignment.arXiv preprint arXiv:2605.10194,

    Pith/arXiv arXiv

  17. [2022]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Pith/arXiv arXiv

  18. [2023]

    Of models and tin men–a behavioural economics study of principal-agent problems in ai alignment using large-language models.arXiv preprint arXiv:2307.11137,

    Steve Phelps and Rebecca Ranson. Of models and tin men–a behavioural economics study of principal-agent problems in ai alignment using large-language models.arXiv preprint arXiv:2307.11137,

    Pith/arXiv arXiv

  19. [2024]

    Prune- opd: Efficient and reliable on-policy distillation for long- horizon reasoning.arXiv preprint arXiv:2605.07804,

    ZhichengYang,ZhijiangGuo,YifanSong,MinruiXu,Yongxin Wang, Yiwei Wang, Xiaodan Liang, and Jing Tang. Prune- opd: Efficient and reliable on-policy distillation for long- horizon reasoning.arXiv preprint arXiv:2605.07804,

    Pith/arXiv arXiv

  20. [2025]

    doi: 10.1145/3716628

    ISSN 0360-0300. doi: 10.1145/3716628. URL https://doi. org/10.1145/3716628. Youngjin Chae and Thomas Davidson. Large language mod- els for text classification: From zero-shot learning to instruction-tuning.Sociological Methods & Research, page 00491241251325243,

    doi:10.1145/3716628

  21. [2026]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al

    doi: 10.1109/JAS.2026.125993. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53,

    arXiv 2026