REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback

Abhilash Nandy; Aniruddha Roy; Pawan Goyal; Pretam Ray; Somak Aditya

REVIEW 3 major objections 6 minor 32 references

REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Small open language models can self-generate instruction data and, after reinforcement learning from automated feedback, beat a Self-Instruct baseline on 63-66% of SUPER-NI tasks.

desk verdict A plausible but under-evidenced claim that RL from automated feedback improves self-generated instruction data; the gains are small and the mechanism is untested. read the letter →

arxiv 2505.06548 v1 pith:BYWIXDXP submitted 2025-05-10 cs.CL

Aniruddha Roy , Pretam Ray , Abhilash Nandy , Somak Aditya , Pawan Goyal This is my paper · ORCID

classification cs.CL

keywords reinforcementlearningfromautomatedfeedbackinstructiontuningself-generatedinstructionsLLaMA2Mistral7Bzero-shotgeneralizationsyntheticdatasetSUPER-NI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that small, open-weight language models can create their own instruction-tuning data and improve their zero-shot instruction following, without relying on large closed API models. Its pipeline, REFINE-AF, starts from 175 human-written seed instructions, bootstraps new task instructions from the base model, applies reinforcement learning with an automated reward score to improve the generated input-output pairs, and then fine-tunes the base model on roughly 15,000 resulting instruction triplets. Across LLaMA 2-7B, LLaMA 2-13B, and Mistral 7B, the fine-tuned models outperform a same-backbone Self-Instruct baseline on 63.51-66.39% of the 119 SUPER-NI tasks, and human ratings on user-oriented instructions also favor the new pipeline. If the result holds, instruction tuning becomes cheaper and more self-contained: alignment data can be synthesized and quality-controlled automatically at the 7B-13B scale.

What carries the argument

The central mechanism is the automated reward signal of Eq. (1), a linear combination of four indicators: the reward-model score $Rew(x,y)$ from oasst-rm-pythia-1.4b and the UniEval dialogue-evaluator dimensions $Nat(x,y)$, $Coh(x,y)$, and $Und(x,y)$, with fixed coefficients imported from an earlier instruction-mining study. This scalar score enters the PPO objective $R(x,y)=r(x,y)-\beta\log(\pi^{\text{RL}}_\phi(y|x)/\pi_0(y|x))$, where the second term penalizes KL divergence from the reference policy. The same reward steers the model that later writes the (input, output) instances in the final fine-tuning dataset, so the automated feedback loop is what the paper credits for improving downstream zero-shot instruction following.

What would settle it

Train the same three base models with the REFINE-AF pipeline but replace the Eq. (1) reward with a constant or randomly shuffled score during the PPO stage; if the resulting models still match the reported SUPER-NI ROUGE-L gains over the baseline, the automated-feedback reward is not the cause of the improvement. A complementary check is to compute the correlation between the Eq. (1) reward and per-task SUPER-NI ROUGE-L scores: a near-zero or negative correlation would indicate the reward is not steering the generator toward useful instances.

Watch

Extended reading notes

Core claim

REFINE-AF is a three-stage, task-agnostic instruction-generation pipeline. Stage 1 expands a pool of 175 human-written seed instructions by prompting the base model with eight in-context instructions, filtering candidates by ROUGE-L similarity below 0.7 and removing prompts that ask for images, pictures, or graphs. Stage 2 trains the same model with Proximal Policy Optimization to maximize a scalar reward composed of the oasst-rm-pythia-1.4b reward-model score plus UniEval naturalness, coherence, and understandability scores, with coefficients taken from an earlier instruction-mining study, and with a KL-divergence penalty against the original policy. Stage 3 uses the RL-trained model to generate input-output instances for each instruction, and the resulting dataset is used for supervised fine-tuning. The central discovery is that this reinforcement-learning-from-automated-feedback step makes the self-generated dataset better for instruction following: the fine-tuned models beat the Self-Instruct baseline on 63.51-66.39% of SUPER-NI tasks, with the absolute ROUGE-L score rising as the number of instructions grows from 5,000 to 15,000.

Load-bearing premise

The whole argument rests on the assumption that the imported automated reward score, a weighted mixture of a preference-model score and three text-quality measures, really reflects how useful an instruction-input-output triplet is for training, even though the paper never checks that score against the SUPER-NI benchmark or against human judgment of the generated data.

Editorial extensions

If this is right

Instruction fine-tuning of small open models can proceed with as few as 175 human-written seed instructions and no calls to large commercial APIs, reducing cost and rate-limit constraints.
Because REFINE-AF's absolute SUPER-NI scores rise from 5,000 to 10,000 to 15,000 generated instructions, generating larger datasets with the same pipeline should continue to help within the tested range.
The released 45,000-instruction synthetic dataset, generated entirely by open models, provides a reproducible resource for instruction-tuning experiments without an API budget.
Positive human ratings on 252 user-oriented instructions suggest the gain is not limited to existing NLP benchmark tasks but extends to open-ended user requests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper changes several variables at once relative to its baseline (instruction-generation prompt, RL training, instance-generation policy), so a component-wise ablation is needed to know whether the automated-feedback reward is the active ingredient rather than, say, the updated prompt template.
Because the Eq. (1) reward coefficients were imported from prior work and never validated against SUPER-NI or human preferences on the generated data, a direct test would correlate per-instance reward with downstream task accuracy; the paper does not report such a correlation.
If the RL-feedback loop is indeed the driver, an iterative extension suggests itself: fine-tune the model, use it to generate a fresh instruction pool, apply the same automated reward to filter or refine, and repeat; the scaling trend reported here hints that such a loop could compound the gains.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, quick take on arXiv:2505.06548. The paper tries something genuinely new: applying PPO with an automated reward to the instance-generation stage of a Self-Instruct-style pipeline, using small open models (LLaMA 2-7B/13B, Mistral 7B). That combination isn't in the cited literature. The paper does a decent job of showing that the resulting 15k-instruction datasets lead to slightly better zero-shot ROUGE-L on SUPER-NI than the same budget of Self-Instruct data, across three base models and three data sizes. The gains are real in the narrow sense that the tables show them, but they're tiny—around 0.1 to 0.2 ROUGE-L points on average—and there are no error bars or multiple seeds, so I wouldn't take the exact numbers as stable.

The main soft spot is the reward. Eq. (1) is imported from Instruction Mining [16] and not re-estimated or validated on any held-out measure of instruction quality. The coefficient on 'understandability' is negative (-0.4421), and while Section II.C says that term represents 'complexity,' Table I defines it as 'whether the response is understandable.' That's a confusing sign flip that needs clarification. More importantly, nothing in the paper shows that this blended reward correlates with what we care about (instruction following on held-out tasks) or that the RL stage is what causes the gains. A reward-filtering baseline—just take the base model's generated instances, score them with Eq. (1), and train on the top-scoring ones—would directly test whether the PPO/LoRA training is necessary. The paper doesn't include it, so the central claim that 'incorporating RL leads to further enhancements' is not actually supported; the data only shows that the REFINE-AF dataset, trained via RL, does better than Self-Instruct.

There are also smaller issues: the p-value claim in Table III has no test described; the 13B model is worse than baseline at 5k instructions (44.54% of tasks better), which is passed over quickly; the human evaluation was run by the authors and one annotator is an author; and there's no code or data link despite the claim of releasing 45k instructions.

On the credit side, the pipeline is clearly described, the comparison across model sizes and data sizes is useful, and the diversity analysis (verb-noun pairs, ROUGE-L to seed) is a genuine attempt to characterize the generated data. The paper would benefit from re-estimating the reward on a small human-labeled set, adding error bars, and running the reward-filtering control.

Bottom line: worth engaging for researchers working on synthetic instruction data for small open models, but it needs experimental tightening before the RL story can be believed. I'd send it to review, but I'd expect major revisions.

Referee Report

3 major / 6 minor

Summary. The paper introduces REFINE-AF, a three-stage pipeline for task-agnostic instruction dataset generation using small open-source LLMs (LLaMA 2-7B, LLaMA 2-13B, Mistral 7B). Stage 1 bootstraps new instructions from 175 human-written seeds; Stage 2 applies PPO with a reward function composed of an external reward model and UniEval metrics; Stage 3 generates (instruction, input, output) triplets, which are then used for supervised instruction fine-tuning. The authors compare the resulting 15,000-instruction datasets against Self-Instruct baselines on the SUPER-NI benchmark, report average ROUGE-L scores per task, report that REFINE-AF outperforms Self-Instruct on 63--66% of tasks, and include a small human evaluation and an analysis of generated-instruction diversity and quality.

Significance. If the central claim holds, the paper makes a useful practical contribution: it proposes a fully open, relatively low-cost alternative to API-based instruction generation, and it releases a 45K-instruction synthetic dataset, which is a concrete asset for the community. The claimed gains are modest in absolute ROUGE-L terms but would still be meaningful if robust. The paper is also honest in not using SUPER-NI in the reward function, so there is no direct circularity between the training objective and the evaluation target. However, the evidence base is currently too thin to support the causal claim that the RL stage is what drives the reported improvements, and the statistical framing overstates the strength of the results.

major comments (3)

[§V.A, Tables II and III] The central claim of 'significant task improvement' is not supported by the reported statistics. Table II reports single runs with no variance, confidence intervals, or number of seeds, and the task-level improvement percentages are presented without any error bar. Table III underlines values said to have p-value < 0.05, but no statistical test is described, no p-values are reported, and no multiple-comparison correction is mentioned. In addition, the LLaMA 2-13B row at 5,000 instructions shows REFINE-AF worse than Self-Instruct (6.4488 vs. 6.5349), which is inconsistent with the blanket claim that REFINE-AF consistently surpasses the baseline. The authors should report per-run scores, standard errors, and a clearly specified significance test, and should discuss the non-monotonic 13B result.
[§II.C, Eq. (1)] The reward function is an unvalidated proxy, and the paper's causal claim that RL 'leads to further enhancements' rests entirely on it. The coefficients in Eq. (1) are imported from [16] without re-estimation, and no evidence is given that the blended score correlates with human judgments or with downstream instruction-following performance for these three base models. The sign of the understandability term is also confusing: Table I defines understandability positively while Eq. (1) assigns it a large negative weight and Section II.C reinterprets it as 'complexity.' The missing control is a reward-filtering baseline: if the same gains are obtained by simply selecting the top-scoring instances generated by the base model without PPO, the RL loop is not the causal mechanism. Adding such an ablation, or at least a correlation analysis between reward scores and downstream quality, is necessary to support the abstract's claim about RL-based enhancement.
[§V.C and Table II] The claim that 'increasing the number of instructions gradually increases the score' is not robustly supported by the data. For REFINE-AF, the trend is mostly increasing, but for the Self-Instruct baseline on LLaMA 2-13B the scores decrease with instruction count (6.5349, 6.5269, 6.4446), and REFINE-AF at 5K is below its own baseline. The absolute ROUGE-L differences are small (e.g., 6.0414 to 6.1636 for LLaMA 2-7B at 15K), so without confidence intervals or multiple runs the apparent scaling trend could be noise. The authors should either provide error bars or soften the scaling claim.

minor comments (6)

[Abstract and Section I] The phrase 'a substantial improvements' is ungrammatical; it should be 'substantial improvements.'
[Fig. 2 caption] The caption reads 'for LLaMa 13B, LLaMa 13B and Mistral 7B'; the first two entries should presumably be LLaMA 2-7B and LLaMA 2-13B.
[§II.C] No value for the KL scaling factor β in Eq. (2) is reported anywhere, despite β being a free parameter of the method. Please state the value used and, ideally, its sensitivity.
[§IV.A and Table V] Table V reports 17,524 instances for LLaMA 2-13B while the text states that 15,000 instructions were generated and that post-filtering produces about 15,000 instances; the discrepancy should be explained.
[§V.B] The human evaluation is performed by the authors themselves; even if the annotators did not know which method produced each response, author involvement is a conflict of interest. This should be disclosed and, if possible, supplemented by independent annotation.
[Section VI] The related work paragraph says the method 'employs a reinforcement learning algorithm through human feedback,' but the paper uses automated feedback; this is inconsistent with the terminology used elsewhere.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported SUPER-NI gains are measured against an external benchmark that never enters the reward function, training data, or Eq. (1).

full rationale

The central result is an average ROUGE-L comparison on the SUPER-NI evaluation set (Table II) between models instruction-tuned on REFINE-AF data and on Self-Instruct data. The derivation chain is: Stage 1 bootstraps instructions from 175 human seeds using a diversity filter (ROUGE-L < 0.7 against the existing pool); Stage 2 runs PPO with reward r(x,y) from Eq. (1), whose components are the external oasst-rm-pythia-1.4b score and UniEval naturalness/coherence/understandability, with coefficients taken from prior external work [16]; Stage 3 generates (instruction, input, output) triplets with the RL-trained model and performs SFT. The evaluation target, SUPER-NI, never appears in the reward, the training data, or the filtering criterion, so the improvement over the Self-Instruct baseline is not a fitted parameter renamed as a prediction. The self-training loop (a model generating data that is later used to train the same model) is the standard Self-Instruct paradigm and is not circular because the evaluation uses held-out external instances and labels. Concerns that Eq. (1) is an unvalidated proxy for downstream instruction-following are correctness or validity risks, not circularity, since no part of the evaluation metric is fed back into the optimization. The reward-versus-step Spearman correlations are process sanity checks, not the evidence for the SUPER-NI claim. No load-bearing self-citation chain or imported uniqueness theorem is present. The manuscript's data-quality human evaluation uses an author-annotator, but that is a quality-control limitation rather than a circular derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the validity and transferability of an imported reward function, on the base models' generative ability, and on the choice of ROUGE-L as the evaluation metric. None of these are derived in this paper; they are assumptions from prior work or experimental choices.

free parameters (3)

Reward coefficients in Eq. (1) = 0.0078, -0.4421, 0.3212, 0.1520, -0.0274
These coefficients are imported from the Instruction Mining paper [16] and are not re-estimated here. The entire RL feedback signal depends on them, and the paper does not validate their transfer to different base models.
Beta (KL scaling factor) in Eq. (2) = not reported
The KL penalty strength controls the trade-off between reward maximization and staying close to the reference policy. The paper states beta > 0 but never gives the value used, so the exact RL objective cannot be reproduced.
ROUGE-L deduplication threshold = 0.7
New instructions are accepted only if their ROUGE-L similarity to any existing instruction is below 0.7. This threshold is inherited from Self-Instruct [5] and affects the diversity and size of the instruction pool.

assumptions (5)

domain assumption Base models (LLaMA 2-7B, LLaMA 2-13B, Mistral 7B) can follow the generation prompts and produce usable instructions and input-output pairs.
The whole pipeline assumes these small models are capable enough to bootstrap from 175 seed instructions, which is tested only indirectly through downstream performance.
domain assumption The oasst-rm-pythia-1.4b reward model and the UniEval metrics are valid proxies for instruction and response quality.
The RL training optimizes this automated reward, and the paper does not provide evidence that the reward correlates with the downstream SUPER-NI evaluation.
domain assumption Reward coefficients from Instruction Mining [16] transfer to this different model family and generation setup.
The coefficients in Eq. (1) were fit in prior work, and the paper applies them without re-validation or sensitivity analysis.
domain assumption ROUGE-L on SUPER-NI is a meaningful measure of instruction following.
The paper uses ROUGE-L because [5] reported it, but for classification tasks in SUPER-NI the standard evaluation is exact match with normalized labels, not text similarity.
domain assumption PPO training with LoRA and 4-bit quantization on a single A100 GPU is stable enough to yield the reported improvements.
The paper monitors a moving-average reward curve and reports moderate Spearman correlations, but does not analyze failure cases or training variance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback." pith.science (2026). https://pith.science/paper/BYWIXDXP

@misc{pith2026250506548,
  author       = {Pith},
  title        = {Pith review of: REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYWIXDXP}},
  note         = {Machine review of arXiv:2505.06548}
}

read the original abstract

Instruction-based Large Language Models (LLMs) have proven effective in numerous few-shot or zero-shot Natural Language Processing (NLP) tasks. However, creating human-annotated instruction data is time-consuming, expensive, and often limited in quantity and task diversity. Previous research endeavors have attempted to address this challenge by proposing frameworks capable of generating instructions in a semi-automated and task-agnostic manner directly from the model itself. Many of these efforts have relied on large API-only parameter-based models such as GPT-3.5 (175B), which are expensive, and subject to limits on a number of queries. This paper explores the performance of three open-source small LLMs such as LLaMA 2-7B, LLama 2-13B, and Mistral 7B, using a semi-automated framework, thereby reducing human intervention, effort, and cost required to generate an instruction dataset for fine-tuning LLMs. Furthermore, we demonstrate that incorporating a Reinforcement Learning (RL) based training algorithm into this LLMs-based framework leads to further enhancements. Our evaluation of the dataset reveals that these RL-based frameworks achieve a substantial improvements in 63-66% of the tasks compared to previous approaches.

Figures

Figures reproduced from arXiv: 2505.06548 by the authors.

**Figure 2.** Moving average of the model rewards over the training [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Length distributions of the instructions, inputs, and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Distribution of Rouge-L scores between generated [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Similarity of instructions generated using REFINE-AF [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Human Evaluation results using LLaMA 7B, LLaMA [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Effect of data size on the performance of the model. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: The top 20 most common root verbs (inner circle) and their top 4 most common direct nouns (outside circle) in the [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 22 canonical work pages

[16]

Instruction mining: When data mining meets large language model finetuning,

Y . Cao, Y . Kang, C. Wang, and L. Sun, “Instruction mining: When data mining meets large language model finetuning,” 2023

work page 2023
[1]

Large language models are human-level prompt engineers,

Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” 2023

2023
[2]

Guess the instruction! flipped learning makes language models stronger zero-shot learners,

S. Ye, D. Kim, J. Jang, J. Shin, and M. Seo, “Guess the instruction! flipped learning makes language models stronger zero-shot learners,” 2023

work page 2023
[3]

Explaining patterns in data with language models via interpretable autoprompting,

C. Singh, J. X. Morris, J. Aneja, A. M. Rush, and J. Gao, “Explaining patterns in data with language models via interpretable autoprompting,” 2023

work page 2023
[4]

Instruction induction: From few examples to natural language task descriptions,

O. Honovich, U. Shaham, S. R. Bowman, and O. Levy, “Instruction induction: From few examples to natural language task descriptions,” 2022

work page 2022
[5]

Self-instruct: Aligning language models with self- generated instructions,

Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” 2023

work page 2023
[6]

Self-alignment with instruction backtranslation,

X. Li, P. Yu, C. Zhou, T. Schick, O. Levy, L. Zettlemoyer, J. Weston, and M. Lewis, “Self-alignment with instruction backtranslation,” 2024

work page 2024
[7]

(2021) Gpt-3.5 turbo documentation

OpenAI. (2021) Gpt-3.5 turbo documentation. [Online]. Available: https://platform.openai.com/docs/models/gpt-3-5

work page 2021

Show all 32 references

[8]

Llama 2: Open foundation and fine-tuned chat models,

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

2023
[9]

Mistral 7b,

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023

2023
[10]

Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks,

Y . Wang, S. Mishra, P. Alipoormolabashi, Y . Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis, H. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia, K. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Parmar, M. Purohi...

2022
[11]

Openassistant conversations – democra- tizing large language model alignment,

A. K ¨opf, Y . Kilcher, D. von R ¨utte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, S. ES, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick, “Openassistant conversations – democra- tizing large la...

2023
[12]

Towards a unified multi-dimensional evaluator for text generation,

M. Zhong, Y . Liu, D. Yin, Y . Mao, Y . Jiao, P. Liu, C. Zhu, H. Ji, and J. Han, “Towards a unified multi-dimensional evaluator for text generation,” 2022

2022
[13]

Deep reinforcement learning from human preferences,

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” inAdvances in Neural Information Processing Systems, 2017, pp. 4299–4307

2017
[14]

Learning to summarize from human feedback,

N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. Christiano, “Learning to summarize from human feedback,”arXiv preprint arXiv:2009.01325, 2020

2009 arXiv
[15]

Prox- imal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[17]

Training language models to follow instructions with human feedback,

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions...

2022
[18]

LoRA: Low-rank adaptation of large language models,

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

2022
[19]

Stackllama: An rl fine-tuned llama model for stack exchange question and answering,

E. Beeching, Y . Belkada, K. Rasul, L. Tunstall, L. von Werra, N. Rajani, and N. Lambert, “Stackllama: An rl fine-tuned llama model for stack exchange question and answering,” 2023. [Online]. Available: https://huggingface.co/blog/stackllama

2023
[20]

Finetuned language models are zero-shot learners,

J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” 2022

2022
[21]

Cross-task generalization via natural language crowdsourcing instructions,

S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi, “Cross-task generalization via natural language crowdsourcing instructions,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), S. Muresan, P. Nakov, and A. V...

2022
[22]

Multitask prompted training enables zero-shot task generalization,

V . Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. She...

2022
[23]

Scaling instruction-finetuned language models,

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V . Zhao, Y . Huang, A. Dai, H. Y...

2022
[24]

Longform: Effective instruction tuning with reverse instructions,

A. K ¨oksal, T. Schick, A. Korhonen, and H. Sch ¨utze, “Longform: Effective instruction tuning with reverse instructions,” 2024

2024
[25]

Revisiting self-training for neural sequence generation,

J. He, J. Gu, J. Shen, and M. Ranzato, “Revisiting self-training for neural sequence generation,” 2020

2020
[26]

Self-training with noisy student improves imagenet classification,

Q. Xie, M.-T. Luong, E. Hovy, and Q. V . Le, “Self-training with noisy student improves imagenet classification,” 2020

2020
[27]

Self-training improves pre-training for natural language understanding,

J. Du, E. Grave, B. Gunel, V . Chaudhary, O. Celebi, M. Auli, V . Stoyanov, and A. Conneau, “Self-training improves pre-training for natural language understanding,” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis...

2021
[28]

Self-training: A survey,

M.-R. Amini, V . Feofanov, L. Pauletto, L. Hadjadj, E. Devijver, and Y . Maximov, “Self-training: A survey,” 2023

2023
[29]

Large language models can self-improve,

J. Huang, S. S. Gu, L. Hou, Y . Wu, X. Wang, H. Yu, and J. Han, “Large language models can self-improve,” 2022

2022
[30]

Prompt consistency for zero-shot task generalization,

C. Zhou, J. He, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Prompt consistency for zero-shot task generalization,” 2022

2022
[31]

Stanford alpaca: An instruction-following llama model,

R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” https://github.com/tatsu-lab/stanford alpaca, 2023

2023
[5418]

Available: https://aclanthology.org/2021.naacl-main.426

[Online]. Available: https://aclanthology.org/2021.naacl-main.426

2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [16]

Instruction mining: When data mining meets large language model finetuning,

Y . Cao, Y . Kang, C. Wang, and L. Sun, “Instruction mining: When data mining meets large language model finetuning,” 2023

work page 2023

[2] [1]

Large language models are human-level prompt engineers,

Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” 2023

2023

[3] [2]

Guess the instruction! flipped learning makes language models stronger zero-shot learners,

S. Ye, D. Kim, J. Jang, J. Shin, and M. Seo, “Guess the instruction! flipped learning makes language models stronger zero-shot learners,” 2023

work page 2023

[4] [3]

Explaining patterns in data with language models via interpretable autoprompting,

C. Singh, J. X. Morris, J. Aneja, A. M. Rush, and J. Gao, “Explaining patterns in data with language models via interpretable autoprompting,” 2023

work page 2023

[5] [4]

Instruction induction: From few examples to natural language task descriptions,

O. Honovich, U. Shaham, S. R. Bowman, and O. Levy, “Instruction induction: From few examples to natural language task descriptions,” 2022

work page 2022

[6] [5]

Self-instruct: Aligning language models with self- generated instructions,

Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” 2023

work page 2023

[7] [6]

Self-alignment with instruction backtranslation,

X. Li, P. Yu, C. Zhou, T. Schick, O. Levy, L. Zettlemoyer, J. Weston, and M. Lewis, “Self-alignment with instruction backtranslation,” 2024

work page 2024

[8] [7]

(2021) Gpt-3.5 turbo documentation

OpenAI. (2021) Gpt-3.5 turbo documentation. [Online]. Available: https://platform.openai.com/docs/models/gpt-3-5

work page 2021