{"id":"0802e9fd-249e-4696-807c-57ab9343ec7f","arxiv_id":"2505.06548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"REFINE-AF generates instruction-tuning data with small open LLMs and adds reinforcement learning from automated feedback, beating a Self-Instruct baseline on 63-66% of SUPER-NI tasks at 15k instructions.","lead":"The authors present REFINE-AF, a pipeline that lets small open-source language models generate their own instruction-following training data and then uses reinforcement learning from automatic quality scores to improve that data. A team could use it to fine-tune a small model without paying for large API models or human annotation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) reward is an unvalidated proxy whose 'understandability' term is used with a large negative coefficient; without a reward-filtering ablation the RL mechanism behind the gains is unsupported.","rationale":"The reader's weakest assumption is the validity/transfer of the Eq. (1) reward. My stress-test agrees and sharpens the point with two observations: (i) the large negative coefficient on UniEval 'understandability' contradicts Table I's own definition and is only glossed as 'complexity' in Section II.C, so the reward signal is internally questionable; (ii) the experiment as reported has no comparison against reward-based selection, so the RL mechanism is not isolated. These issues do not prove the headline performance numbers are fabricated or wrong; the tabulated comparisons and human evaluation point in the same direction. However, they make the causal and generalizability claims conditional: the reported 63-66% task-level improvements could survive even if the reward proxy is poor, so the paper should either validate the reward or add the filtering ablation before the RL-alignment conclusion is accepted. This does not change the reader's CONDITIONAL verdict.","tokens_in":13585,"tokens_out":7978,"duration_ms":89228,"concrete_test":"For each of the three base models, keep the Stage 1 instruction pool fixed; use the untrained base model (not the PPO policy) to generate instances for the same 15k instructions; score every instance with Eq. (1); train the SFT model on the top 15k instances by reward, using the paper's SFT hyperparameters; evaluate zero-shot on SUPER-NI. If this reward-filtered SFT matches or exceeds REFINE-AF's Table II scores, the RL stage is not the causal source of the gains, and the central claim about RL-driven improvement would need to be withdrawn or restated as a selection effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that REFINE-AF's RL stage causes the reported SUPER-NI gains rests entirely on Eq. (1). The coefficients are imported from [16] without re-estimation, and the reward combines oasst-rm-pythia-1.4b with UniEval naturalness/coherence/understandability. Table I defines understandability as 'whether the response is understandable,' yet Eq. (1) gives it a -0.4421 weight, and Section II.C reinterprets it as 'complexity.' No evidence is provided that this blended score correlates with human quality or with held-out instruction following for LLaMA 2-7B/13B or Mistral 7B. Thus the paper cannot distinguish the hypothesized 'RL alignment improves instance quality' from an artifact of extra PPO/LoRA training, reward overfitting, or selection of a narrow output style. The missing control is a reward-filtering baseline: if simply selecting the top-scoring instances from the base model reproduces the gains, the RL loop is not the causal mechanism. This directly bears on the abstract's claim that 'incorporating RL ... leads to further enhancements.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces REFINE-AF, a three-stage pipeline for task-agnostic instruction dataset generation using small open-source LLMs (LLaMA 2-7B, LLaMA 2-13B, Mistral 7B). Stage 1 bootstraps new instructions from 175 human-written seeds; Stage 2 applies PPO with a reward function composed of an external reward model and UniEval metrics; Stage 3 generates (instruction, input, output) triplets, which are then used for supervised instruction fine-tuning. The authors compare the resulting 15,000-instruction datasets against Self-Instruct baselines on the SUPER-NI benchmark, report average ROUGE-L scores per task, report that REFINE-AF outperforms Self-Instruct on 63--66% of tasks, and include a small human evaluation and an analysis of generated-instruction diversity and quality.","tokens_in":13823,"tokens_out":2868,"duration_ms":30794,"significance":"If the central claim holds, the paper makes a useful practical contribution: it proposes a fully open, relatively low-cost alternative to API-based instruction generation, and it releases a 45K-instruction synthetic dataset, which is a concrete asset for the community. The claimed gains are modest in absolute ROUGE-L terms but would still be meaningful if robust. The paper is also honest in not using SUPER-NI in the reward function, so there is no direct circularity between the training objective and the evaluation target. However, the evidence base is currently too thin to support the causal claim that the RL stage is what drives the reported improvements, and the statistical framing overstates the strength of the results.","major_comments":[{"comment":"The central claim of 'significant task improvement' is not supported by the reported statistics. Table II reports single runs with no variance, confidence intervals, or number of seeds, and the task-level improvement percentages are presented without any error bar. Table III underlines values said to have p-value < 0.05, but no statistical test is described, no p-values are reported, and no multiple-comparison correction is mentioned. In addition, the LLaMA 2-13B row at 5,000 instructions shows REFINE-AF worse than Self-Instruct (6.4488 vs. 6.5349), which is inconsistent with the blanket claim that REFINE-AF consistently surpasses the baseline. The authors should report per-run scores, standard errors, and a clearly specified significance test, and should discuss the non-monotonic 13B result.","section":"§V.A, Tables II and III"},{"comment":"The reward function is an unvalidated proxy, and the paper's causal claim that RL 'leads to further enhancements' rests entirely on it. The coefficients in Eq. (1) are imported from [16] without re-estimation, and no evidence is given that the blended score correlates with human judgments or with downstream instruction-following performance for these three base models. The sign of the understandability term is also confusing: Table I defines understandability positively while Eq. (1) assigns it a large negative weight and Section II.C reinterprets it as 'complexity.' The missing control is a reward-filtering baseline: if the same gains are obtained by simply selecting the top-scoring instances generated by the base model without PPO, the RL loop is not the causal mechanism. Adding such an ablation, or at least a correlation analysis between reward scores and downstream quality, is necessary to support the abstract's claim about RL-based enhancement.","section":"§II.C, Eq. (1)"},{"comment":"The claim that 'increasing the number of instructions gradually increases the score' is not robustly supported by the data. For REFINE-AF, the trend is mostly increasing, but for the Self-Instruct baseline on LLaMA 2-13B the scores decrease with instruction count (6.5349, 6.5269, 6.4446), and REFINE-AF at 5K is below its own baseline. The absolute ROUGE-L differences are small (e.g., 6.0414 to 6.1636 for LLaMA 2-7B at 15K), so without confidence intervals or multiple runs the apparent scaling trend could be noise. The authors should either provide error bars or soften the scaling claim.","section":"§V.C and Table II"}],"minor_comments":[{"comment":"The phrase 'a substantial improvements' is ungrammatical; it should be 'substantial improvements.'","section":"Abstract and Section I"},{"comment":"The caption reads 'for LLaMa 13B, LLaMa 13B and Mistral 7B'; the first two entries should presumably be LLaMA 2-7B and LLaMA 2-13B.","section":"Fig. 2 caption"},{"comment":"No value for the KL scaling factor β in Eq. (2) is reported anywhere, despite β being a free parameter of the method. Please state the value used and, ideally, its sensitivity.","section":"§II.C"},{"comment":"Table V reports 17,524 instances for LLaMA 2-13B while the text states that 15,000 instructions were generated and that post-filtering produces about 15,000 instances; the discrepancy should be explained.","section":"§IV.A and Table V"},{"comment":"The human evaluation is performed by the authors themselves; even if the annotators did not know which method produced each response, author involvement is a conflict of interest. This should be disclosed and, if possible, supplemented by independent annotation.","section":"§V.B"},{"comment":"The related work paragraph says the method 'employs a reinforcement learning algorithm through human feedback,' but the paper uses automated feedback; this is inconsistent with the terminology used elsewhere.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is a practical pipeline and a released dataset, and the core idea is reasonable. The main risk is that the RL component is not actually identified as the cause of the gains, and the statistical evidence is thin. I would like the editor to weigh whether the journal's standards require the reward-filtering ablation; without it, the paper would still be acceptable as an empirical pipeline description, but not as a demonstration that RLAF is the effective ingredient. The human evaluation by authors is a secondary concern that should be disclosed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2505.06548. The paper tries something genuinely new: applying PPO with an automated reward to the instance-generation stage of a Self-Instruct-style pipeline, using small open models (LLaMA 2-7B/13B, Mistral 7B). That combination isn't in the cited literature. The paper does a decent job of showing that the resulting 15k-instruction datasets lead to slightly better zero-shot ROUGE-L on SUPER-NI than the same budget of Self-Instruct data, across three base models and three data sizes. The gains are real in the narrow sense that the tables show them, but they're tiny—around 0.1 to 0.2 ROUGE-L points on average—and there are no error bars or multiple seeds, so I wouldn't take the exact numbers as stable.\n\nThe main soft spot is the reward. Eq. (1) is imported from Instruction Mining [16] and not re-estimated or validated on any held-out measure of instruction quality. The coefficient on 'understandability' is negative (-0.4421), and while Section II.C says that term represents 'complexity,' Table I defines it as 'whether the response is understandable.' That's a confusing sign flip that needs clarification. More importantly, nothing in the paper shows that this blended reward correlates with what we care about (instruction following on held-out tasks) or that the RL stage is what causes the gains. A reward-filtering baseline—just take the base model's generated instances, score them with Eq. (1), and train on the top-scoring ones—would directly test whether the PPO/LoRA training is necessary. The paper doesn't include it, so the central claim that 'incorporating RL leads to further enhancements' is not actually supported; the data only shows that the REFINE-AF dataset, trained via RL, does better than Self-Instruct.\n\nThere are also smaller issues: the p-value claim in Table III has no test described; the 13B model is worse than baseline at 5k instructions (44.54% of tasks better), which is passed over quickly; the human evaluation was run by the authors and one annotator is an author; and there's no code or data link despite the claim of releasing 45k instructions.\n\nOn the credit side, the pipeline is clearly described, the comparison across model sizes and data sizes is useful, and the diversity analysis (verb-noun pairs, ROUGE-L to seed) is a genuine attempt to characterize the generated data. The paper would benefit from re-estimating the reward on a small human-labeled set, adding error bars, and running the reward-filtering control.\n\nBottom line: worth engaging for researchers working on synthetic instruction data for small open models, but it needs experimental tightening before the RL story can be believed. I'd send it to review, but I'd expect major revisions.","headline":"A plausible but under-evidenced claim that RL from automated feedback improves self-generated instruction data; the gains are small and the mechanism is untested.","tokens_in":14377,"tokens_out":2577,"would_cite":false,"duration_ms":23873,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small open language models can self-generate instruction data and, after reinforcement learning from automated feedback, beat a Self-Instruct baseline on 63-66% of SUPER-NI tasks.","keywords":["reinforcement learning from automated feedback","instruction tuning","self-generated instructions","LLaMA 2","Mistral 7B","zero-shot generalization","synthetic instruction dataset","SUPER-NI"],"falsifier":"Train the same three base models with the REFINE-AF pipeline but replace the Eq. (1) reward with a constant or randomly shuffled score during the PPO stage; if the resulting models still match the reported SUPER-NI ROUGE-L gains over the baseline, the automated-feedback reward is not the cause of the improvement. A complementary check is to compute the correlation between the Eq. (1) reward and per-task SUPER-NI ROUGE-L scores: a near-zero or negative correlation would indicate the reward is not steering the generator toward useful instances.","tokens_in":13380,"feed_emoji":"🤖","tokens_out":10493,"duration_ms":91344,"temperature":0.7,"pith_summary":"The paper sets out to show that small, open-weight language models can create their own instruction-tuning data and improve their zero-shot instruction following, without relying on large closed API models. Its pipeline, REFINE-AF, starts from 175 human-written seed instructions, bootstraps new task instructions from the base model, applies reinforcement learning with an automated reward score to improve the generated input-output pairs, and then fine-tunes the base model on roughly 15,000 resulting instruction triplets. Across LLaMA 2-7B, LLaMA 2-13B, and Mistral 7B, the fine-tuned models outperform a same-backbone Self-Instruct baseline on 63.51-66.39% of the 119 SUPER-NI tasks, and human ratings on user-oriented instructions also favor the new pipeline. If the result holds, instruction tuning becomes cheaper and more self-contained: alignment data can be synthesized and quality-controlled automatically at the 7B-13B scale.","feed_headline":"Reward feedback boosts self-trained small models on 63-66% of tasks","feed_subtitle":"No large API models needed: LLaMA 2 and Mistral 7B generate 15K instructions and tune themselves.","key_machinery":"The central mechanism is the automated reward signal of Eq. (1), a linear combination of four indicators: the reward-model score $Rew(x,y)$ from oasst-rm-pythia-1.4b and the UniEval dialogue-evaluator dimensions $Nat(x,y)$, $Coh(x,y)$, and $Und(x,y)$, with fixed coefficients imported from an earlier instruction-mining study. This scalar score enters the PPO objective $R(x,y)=r(x,y)-\\beta\\log(\\pi^{\\text{RL}}_\\phi(y|x)/\\pi_0(y|x))$, where the second term penalizes KL divergence from the reference policy. The same reward steers the model that later writes the (input, output) instances in the final fine-tuning dataset, so the automated feedback loop is what the paper credits for improving downstream zero-shot instruction following.","core_discovery":"REFINE-AF is a three-stage, task-agnostic instruction-generation pipeline. Stage 1 expands a pool of 175 human-written seed instructions by prompting the base model with eight in-context instructions, filtering candidates by ROUGE-L similarity below 0.7 and removing prompts that ask for images, pictures, or graphs. Stage 2 trains the same model with Proximal Policy Optimization to maximize a scalar reward composed of the oasst-rm-pythia-1.4b reward-model score plus UniEval naturalness, coherence, and understandability scores, with coefficients taken from an earlier instruction-mining study, and with a KL-divergence penalty against the original policy. Stage 3 uses the RL-trained model to generate input-output instances for each instruction, and the resulting dataset is used for supervised fine-tuning. The central discovery is that this reinforcement-learning-from-automated-feedback step makes the self-generated dataset better for instruction following: the fine-tuned models beat the Self-Instruct baseline on 63.51-66.39% of SUPER-NI tasks, with the absolute ROUGE-L score rising as the number of instructions grows from 5,000 to 15,000.","pith_inferences":["The paper changes several variables at once relative to its baseline (instruction-generation prompt, RL training, instance-generation policy), so a component-wise ablation is needed to know whether the automated-feedback reward is the active ingredient rather than, say, the updated prompt template.","Because the Eq. (1) reward coefficients were imported from prior work and never validated against SUPER-NI or human preferences on the generated data, a direct test would correlate per-instance reward with downstream task accuracy; the paper does not report such a correlation.","If the RL-feedback loop is indeed the driver, an iterative extension suggests itself: fine-tune the model, use it to generate a fresh instruction pool, apply the same automated reward to filter or refine, and repeat; the scaling trend reported here hints that such a loop could compound the gains."],"forward_implications":["Instruction fine-tuning of small open models can proceed with as few as 175 human-written seed instructions and no calls to large commercial APIs, reducing cost and rate-limit constraints.","Because REFINE-AF's absolute SUPER-NI scores rise from 5,000 to 10,000 to 15,000 generated instructions, generating larger datasets with the same pipeline should continue to help within the tested range.","The released 45,000-instruction synthetic dataset, generated entirely by open models, provides a reproducible resource for instruction-tuning experiments without an API budget.","Positive human ratings on 252 user-oriented instructions suggest the gain is not limited to existing NLP benchmark tasks but extends to open-ended user requests."],"supporting_citations":[{"why":"Supplies the seed-instruction bootstrapping method, the ROUGE-L diversity filter, and the Self-Instruct baseline that REFINE-AF is compared against.","marker":"[5]"},{"why":"Supplies the linear reward function and the fixed coefficients used in Eq. (1) for automated feedback.","marker":"[16]"},{"why":"Provides the oasst-rm-pythia-1.4b reward-model score used as one component of the automated reward.","marker":"[11]"},{"why":"Provides the UniEval naturalness, coherence, and understandability scores used in the reward.","marker":"[12]"},{"why":"Provides the Proximal Policy Optimization algorithm used to maximize the reward with a KL penalty.","marker":"[15]"},{"why":"Supplies the reference-model KL-divergence objective that shapes Eq. (2).","marker":"[17]"},{"why":"Defines the 119-task SUPER-NI evaluation set used to measure zero-shot instruction following.","marker":"[10]"},{"why":"Are the LLaMA 2-7B and LLaMA 2-13B base models used for generation and fine-tuning.","marker":"[8]"},{"why":"Is the Mistral 7B base model used for generation and fine-tuning.","marker":"[9]"}],"fun_headline_variants":["Self-taught AI: small models improve via RL on own instructions","REFINE-AF: small LLMs teach themselves, beat baselines on 64% tasks","No big APIs needed: RL makes self-generated data better for LLMs","Small open-source models outperform with self-instruction and RL","Reinforcement learning from automated feedback sharpens small LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the assumption that the imported automated reward score, a weighted mixture of a preference-model score and three text-quality measures, really reflects how useful an instruction-input-output triplet is for training, even though the paper never checks that score against the SUPER-NI benchmark or against human judgment of the generated data.","fun_headline_variants_meta":{"raw":{"variants":["Self-taught AI: small models improve via RL on own instructions","REFINE-AF: small LLMs teach themselves, beat baselines on 64% tasks","No big APIs needed: RL makes self-generated data better for LLMs","Small open-source models outperform with self-instruction and RL","Reinforcement learning from automated feedback sharpens small LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1661,"prompt_tokens":1028,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":538}},"tokens_in":644,"tokens_out":633,"duration_ms":5982,"temperature":1.0,"reasoning_tokens":538,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:39:28.594016+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same three base models with the REFINE-AF pipeline but replace the Eq. (1) reward with a constant or randomly shuffled score during the PPO stage; if the resulting models still match the reported SUPER-NI ROUGE-L gains over the baseline, the automated-feedback reward is not the cause of the improvement. A complementary check is to compute the correlation between the Eq. (1) reward and per-task SUPER-NI ROUGE-L scores: a near-zero or negative correlation would indicate the reward is not steering the generator toward useful instances.","supporting_citations":[{"cited_title":"Self-instruct: Aligning language models with self- generated instructions,","cited_arxiv_id":null,"evidence_quote":"Supplies the seed-instruction bootstrapping method, the ROUGE-L diversity filter, and the Self-Instruct baseline that REFINE-AF is compared against."},{"cited_title":"Instruction mining: When data mining meets large language model finetuning,","cited_arxiv_id":null,"evidence_quote":"Supplies the linear reward function and the fixed coefficients used in Eq. (1) for automated feedback."},{"cited_title":"Openassistant conversations – democra- tizing large language model alignment,","cited_arxiv_id":null,"evidence_quote":"Provides the oasst-rm-pythia-1.4b reward-model score used as one component of the automated reward."},{"cited_title":"Towards a unified multi-dimensional evaluator for text generation,","cited_arxiv_id":null,"evidence_quote":"Provides the UniEval naturalness, coherence, and understandability scores used in the reward."},{"cited_title":"Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks,","cited_arxiv_id":null,"evidence_quote":"Defines the 119-task SUPER-NI evaluation set used to measure zero-shot instruction following."}],"review_version":1}