{"id":"c0512cba-e463-4ee2-9a56-bc6c92d08603","arxiv_id":"2502.04774","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"SeDi-Instruct generates instruction data by relaxing duplicate filtering, sampling cluster-balanced batches, and replacing low-scoring seeds with instructions from high-gradient-norm batches.","lead":"SeDi-Instruct is a new recipe for creating instruction-tuning data for language models. It says it can match or beat the standard Self-Instruct method while cutting expensive API calls by about a third.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 5.2% accuracy gain is not supported by Table 1: the reported averages differ by 0.8 points, and the 5.2% figure appears to be the mean of per-benchmark relative improvements dominated by a tiny AlpacaEval change.","rationale":"The paper's stated contribution is to generate high-quality instruction data at lower cost, with the headline numbers being 5.2% accuracy improvement and 36% cost reduction. The cost claim is at least internally consistent: relaxing the ROUGE-L filtering threshold from 0.7 to 0.85 mechanically retains more generated instructions, and Fig. 8 reports 36% fewer API calls to reach 10,000 kept instructions. The accuracy claim is the fragile pillar. Table 1 directly contradicts a literal reading of the 5.2% claim, and the only way to reproduce 5.2% is by averaging per-benchmark relative improvements, which is a misleading aggregation. The paper provides no code, no data, no error bars, and no significance tests; the repository link is empty. The iterative feedback component is also asserted rather than demonstrated: the choice of gradient norm is justified only by a reference to an \"empirical study\" that is not shown, and no ablation isolates the seed-replacement mechanism from the diversity-based filtering. My reading does not identify a logical inconsistency in the algorithm, but the central empirical claim is not established by the included evidence. This supports the reader's rejection, though my primary concern is the headline number itself rather than the gradient-norm assumption, hence partial agreement.","tokens_in":20108,"tokens_out":6990,"duration_ms":73031,"concrete_test":"Recompute the aggregate improvement from Table 1 in a single consistent unit. First, report the absolute percentage-point difference (46.9 − 46.1 = 0.8) and the per-benchmark relative improvements separately. Then re-run Llama-3-8B + Self-Instruct and Llama-3-8B + SeDi-Instruct with at least three random seeds each and report per-benchmark means with standard deviations. If the MMLU and Hellaswag differences fall within noise, the central accuracy claim is not supported regardless of how the 5.2% was computed. In addition, run a seed-replacement ablation that selects candidate batches at random instead of by highest gradient norm; if the gradient-norm selection does not significantly outperform random selection, the iterative feedback component contributes nothing beyond diversity-based filtering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is not reproducible from the paper's own results. Table 1 reports average accuracy 46.1 for Llama-3-8B + Self-Instruct and 46.9 for Llama-3-8B + SeDi-Instruct, an absolute improvement of 0.8 percentage points. The paper nonetheless claims \"5.2% higher accuracy on average.\" The only way to recover 5.2% from Table 1 is to average per-benchmark relative improvements: AlpacaEval 5.4/4.6−1 = 17.4%, MMLU 56.6/56.5−1 = 0.18%, Hellaswag 56.1/55.7−1 = 0.72%, and ARC 69.3/67.7−1 = 2.36%, which average to about 5.2%. This aggregation is methodologically unsound because it mixes scales and lets a 0.8-point absolute change in AlpacaEval dominate while the MMLU and Hellaswag differences are 0.1 and 0.4 points, respectively, plausibly within run-to-run noise. No error bars, multiple seeds, or significance tests are provided. Thus the headline accuracy claim is either arithmetically inconsistent with the reported aggregate or an unsupported artifact of a chosen aggregation formula. The iterative feedback mechanism adds a second unvalidated assumption, namely that the batch with the highest gradient norm contains the most valuable instructions, but the unsubstantiated 5.2% claim is the more direct threat to the paper's central contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SeDi-Instruct is a data-generation framework for instruction tuning that relaxes the ROUGE-L filtering threshold to retain more generated instructions and uses clustering-based batch construction to preserve local diversity. It additionally introduces an iterative feedback loop: during training, the batch with the highest gradient norm is identified every ten iterations, its instructions are filtered for novelty against the seed set, and low-scoring seed instructions (measured by a SeedKept/SeedGen ratio) are replaced. The authors evaluate Llama-3-8B fine-tuned on 30,164 instructions generated with SeDi-Instruct against Self-Instruct and several off-the-shelf instruct models on AlpacaEval, MMLU, Hellaswag, and ARC, and claim a 5.2% accuracy improvement and a 36% reduction in API cost. They also report model-collapse and safety analyses.","tokens_in":20478,"tokens_out":6168,"duration_ms":60554,"significance":"The paper tackles a practical and timely problem: reducing the API cost of self-instructed data generation while maintaining or improving quality. Its cost analysis (Fig. 8) and its explicit check of model collapse with a smaller generator (Table 2) are valuable and give the reader concrete efficiency numbers. The diversity-based filtering idea is plausible and the framework is simple enough to build on. However, the headline accuracy gain is not reproducible from the reported aggregate, the gains on two of the four benchmarks are negligible without uncertainty quantification, and the iterative feedback mechanism—the main novelty—has no ablation or direct evidence. As it stands, the empirical case for the method is not established.","major_comments":[{"comment":"The headline '5.2% higher accuracy on average' is not supported by the aggregate in Table 1. Table 1 reports average accuracy 46.1 for Self-Instruct and 46.9 for SeDi-Instruct, an absolute gain of 0.8 points (approximately 1.7% relative). The 5.2% figure is recovered only by averaging per-benchmark relative improvements: AlpacaEval (5.4/4.6 - 1 = 17.4%), MMLU (56.6/56.5 - 1 = 0.18%), Hellaswag (56.1/55.7 - 1 = 0.72%), and ARC (69.3/67.7 - 1 = 2.36%), which average to about 5.2%. Such averaging mixes incomparable metrics and lets the AlpacaEval win-rate change dominate. Please report absolute average accuracy (or a clearly defined primary metric) with uncertainty.","section":"Evaluation, Table 1"},{"comment":"No error bars, multiple seeds, or significance tests are reported. The MMLU difference (56.6 vs 56.5) and Hellaswag difference (56.1 vs 55.7) are 0.1 and 0.4 points, respectively, and are plausibly within run-to-run variation; the claim that SeDi-Instruct broadly 'outperforms' the Self-Instruct baseline needs statistical support or an explicit statement that these are single-run results.","section":"Table 1"},{"comment":"The core novelty is the selection of seed instructions from batches with the highest gradient norm during training, but the paper provides no evidence for this criterion. The sentence 'Based on our empirical study, we decide to use the gradient norm' refers to an unpublished study, and no ablation separates the iterative feedback component from the diversity-based filtering component. Table 1 only compares the full pipeline against Self-Instruct, so the contribution of the feedback loop is unsubstantiated.","section":"Iterative Feedback Task Generation, Section 4.2"},{"comment":"The candidate-seed selection rule is internally inconsistent. The text states that the algorithm includes instructions 'whose ROUGE-L similarity s_ij exceeds theta_keep = 0.7', yet the goal is to find instructions 'not similar to those in S', which requires keeping instructions with similarity below the threshold (as in Algorithm 1's rejection rule). Please correct the pseudocode or the prose.","section":"Algorithm 1, Section 4.2"}],"minor_comments":[{"comment":"Table 3 does not specify several hyperparameters needed for reproduction: the clustering/classifier details, the PCA dimensionality (log(batch size) is mentioned in the text but not the implementation), the number of feedback iterations, the seed replacement frequency, and the details of the 'empirical study' that selected gradient norm. Please add these.","section":"Appendix, Table 3"},{"comment":"The text says 'SeDi-Instruct achieves 1.6x reduction in the cost' and later 'reducing expenses by up to 36%'; these figures are consistent (1/1.6 = 0.625), but the relation should be stated explicitly to avoid confusion.","section":"Cost Analysis, Section 6.3"},{"comment":"The MMLU humanities subgroup shows SeDi-Instruct (49.8) below Self-Instruct (51.0); the statement 'does not fall behind in overall performance' is acceptable, but please mention subgroup differences.","section":"Appendix, Table 5"},{"comment":"The table title contains a typo: 'hyterparameter' should be 'hyperparameter'.","section":"Appendix, Table 3"},{"comment":"The code link is a placeholder ('https://github.com/'), which prevents reproducibility checks. Please provide a working repository or state that code will be released after publication.","section":"Evaluation, Training Recipe"},{"comment":"The safety section candidly shows that SeDi-Instruct-trained models answer harmful queries without refusal, and the proposed prompt-based mitigation is only a case study; this important limitation should be highlighted in the abstract or conclusion rather than only in a short subsection.","section":"Safety"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an early draft: the code link is a placeholder, an 'empirical study' supporting the gradient-norm choice is referenced but not shown, and the central 5.2% claim is an artifact of averaging per-benchmark relative improvements. If the editor is considering a full review, I would request a revised version with corrected claims, multiple seeds, and an ablation of the feedback loop before external review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper combines two known ideas—relaxed ROUGE-L filtering with cluster-balanced batching, and gradient-norm-based seed replacement—into a pipeline that plausibly cuts API costs by roughly a third. The cost result is believable. The accuracy claim is not. The abstract and conclusion say \"5.2% higher accuracy on average,\" but Table 1 shows absolute averages of 46.1 vs 46.9 for Self-Instruct vs SeDi-Instruct, a 0.8-point difference. The 5.2% is the mean of per-benchmark relative gains, dominated by AlpacaEval moving from 4.6 to 5.4 win-rate; MMLU and Hellaswag go up 0.1 and 0.4 points, which is noise. No error bars, no multiple seeds, no significance tests. That is the load-bearing flaw.\n\nWhat is genuinely new: the specific combination does not appear in prior work, and the pipeline is clearly described with algorithms and figures. The authors also test model collapse and safety, which is more than many data-generation papers do. The cost analysis (36% fewer API calls to produce 10k kept instructions) is useful and consistent, and the 1.6× cost reduction roughly matches the abstract's 36%.\n\nThe soft spot beyond the headline number is the gradient-norm mechanism. The paper says \"Based on our empirical study, we decide to use the gradient norm (GN) because it is the most suitable one,\" but the study is not shown and there is no ablation. If gradient norm does not track instruction quality, the seed-replacement loop, which is the main novel component, has no demonstrated effect. Also, the code link is just a placeholder, so reproducibility is limited.\n\nThis is not a bad paper to send to review. The idea is worth exploring and the cost savings are plausible, but the evidence as presented does not support the central claim. A serious referee should ask for a corrected accuracy number, error bars or multiple seeds, and an ablation of the gradient-norm selection. The authors also need to release the code.\n\nBottom line: I would not cite this in its current form, but I would bring it to a reading group that works on instruction data generation, and it deserves peer review with major revision.","headline":"A reasonable combination of known ideas for cheaper instruction generation, but the headline 5.2% accuracy gain is an artifact of averaging per-benchmark relative gains and is not supported by the paper's own Table 1.","tokens_in":20995,"tokens_out":2192,"would_cite":false,"duration_ms":23016,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-directed instruction generation framework claims to improve instruction-tuned model accuracy by 5.2% while cutting the API cost of dataset construction by 36%.","keywords":["instruction tuning","self-instruct","data generation","diversity-based filtering","iterative feedback","gradient norm","seed replacement","cost efficiency"],"falsifier":"Run SeDi-Instruct's training loop while scrambling the batch selection rule, for example replacing the highest-gradient-norm batch with a randomly chosen batch or with the lowest-gradient-norm batch, and compare final benchmark accuracy. If the randomly replenished model matches or beats the gradient-norm model, the feedback mechanism is not doing the claimed work; the paper does not report this control. A simpler check is to verify whether instructions added from high-gradient-norm batches actually raise the kept-instruction quality of later generations, measured by the SeedKept/SeedGen ratio over successive rounds.","tokens_in":19898,"feed_emoji":"🧠","tokens_out":7125,"duration_ms":65852,"temperature":0.7,"pith_summary":"This paper proposes SeDi-Instruct, a data-generation framework that creates instruction-tuning data for large language models at lower cost and higher quality than the Self-Instruct baseline. It claims that loosening the similarity filter (raising the ROUGE-L threshold from 0.7 to 0.85) while balancing each training batch across instruction clusters lets a model keep more generated instructions without losing accuracy. It also claims that ranking training batches by gradient norm and feeding instructions from the best batches back into the seed set improves the quality of later generations. Evaluated on Llama-3-8B, the method reports an average accuracy gain of 5.2% over the Self-Instruct baseline on AlpacaEval, MMLU, HellaSwag, and ARC, while needing about 36% fewer API calls to produce 10,000 kept instructions.","feed_headline":"Instruction data pipeline cuts cost 36% and lifts accuracy 5.2%","feed_subtitle":"SeDi-Instruct keeps more generated instructions and reuses training signals to improve the seeds that synthesize them.","key_machinery":"The machinery is a two-part pipeline: diversity-based filtering and iterative feedback task generation. Diversity-based filtering uses a ROUGE-L threshold of 0.85 instead of Self-Instruct's 0.7, so only highly redundant instructions are dropped; accepted instructions are classified via PCA-reduced embeddings into as many clusters as the batch size, and each batch samples one instruction per cluster. Iterative feedback task generation tracks batch gradient norms during training, selects the highest-norm batch every ten iterations, and uses those instructions to replenish the seed set while a seed score table (the ratio SeedKept/SeedGen) identifies which seeds to evict.","core_discovery":"The central claim is that instruction-tuning data can be synthesized more cheaply and effectively by closing the loop between generation and training. SeDi-Instruct replaces Self-Instruct's static, aggressive filtering with two mechanisms: a relaxed similarity threshold that retains up to 80% of generated instructions, and a clustering-based batch sampler that groups similar instructions and draws one per cluster so each batch remains diverse. It also monitors training quality and, every ten iterations, selects the batch with the highest gradient norm, adds any of its instructions not already similar to the seed set, and evicts seed instructions with low SeedKept/SeedGen ratios. The paper reports that a Llama-3-8B model tuned on 30,164 instructions produced this way outperforms the equivalent Self-Instruct model on four benchmarks and on all five competitive evaluation test sets, and that generating 10,000 kept instructions costs 36% less because fewer API calls are wasted on discarded data.","pith_inferences":["A random-batch control would separate the contribution of the gradient-norm selector from the contribution of simply adding more diverse instructions to the seed set; the reported experiments do not include this control.","The seed score (SeedKept/SeedGen) rewards seeds that generate many retained instructions, but it may also favor generic seeds that produce a large volume of redundant variants; the paper does not test whether a diversity-aware score works differently.","If the cost saving scales with the relaxed threshold, even looser thresholds (with stronger clustering) could push API waste lower, but the accuracy tradeoff beyond 0.85 is not measured.","The model-collapse experiments suggest the method depends on using a generator larger than the target model; swapping the generator for a smaller one degraded scores in the reported re-evaluation."],"forward_implications":["If the reported numbers hold, instruction-tuning pipelines can cut API spend by about a third without sacrificing model quality.","The results support loosening similarity-based filtering (threshold 0.7 to 0.85) whenever batch construction enforces local diversity.","Training signals such as gradient norms can serve as a no-human-in-the-loop signal for which synthetic instructions to reuse as seeds.","The pipeline reproduces its advantage on five competitive evaluation sets, not just on accuracy benchmarks, suggesting the effect is not benchmark-specific.","The method carries a known safety cost: without a refusal system prompt, the tuned model answers harmful queries, so real deployments would need an alignment guard."],"supporting_citations":[{"why":"Supplies the Self-Instruct baseline framework and the initial seed instructions that SeDi-Instruct starts from.","marker":"Wang et al. 2023"},{"why":"AlpaGasus shows that a 9K high-quality subset of Alpaca 52K can match the full dataset, motivating quality-oriented filtering.","marker":"Chen et al. 2024"},{"why":"Alpaca exemplifies the low-quality instruction data that Self-Instruct-style synthesis can produce.","marker":"Taori et al. 2023"},{"why":"Provides the Llama-3-8B base model and the Llama-3-8B-Instruct comparison model.","marker":"Dubey et al. 2024"},{"why":"Defines the AlpacaEval win-rate benchmark used for the alignment evaluation.","marker":"Dubois et al. 2024"},{"why":"Provides the MMLU benchmark used in the accuracy comparison.","marker":"Hendrycks et al. 2021"},{"why":"Supports the model-collapse discussion and the claim that filtering most of a synthesized dataset can improve results.","marker":"Feng et al. 2024"},{"why":"Supports the claim that a larger instruction generation model yields more diverse instructions.","marker":"Zhang et al. 2023"}],"fun_headline_variants":["Closed-loop instruction tuning cuts data cost 36% and boosts accuracy 5.2%","Cost 36% lower and 5.2% more accurate with self-directed instruction","Reuse training signals to cut data cost 36% and lift accuracy 5.2%","Looped instruction generation: 36% cheaper data, 5.2% better models","Keep more instruction data: cut cost 36%, gain accuracy 5.2%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training batch with the highest gradient norm contains the most valuable instructions for future generation; if gradient norm does not track instruction quality, the seed-replacement loop that forms the paper's main novelty has no effect.","fun_headline_variants_meta":{"raw":{"variants":["Closed-loop instruction tuning cuts data cost 36% and boosts accuracy 5.2%","Cost 36% lower and 5.2% more accurate with self-directed instruction","Reuse training signals to cut data cost 36% and lift accuracy 5.2%","Looped instruction generation: 36% cheaper data, 5.2% better models","Keep more instruction data: cut cost 36%, gain accuracy 5.2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":3047,"prompt_tokens":956,"completion_tokens":2091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1976}},"tokens_in":572,"tokens_out":2091,"duration_ms":12977,"temperature":1.0,"reasoning_tokens":1976,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:31:07.421587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SeDi-Instruct's training loop while scrambling the batch selection rule, for example replacing the highest-gradient-norm batch with a randomly chosen batch or with the lowest-gradient-norm batch, and compare final benchmark accuracy. If the randomly replenished model matches or beats the gradient-norm model, the feedback mechanism is not doing the claimed work; the paper does not report this control. A simpler check is to verify whether instructions added from high-gradient-norm batches actually raise the kept-instruction quality of later generations, measured by the SeedKept/SeedGen ratio over successive rounds.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AlpaGasus shows that a 9K high-quality subset of Alpaca 52K can match the full dataset, motivating quality-oriented filtering."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Alpaca exemplifies the low-quality instruction data that Self-Instruct-style synthesis can produce."}],"review_version":1}