{"id":"58cc5096-3050-4c8a-8fb4-4543e2929326","arxiv_id":"2509.10708","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SearchInstruct expands seed questions with an LLM, retrieves relevant documents, and grounds answers in them, improving domain-specific SFT and enabling targeted model updates.","lead":"SearchInstruct creates training data for language models by turning a handful of expert-written questions into many new questions, then looking up real documents to write grounded answers. Fine-tuning with this data beat the previous MATINA models in blind human evaluation on Persian cuisine and tourism about two-thirds of the time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 2 does not isolate SearchInstruct's retrieval mechanism: 'with' models get extra SFT data and targeted iterative data, so win rates don't prove retrieval-grounded answer generation causes the gains.","rationale":"The paper's stated aim is to show that SearchInstruct's pipeline—query expansion plus retrieval-grounded answer generation—produces SFT datasets that improve domain performance. The only direct evidence is the before/after comparison in Figure 2. For that claim to hold, the improvement must be attributable to the SearchInstruct construction method, not merely to the addition of more training data or to targeting the exact weaknesses later probed. The text shows that the augmented models received substantially more SFT data (Table 1) created through multiple iterative feedback rounds (Figure 3), and that the evaluation benchmark was designed by annotators who were not involved in seed generation but who evaluated 'previously weak areas.' No control condition adds comparable data via an alternative method, so the observed win rates are compatible with a simple data-quantity effect or even with query expansion alone. This is the most load-bearing concern because it undercuts the causal reading of the central claim. The model editing application, while acknowledged as shallow by the authors, also does not help: Table 2 shows uniform MMLU declines without significance testing. I therefore agree with the reader's weakest_assumption. The verdict should remain CONDITIONAL, pending a controlled ablation that isolates retrieval from data quantity and benchmarking effects.","tokens_in":24430,"tokens_out":4946,"duration_ms":64007,"concrete_test":"Construct a control SFT set of the same size as the SearchInstruct augmentation (e.g., 8,932 culinary samples) by applying only the query-expansion stage of SearchInstruct (same seeds, same expansion prompt) but generating answers with the same strong LLM without retrieved contexts (parametric-only); optionally, a second control using the original Evol-Instruct/document-QA pipeline. Fine-tune the pre-augmentation MATINA model on original data + control data with identical hyperparameters and training order, then rerun the same 100-question blind human evaluation. If either control model matches the Figure 2 win rates, the improvement cannot be attributed to retrieval-grounded answer construction; if both lose clearly (with reported confidence intervals), the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical support for SearchInstruct is the blind head-to-head in §4.1.1/Figure 2, where models augmented with SearchInstruct data win 62–68% against 'one trained prior to SearchInstruct augmentation.' This comparison does not control for the amount or source of additional SFT data, training recency, or the iterative targeting of the evaluation benchmark. The 'with' condition is trained on the original MATINA data plus 7,560–8,932 new pairs (Table 1), generated in three feedback-driven stages (§3.2: 'After initial training, we identify weaknesses in specific subdomains... Additional seeds are then generated... to target these weaknesses'); the 'without' condition receives none of this data. Any additional SFT data—even non-retrieval data aimed at those same underrepresented question types—could plausibly produce the same win rates. Thus Figure 2 does not isolate retrieval-grounded answer construction, which is the distinctive component of SearchInstruct. The model-editing experiments (Table 2) do not rescue this: all MMLU categories decline and no significance test is reported. Unless a control adds equally sized data without retrieval, the central claim is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SearchInstruct, a four-stage pipeline for constructing supervised fine-tuning (SFT) datasets: seed generation, LLM-based query expansion, document retrieval, and retrieval-grounded answer synthesis. The authors apply the pipeline to two specialized Persian-language domains (traditional cuisine and domestic tourism), fine-tuning MATINA models with the generated data and comparing them to prior MATINA checkpoints via a blind human evaluation. They also apply the framework to model editing: they retrieve up-to-date documents, minimally revise outdated Gemma-3-27B answers with a secondary LLM, and train with ORPO on the resulting preference pairs. The central claim is that retrieval-based dataset construction yields measurable improvements in domain LLM performance and enables lightweight knowledge updates. The paper releases the code, prompts, and generated instruction–response pairs.","tokens_in":24772,"tokens_out":5314,"duration_ms":65923,"significance":"If the central claim were fully supported, SearchInstruct would be a practical contribution to low-resource domain adaptation: it offers a concrete way to generate diverse, realistic, and evidence-grounded instruction data without large-scale human annotation, and it is accompanied by a public repository containing code, prompts, and generated data, which is a genuine strength for reproducibility. The model-editing application is also interesting, and the authors honestly disclose its limitation to surface-level fact replacement. However, the current evaluation does not establish that the observed gains come from the distinctive retrieval-based component. The head-to-head comparison in Figure 2 conflates the method with additional data quantity and iterative targeting, and the MMLU claim in Table 2 is statistically unsupported. The contribution is defensible but needs stronger, controlled evidence.","major_comments":[{"comment":"The main human-evaluation comparison does not isolate the effect of retrieval-grounded answer construction. The 'with' condition is trained on the original MATINA data plus 7,560–8,932 additional SFT pairs (Table 1), while the 'without' condition receives no additional data. The new data were generated in three feedback-driven stages (§3.2) that target the exact query types later probed by the 100-question benchmark. Any additional SFT data—especially data aimed at underrepresented question types—could plausibly produce the same win rates. A control condition that adds an equal-sized dataset produced by the same seed set and expansion/feedback loop but without retrieval (e.g., self-instruct-style or Evol-Instruct-style answers) is required to attribute the win rates to SearchInstruct's retrieval mechanism. As reported, the 62–68% win rates also lack error bars, inter-annotator agreement,","section":"§4.1.1, Figure 2, Table 1"},{"comment":"The independence of the 100-question human benchmark from the training-data distribution is not established. Five annotators each wrote 20 test questions after the iterative refinement loop had already identified weak areas and generated targeted seeds to address them (§3.2, Figure 3). If the test-question authors used guidelines similar to the seed-generation guidelines in Appendix C, the test set is effectively drawn from the same distribution as the training data. This makes the evaluation circular in practice, even if the annotators were not the same people who wrote the seeds. A held-out evaluation set designed before the augmentation, or an external benchmark, would be needed to support the general claim that SearchInstruct improves domain adaptation beyond the targeted distribution.","section":"§4.1.1, Figure 3, Appendix C"},{"comment":"The statement that 'results in Table 2 show no significant drop in accuracy' is not supported. The table shows declines in every MMLU category, from -1.43 to -3.18 points, with an average decline of -1.99. No standard errors, confidence intervals, or significance tests are reported, so 'no significant drop' is a statistical claim without statistical evidence. The STEM decline of -3.18 is especially concerning. Additionally, the targeted evaluation is described only qualitatively ('human evaluation confirmed'); please report the number of test items, win rates, and agreement statistics, or soften the claim accordingly.","section":"§4.2.1, Table 2"},{"comment":"The abstract and introduction claim that SearchInstruct 'notably enhances both the diversity and quality of SFT datasets,' but diversity and quality are not directly measured. Table 1 reports only sample counts; no diversity metric (e.g., embedding dispersion, n-gram coverage, or category entropy) and no quality metric (e.g., factual accuracy or human-rated answer quality) is provided. The human evaluation measures preference over model outputs, which is an indirect and confounded proxy. The paper should either report direct diversity/quality measurements or restrict the claim to the observed preference outcomes.","section":"Abstract, §4.1.1, Table 1"}],"minor_comments":[{"comment":"Please report the raw number of pairwise comparisons per condition and explicitly state how ties are handled. The 'lose' shares (2–4%) are nearly invisible in the stacked bars; a table alongside the figure would improve clarity.","section":"Figure 2"},{"comment":"The seed-generation process is described in detail, but the paper does not report the number of seeds per domain, the number of annotators, or any inter-annotator agreement. These details would help readers assess the reproducibility of the seed set.","section":"§3.2, Appendix C"},{"comment":"The DeepSpeed column uses undefined abbreviations (Z0, Z3) and the meaning of 'Pref.β' is not explicitly defined (presumably the ORPO beta). Please add a footnote or expand the table caption.","section":"Appendix E, Table 5"},{"comment":"There is a typo in the English header: 'Tow' should be 'Two'. Also, the English translation of the second row contains an unbalanced parenthesis; please proofread the appendix tables.","section":"Appendix D, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful pipeline and releases code and data, but the evaluation is not yet at the level needed to support the central claim. The main fix is a controlled ablation that isolates retrieval from data quantity and iterative targeting; this is well within the scope of a revision. The MMLU 'no significant drop' claim must be corrected or statistically justified. The baseline is the authors' own MATINA system; adding an independent baseline (e.g., a non-retrieval data-generation method) would substantially increase confidence. I would not reject the paper, but I would require a major revision before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the four-stage pipeline is a sensible, clearly described way to build domain SFT data — seed expansion, then retrieval-grounded answer generation. The novelty is modest but real: retrieval during dataset construction rather than at inference, plus a human-in-the-loop iterative feedback stage. What the paper does well: the methodology is concrete, prompt templates are in the appendix, and the GitHub promises code and the full generated pairs. The limitations section is unusually honest, especially about shallow model editing (surface facts update, but connected knowledge does not). The before/after examples in Appendices B and D are genuinely illustrative.\n\nThe soft spots are in the empirical evaluation. The central Figure 2 compares models trained on the original MATINA data plus 7,560–8,932 new SearchInstruct pairs against models trained on only the original data. Any additional SFT data — even non-retrieval data generated for the same weak areas — could plausibly produce those 62–68% win rates. The iterative loop explicitly identifies weak subdomains in the same model family and then generates targeted data, so the 100-question benchmark created afterward aligns with the generated data. That is a real confound. The stress-test note is right. The paper needs at least one control: an equally sized dataset generated without retrieval (e.g., Self-Instruct-style expansion) and a head-to-head against it.\n\nThe MMLU model-editing section overstates its case: every category declines by roughly 1.5–3 points, and “no significant drop” is stated without any significance test or confidence interval. That is a minor overclaim, easy to fix. The human evaluation also lacks error bars and inter-annotator agreement, which would strengthen Figure 2.\n\nOne more thing: the paper does not compare against the closest baselines (Self-Instruct with the same budget, RAG-style inference-time retrieval, or CRAFT-style fine-tuning). That weakens the novelty claim, but it is not fatal for a reproducibility-focused venue.\n\nOverall: the method is plausible, the writeup is clear, and the promised artifacts will be useful to the community. The evidence as reported underdetermines the central claim, so I would not accept the results as-is. But the paper deserves serious referee time — with a request for a controlled ablation, significance testing, and a corrected MMLU interpretation. I would send it to peer review rather than desk-reject.","headline":"A practical retrieval-grounded SFT data pipeline whose central comparison is confounded by data quantity and targeted generation; the method is plausible and well-described but needs a controlled ablation.","tokens_in":25214,"tokens_out":2582,"would_cite":true,"duration_ms":29748,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SearchInstruct generates supervised fine-tuning datasets by expanding a handful of seed questions with an LLM and grounding each answer in retrieved documents, claiming measurable gains in specialized domains and a lightweight method for up","keywords":["domain adaptation","instruction dataset generation","retrieval-augmented generation","query expansion","supervised fine-tuning","model editing","Persian language models","cultural domain"],"falsifier":"Train two models on equal-sized SFT sets generated from the same expanded question pool, one with answers grounded in retrieved documents and one with ungrounded LLM answers; if the grounded model does not win significantly on a held-out benchmark that was not used for iterative refinement, the claimed advantage of retrieval-grounded construction is not supported. Alternatively, if a model trained on non-retrieved expansions of the same seeds matches the win rates, the effect is just extra data.","tokens_in":24324,"feed_emoji":"🔎","tokens_out":4323,"duration_ms":42427,"temperature":0.7,"pith_summary":"The paper claims that supervised fine-tuning datasets for specialized domains can be built automatically by combining LLM-based question expansion with retrieval-grounded answer generation. Starting from a small set of human-written seed questions, the SearchInstruct pipeline paraphrases and recombines them, fetches relevant domain documents, and produces answers grounded in that evidence. The authors argue this yields more diverse and realistic instruction–response pairs than document-only or self-instruct-style generation, and they report that models fine-tuned on this data beat earlier models in 62–68% of blind human comparisons in Iranian cuisine and tourism. The same pipeline is repurposed for targeted model editing, updating specific facts from current documents while leaving unrelated knowledge largely intact. A sympathetic reader would take away that retrieval during dataset construction, not just at inference time, is the key to turning scarce domain resources into trainable behavior.","feed_headline":"Retrieval-built instruction data wins 62–68% of blind tests","feed_subtitle":"SearchInstruct expands seed questions, grounds answers in live documents, and fine-tunes better niche-domain models.","key_machinery":"The four-stage pipeline: (1) seed generation from human or human-LLM collaborative writing; (2) LLM-based query expansion, paraphrasing and recombining seeds into a larger instruction pool; (3) document retrieval using web search, RAG, or domain APIs, with an LLM rewrite step to produce search-oriented queries; and (4) response construction, where the instruction is paired with filtered retrieved contexts and an LLM generates an answer grounded in that evidence. For model editing, the pipeline is adapted to retrieve updated documents, generate minimal edits to the base model's old answer, and use the revised answer as the 'chosen' response in preference optimization. The load-bearing idea is","core_discovery":"SearchInstruct's central claim is that query diversity matters more than answer diversity: expanding a modest seed set of human questions into many realistic variants, then grounding each answer in retrieved documents, produces SFT data that fixes coverage gaps ordinary synthetic pipelines miss. The method moves retrieval from inference time to data-construction time, so the fine-tuned model internalizes external knowledge rather than depending on it per query. Evidence is a blind human evaluation where models trained with SearchInstruct data win 62–68% of pairwise comparisons against prior models, and a model-editing experiment where a 27B model is updated with recent facts (political chang","pith_inferences":["A matched control experiment separating 'extra data' from 'retrieval grounding' would sharpen the causal claim; the paper's current comparison confounds quantity, recency, and targeted refinement.","The approach could translate to other knowledge-scarce domains (legal, medical, regional culture) where seed questions are cheap but documents are scattered; a testable prediction is that benefits scale with document availability and query-document vocabulary mismatch.","The shallow-editing limitation hints that retrieval-grounded SFT injects facts but not the relational structure around them; combining the pipeline with knowledge-graph or entity-link supervision might deepen integration.","The query-expansion stage might serve as a data-augmentation tool for evaluation benchmarks too, since it generates realistic user phrasings that document-grounded pipelines tend to miss."],"forward_implications":["If correct, domain adaptation for low-resource languages and niches can be done with a small seed set and a retrieval source, removing the need for large human-annotated corpora.","Models fine-tuned on retrieval-grounded data should rely less on retrieval at inference time, making deployment simpler and cheaper.","The iterative refinement loop (generate data targeting identified weaknesses) provides a practical recipe for continuously improving models after release.","The model-editing experiment suggests a lightweight way to refresh factual knowledge in existing models with minimal general-knowledge degradation, though integration depth is limited.","The method's effectiveness depends on retrieval quality; in noisy or sparse document collections, gains will shrink."],"fun_headline_variants":["Retrieval-built SFT data wins 62–68% blind evals","SearchInstruct: data-time retrieval improves domain LLMs","Ground answers via search, then fine-tune domain models","Query diversity, not answer diversity, drives SFT gains"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The win rates in the human evaluation are attributed to the SearchInstruct method, but the compared models differ in training data quantity, training recency, and the fact that the added data was iteratively refined to fix exactly the weaknesses probed by the 100-question benchmark, so the reported improvements may not isolate the effect of retrieval-based construction.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-built SFT data wins 62–68% blind evals","SearchInstruct: data-time retrieval improves domain LLMs","Ground answers via search, then fine-tune domain models","Query diversity, not answer diversity, drives SFT gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2334,"prompt_tokens":737,"completion_tokens":1597,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1525}},"tokens_in":481,"tokens_out":1597,"duration_ms":17504,"temperature":1.0,"reasoning_tokens":1525,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:35:17.207782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two models on equal-sized SFT sets generated from the same expanded question pool, one with answers grounded in retrieved documents and one with ungrounded LLM answers; if the grounded model does not win significantly on a held-out benchmark that was not used for iterative refinement, the claimed advantage of retrieval-grounded construction is not supported. Alternatively, if a model trained on non-retrieved expansions of the same seeds matches the win rates, the effect is just extra data.","supporting_citations":[],"review_version":1}