{"id":"f960d847-ab7c-49f5-9564-b8aea9cec0c3","arxiv_id":"2506.03968","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 1M-instruction dataset, generated by attributing real instructions to documents, users and motivations and then reversing the process over web documents, improves instruction-following benchmarks for 7B-14B models.","lead":"Researchers built a two-step pipeline that first attaches real user instructions to a web document plus a simulated user and motivation, then reverses the process to mass-produce new instructions from web pages. The result is SynthQuestions, a 1-million-item instruction dataset that improves LLaMA-3-8B and Qwen2.5 alignment benchmarks when used for fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Arena Hard comparison may be a selection artifact because the same modified Arena Hard rubric is used to filter the training data and to judge the headline benchmark; a human-agreement study of the rubric would settle this.","rationale":"The reader's CONDITIONAL verdict is appropriate, and the weakest assumption identified by the reader is also the one I would stress-test first. The circularity between data filtering and evaluation is concrete and load-bearing: the same seven dimensions are used to filter REALQUESTIONS and SynthQuestions (Sections 3 and 4.2) and to judge Arena Hard (Section 6.1 and Appendix I). Without human agreement or a filter ablation, the Arena Hard gain cannot be cleanly separated from optimizing the evaluator's own checklist. This is a correctness risk, not a stylistic concern, because Arena Hard is one of the two headline benchmarks and the DPO comparison that is claimed to match models trained with 10x more data also relies on the same benchmark. The Alpaca Eval 2.0 result is genuinely independent of this loop and remains the best support for the central claim; a reader who cares only about that result can accept the empirical core. The paper also has real merits: cross-model validation on Qwen2.5-7B/14B, a scaling curve, ablations that isolate attributed grounding and math/code documents, and an honest limitation section that acknowledges no >1M scaling, no corpus-selection study, and no hallucination assessment. The missing code and data, unspecified response-generation step, and corrupted table formatting block exact reproduction, but they are secondary to the selection-effect concern. If the proposed human-agreement check passes, the CONDITIONAL can be upgraded; if it fails, the Arena Hard-specific claims should be explicitly downgraded while the Alpaca Eval-based claim would remain credible.","tokens_in":22750,"tokens_out":8422,"duration_ms":86843,"concrete_test":"Stratified-sample human-agreement study of the filtering rubric: draw 300 instructions from the unfiltered generation pool covering all 0-7 scores assigned by LLaMA-3-70B-Instruct, have at least three annotators rate each instruction on the seven dimensions in Appendix I, and compute the quadratic-weighted kappa between the LLM judge's total score and the human modal score against a pre-registered threshold (e.g., kappa >= 0.6). If agreement falls below threshold, the Section 3/4.2 filter is not validated as a proxy for human-perceived instruction quality, and the Arena Hard comparisons should be treated as selection-on-evaluator evidence rather than evidence of improved instruction-following.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is the validity of the modified Arena Hard rubric as a measure of instruction quality, and it is under-supported in exactly the places where it matters. In Section 3 (Data Filtering), REALQUESTIONS keeps only human instructions scoring 7/7 on criteria described as modified from Arena Hard, with the seven dimensions listed in Appendix I. In Section 4.2, synthetic instructions below a score of 3 are discarded, and the final 1M SynthQuestions is built by including the highest-scoring instructions in each topic. That same seven-dimension rubric is the basis of the Arena Hard benchmark used as a headline result in Section 6.1. The paper provides no human-agreement study, no description of what the claimed modification of Arena Hard consists of, and no comparison of models trained with versus without the score filter. If LLaMA-3-70B-Instruct's rubric is a poor proxy for human-perceived instruction quality, the Arena Hard WR of 15.4 and the DPO result of 24.8 are partly optimizing the evaluator's checklist rather than improving general instruction-following. The Alpaca Eval 2.0 WR of 19.15 is judged by GPT-4-1106 under a different protocol and is not the same checklist used in filtering; it is therefore the cleaner evidence for the central claim. The paper should rest its headline on Alpaca Eval 2.0 (and the closed-form benchmarks) until the rubric is validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for synthesizing large-scale instruction data using \"attributed grounding,\" in which a set of high-quality human instructions (REALQUESTIONS) is first attributed to documents, users, and motivations, and then new instructions are generated from web documents by reversing this process with LLM prompts. The authors construct a 1M-instruction dataset (SynthQuestions), filter it using a seven-dimension rubric based on Arena Hard scored by LLaMA-3-70B-Instruct, and fine-tune LLaMA-3-8B and Qwen2.5 models. They report strong performance on Alpaca Eval 2.0 and Arena Hard, as well as on closed-form benchmarks, with ablations showing the benefit of attributed grounding and the inclusion of math/code documents. The paper also includes a DPO extension and scaling experiments.","tokens_in":22841,"tokens_out":5319,"duration_ms":45368,"significance":"If the results are robust, the attributed-grounding pipeline is a valuable and scalable method for producing diverse, complex instruction data, potentially reducing reliance on expensive human annotation. The paper's empirical core is well structured: the ablation in Table 5 supports the importance of grounding (+~5 Alpaca Eval points) and math/code documents (+~8 GSM8K points), and the cross-model evaluations on LLaMA-3-8B and Qwen2.5 demonstrate generalizability. The release of the dataset and code is a further positive. However, the headline Arena Hard results are compromised by a selection-bias concern: the same rubric is used to filter both the seed and synthetic data and to evaluate the final model. The Alpaca Eval 2.0 results, judged by GPT-4-1106 under a different protocol, are the cleaner evidence for the core claim.","major_comments":[{"comment":"The Arena Hard result in Table 1 is potentially a selection artifact because the same seven-dimension rubric that filters REALQUESTIONS (score 7/7 in Section 3) and selects synthetic instructions (score >=3 in Section 4.2, then highest-score-per-topic selection) is also the foundation of the Arena Hard benchmark used in Section 6.1. The paper provides no human-agreement study for this rubric, and Appendix I does not describe what the claimed 'modification' of Arena Hard consists of; the criteria in Appendix I are an absolute scoring rubric, not the pairwise comparison protocol of the original Arena Hard. To make the headline claim credible, the authors should either validate the rubric against human judgments or present Alpaca Eval 2.0 as the primary evidence and explicitly flag Arena Hard as potentially optimistic due to rubric overlap.","section":"Sections 3, 4.2, and 6.1"},{"comment":"The statement that SynthQuestions-trained models are \"comparable with models trained with 10 times more data and preference training\" is not supported by the reported numbers. LLaMA-3-8B-Instruct (which uses >10M data) outperforms SynthQuestions on both Arena Hard (20.6 vs 15.4) and Alpaca Eval 2.0 (22.57 vs 19.15). While the DPO variant in Table 3 reaches AE 33.81 and AR 24.8, that improvement comes after preference optimization and is not what the introduction claims. The claim should be rephrased to be bounded by the actual comparison group, e.g., 'among models trained exclusively on open-source data.'","section":"Section 1 and Table 1"},{"comment":"The scaling experiment does not support the abstract's claim of improvements 'with more web corpora.' The experiment draws random subsets from the already-filtered 1M SynthQuestions, so it demonstrates scaling with instruction count, not with an expanding set of web documents. To substantiate the claim, the authors should hold the synthesis pipeline fixed and vary the underlying document corpus (e.g., additional FineWeb subsets), observing how model performance changes as more documents are used.","section":"Section 6.2 and abstract"},{"comment":"The rationale for the score threshold 3 is unclear and inconsistent with the stated criteria. The text says that 'Specificity, Problem-Solving and Technical Accuracy are three more fundamental requirements,' but the threshold is applied to the total score across all seven dimensions, not to those three dimensions specifically. This ambiguity affects reproducibility of the dataset construction. Please clarify the scoring rule and provide an ablation or sensitivity analysis of the threshold's effect on downstream performance, or at least a distribution plot showing how scores correlate with quality.","section":"Section 4.2"}],"minor_comments":[{"comment":"The word 'sufficientlydiversified' is missing a space in the abstract; also 'diversified' and 'complicated' appear in an odd concatenation. Please fix the typography.","section":"Abstract and Section 1"},{"comment":"The word 'repectively' should be 'respectively' in the first paragraph of Section 6.1.","section":"Section 6.1"},{"comment":"Several model names are misspelled: 'Proprietay' should be 'Proprietary', 'LLaMA-3-8B-Insturct' should be 'LLaMA-3-8B-Instruct', and 'Qwen-2-7B-Insturct' should be 'Qwen-2-7B-Instruct'. Also, the table notes should clarify whether the unreported MAmmoTH2 Alpaca Eval 2.0 result is intentionally omitted.","section":"Tables 1 and 2"},{"comment":"The t-SNE plots are referenced as 'Figure ??' in the text; the figure numbers should be fixed. The violin plot in Figure 6 lacks axis labels and a description of what the y-axis range represents (0-7), which makes the score distribution comparison difficult to interpret.","section":"Section 5.2 and Figure 6"},{"comment":"The criteria listed in Appendix I are the same seven items used in the filtering prompt in Appendix J.3. The paper says these criteria are 'modified from Arena Hard,' but the modification is never described. This is important because the original Arena Hard uses a pairwise-comparison protocol; the absolute 7-dimensional scoring used here is substantially different and should be documented exactly.","section":"Appendix I"},{"comment":"The safety analysis reports that 4.32% of data points have potential risks, but then states that 'for all the other categories, the potentially risky data points are less than 0.2%.' This is misleading because the 'Specialized Advice' category alone is 3.60%, and the sum of all other categories is well under 1%; please report the percentages consistently and clarify whether 'less than 0.2%' refers to each category individually.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is promising and the ablations are convincing, but the Arena Hard headline is compromised by the same-rubric selection bias. I recommend asking the authors to (1) either provide a human-agreement validation of their rubric or demote Arena Hard to a secondary metric, (2) revise the introductory claim about comparability with 10x-data models, and (3) clarify the scaling claim about web corpora. These are fixable within the scope of the manuscript, so major revision rather than reject seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the attributed-grounding mechanism is a real contribution. Taking real instructions, attributing them to (document, user, motivation) triples, then reversing that to synthesize new instructions from fresh web documents is a new combination of known pieces. The paper supports it with honest ablations — removing grounding drops Alpaca Eval from 15.63 to about 10.7, and adding math/code documents adds 8 GSM8K points. Cross-model gains on Qwen2.5-7B/14B and a scaling curve add credibility. That is solid, useful work.\n\nThe soft spots are real. The headline Arena Hard result is partly a selection artifact. The same seven-dimension rubric, scored by LLaMA-3-70B-Instruct, filters the seed data (only 7/7 kept), filters synthetic data (score >= 3), and picks the final 1M (top scores per topic). That same rubric is then the headline evaluation. No human-agreement study is provided, and the claimed modification of Arena Hard is unspecified. So the 15.4 Arena Hard WR is not independent evidence. The Alpaca Eval 2.0 result, judged by GPT-4-1106 under a different protocol, is the cleaner evidence and should be the headline.\n\nThe intro framing overreaches: 'comparable with models trained with 10 times more data and preference training' is not supported by the tables. The SFT model trails stock LLaMA-3-8B-Instruct by 3.4 to 5.2 points on both headline benchmarks, and the DPO result that beats LLaMA-3-70B-Instruct on Alpaca Eval WR has no error bars. The abstract's more modest claim — leading among open-source-data models with continual scaling — is what the evidence backs.\n\nReproduction is currently impossible. Data and code are promised but absent, Google retrieval is nondeterministic, the rubric modification is unspecified, and the paper never explains how (instruction, response) pairs were formed from an instruction-only dataset. Several tables are corrupted, blocking verification of specific numbers. The diversity evidence (t-SNE area) is weak on its own, though the Vendi Score supplements it.\n\nThis paper is for researchers working on synthetic instruction data and LLM alignment. It deserves a serious referee. The mechanism is worth testing, the ablations are informative, and the scaling behavior is useful. But the authors should validate the rubric with human annotations, rest the headline on Alpaca Eval 2.0, fix the tables, and release the data and code before the Arena Hard claims are taken at face value. I would engage with it and send it to peer review, with a request for major revision around evaluation validity.","headline":"Attributed grounding is a genuinely new recipe for scalable instruction synthesis and the Alpaca Eval gains look real, but the Arena Hard headline is partly a selection artifact from using the same rubric for filtering and evaluation.","tokens_in":23583,"tokens_out":3080,"would_cite":true,"duration_ms":27283,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Synthesizing instructions by first imagining who would ask them and why, then generating them from web documents, yields a 1M dataset that fine-tunes LLaMA-3-8B to the best Alpaca Eval 2.0 win rate among models trained only on open data.","keywords":["synthetic instruction data","attributed grounding","instruction tuning","LLM alignment","data diversity","web-scale synthesis","supervised fine-tuning","Arena Hard rubric"],"falsifier":"Have human annotators score a random sample of SynthQuestions instructions with the same seven dimensions and measure their agreement with LLaMA-3-70B-Instruct; if agreement is low, or if a model trained on unfiltered SynthQuestions beats the rubric-filtered model in a blind pairwise human-preference evaluation, the reported gains are an artifact of the judge rather than of instruction quality.","tokens_in":22311,"feed_emoji":"🤖","tokens_out":11999,"duration_ms":101366,"temperature":0.7,"pith_summary":"Large language models learn to follow instructions from example pairs, but human-written examples are scarce and existing synthetic generation methods produce narrow or shallow prompts. This paper tries to break that bottleneck with attributed grounding: first attribute real user instructions to a document, a user, and a motivation, then reverse the process and generate new instructions from arbitrary web documents by imagining who would ask them and why. The resulting 1-million-instruction dataset, SynthQuestions, fine-tunes LLaMA-3-8B to a 19.15 win rate on Alpaca Eval 2.0, the best among models trained on open-source data, and to 15.4 on Arena Hard, where only the 10M-instruction MAmmoTH2 baseline does better. The authors describe this as comparable to models trained with ten times more data plus preference training. A sympathetic reader would care because the method turns the open web into a scalable source of alignment data, with measured improvements continuing as the corpus grows.","feed_headline":"1M grounded prompts match training runs with 10x more data","feed_subtitle":"LLaMA-3-8B tuned on the synthetic set posts the best Alpaca Eval win rate of any open-data model.","key_machinery":"The load-bearing object is the attributed-grounding loop expressed in Equations (1)-(3). Equation (1) builds the attributed seed set $RQ_\\alpha = \\{(i, d, u, m)\\}$ by linking each curated real instruction $i$ to a retrieved web document $d$, a user $u$, and a motivation $m$. Equation (2) reverses the direction: given a fresh web document $d'$, LLaMA generates a grounded situation (user $u'$, motivation $m'$) using attributed samples from $RQ_\\alpha$ as in-context demonstrations, and Equation (3) then asks the model to speak as that user and produce the instruction $i'$ most likely to arise in that situation. The same seven-dimension rubric drawn from Arena Hard — specificity, domain knowledge, complexity, problem-solving, creativity, technical accuracy, and real-world application — scored by LLaMA-3-70B-Instruct, selects the 29K seed instructions (only full-score 7 entries are kept), discards generated instructions scoring below 3, and ranks the final dataset after BERTopic topic modeling keeps the highest-scoring instructions per topic.","core_discovery":"On its own terms, the paper's claim is that the missing ingredient in synthetic instruction data is situatedness: instructions that align models efficiently are typically grounded in a document (the background knowledge), a user (who asks), and a motivation (why they ask), and synthetic instructions inherit that grounding only when the generating model is forced to construct all three. The paper implements this as a two-stage loop — top-down attribution of 29K curated human instructions (REALQUESTIONS) into document-user-motivation triples, then bottom-up synthesis that starts from web documents, generates a situation, and then an instruction — and shows that LLaMA-3-8B fine-tuned on the resulting 1M SynthQuestions outperforms all open-data-trained baselines on Alpaca Eval 2.0 with a 19.15 win rate, and trails only MAmmoTH2 on Arena Hard (15.4 versus 16.6), where MAmmoTH2 used 10M data plus additional math and code fine-tuning. The paper further claims the data scales: performance improves monotonically as the training subset grows from 200K toward 1M, transfers to Qwen2.5-7B and Qwen2.5-14B with large margins on MATH, and supports preference optimization, with a SynthQuestions-plus-DPO model reaching a 33.81 Alpaca Eval win rate that surpasses the LLaMA-3-70B-Instruct generator itself.","pith_inferences":["Beyond the paper: the attribution loop could be iterated — instructions synthesized in one pass could themselves be attributed to documents, users, and motivations and serve as demonstrations for a second pass, a self-amplifying diversity loop the authors do not test.","Beyond the paper: because mixing math and code documents shifts gains specifically to GSM8K, document selection acts as a difficulty dial, and deliberately curating web corpora by cognitive-demand profile could steer which capabilities alignment data improves.","Beyond the paper: since one judge model supplies the scores that select the seed set, filter generations, and rank the final dataset, part of the reported gain may be self-consistent optimization against that judge; a blind human-preference comparison between filtered and unfiltered models would separate genuine alignment gains from rubric conformity."],"forward_implications":["If the scaling curve holds, adding more web corpora to the attribution loop should keep improving instruction-following performance, since document volume is effectively unlimited.","The recipe transfers across base models: Qwen2.5-7B and Qwen2.5-14B trained on SynthQuestions subsets beat the same models trained on OpenHermes, GenQA, and MAmmoTH2 subsets on most benchmarks, with particularly large margins on MATH.","The data is compatible with preference optimization: applying DPO on 100K sampled instructions raises the Alpaca Eval win rate from 19.15 to 33.81, surpassing the LLaMA-3-70B-Instruct data generator.","Choosing the document corpus is a direct control over which capabilities improve: mixing math and code documents (PILE, MathPILE) into FineWeb raises GSM8K from 50.34 to 58.30 without hurting Alpaca Eval."],"supporting_citations":[{"why":"Supplies the seven-dimension scoring rubric used at every stage of dataset construction and the Arena Hard benchmark used for headline evaluation.","marker":"[20]"},{"why":"The main evaluation benchmark; the paper's win-rate results and scaling curve are measured on Alpaca Eval 2.0.","marker":"[10]"},{"why":"The primary web corpus whose documents seed the bottom-up synthesis of grounding situations and instructions.","marker":"[28]"},{"why":"The 10M-instruction web-derived baseline that SynthQuestions is compared against on Arena Hard and Alpaca Eval 2.0.","marker":"[40]"},{"why":"The math corpus whose removal drops GSM8K from 58.30 to 50.34, demonstrating the effect of document mix on reasoning gains.","marker":"[34]"},{"why":"The additional reasoning-heavy document source mixed with FineWeb to produce complex math and coding instructions.","marker":"[13]"}],"fun_headline_variants":["Attributed grounding: 1M synthetic instructions from web docs","Synthetic data with real context: LLaMA-3-8B tops open-data models","From real questions to 1M grounded synthetic instructions","Situated synthesis: web docs spawn complex user instructions","1M grounded prompts: efficient alignment, scalable with web corpus"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the seven-dimension rubric — specificity, domain knowledge, complexity, problem-solving, creativity, technical accuracy, and real-world application — as scored by the single judge model LLaMA-3-70B-Instruct, is a trustworthy measure of instruction quality, because the same scoring selects the seed set, filters generated instructions, and is used by the Arena Hard benchmark itself, with no human-agreement check reported.","fun_headline_variants_meta":{"raw":{"variants":["Attributed grounding: 1M synthetic instructions from web docs","Synthetic data with real context: LLaMA-3-8B tops open-data models","From real questions to 1M grounded synthetic instructions","Situated synthesis: web docs spawn complex user instructions","1M grounded prompts: efficient alignment, scalable with web corpus"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000341,"raw_usage":{"total_tokens":1942,"prompt_tokens":1069,"completion_tokens":873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":784}},"tokens_in":685,"tokens_out":873,"duration_ms":8649,"temperature":1.0,"reasoning_tokens":784,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:53:17.620671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators score a random sample of SynthQuestions instructions with the same seven dimensions and measure their agreement with LLaMA-3-70B-Instruct; if agreement is low, or if a model trained on unfiltered SynthQuestions beats the rubric-filtered model in a blind pairwise human-preference evaluation, the reported gains are an artifact of the judge rather than of instruction quality.","supporting_citations":[{"cited_title":"Gonzalez, and Ion Stoica","cited_arxiv_id":null,"evidence_quote":"Supplies the seven-dimension scoring rubric used at every stage of dataset construction and the Arena Hard benchmark used for headline evaluation."},{"cited_title":"The fineweb datasets: Decanting the web for the finest text data at scale, 2024","cited_arxiv_id":null,"evidence_quote":"The primary web corpus whose documents seed the bottom-up synthesis of grounding situations and instructions."},{"cited_title":"Generative ai for math: Part i – mathpile: A billion-token-scale pretraining corpus for math, 2023","cited_arxiv_id":null,"evidence_quote":"The math corpus whose removal drops GSM8K from 58.30 to 50.34, demonstrating the effect of document mix on reasoning gains."}],"review_version":1}