{"id":"1016e769-43f2-4207-ba0d-0c2d5a2bdf17","arxiv_id":"2501.18511","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 50-million-conversation synthetic chat dataset built from 54 open-weight models, plus an SFT mix that beats Tulu-3's mix with fewer samples.","lead":"The authors release WILDCHAT-50M, a large chat dataset with responses from over 50 open-weight language models, and show that a small mix sampled from it, RE-WILD, outperforms an established Allen AI SFT mixture on several post-training benchmarks. The value is a public resource for studying how the choice of data-generating model affects downstream fine-tuned model quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DGM-quality and RE-WILD superiority claims are confounded by unreported, per-model generation settings that Appendix F itself says materially affect output quality.","rationale":"The strongest claim is comparative: the field gains evidence that a small, benchmark-tuned mix built from a selected open-weight DGM can beat a larger curated SFT mix. For that to be true, the WILDCHAT-50M responses must be comparable across models, and RE-WILD's Q72 component must reflect Q72's data quality rather than favorable generation settings. Neither condition is established. The context-window heuristic means different models saw different prompt truncation, so Table 2's comparison of Q72 with a 20,000-token window against L70 with an 8,192-token window is precisely the kind of setting where response length and style can masquerade as quality. Appendix F makes the omission concrete by warning that temperature materially changes output diversity and quality, yet no temperature values are reported anywhere in the paper. The paper's own styling-inheritance results show that SFT models adopt DGM formatting, so any configuration-driven stylistic difference would be inherited and would propagate into benchmark scores. This makes the concern load-bearing rather than a mere reproducibility nicety. The reader's weakest-assumption diagnosis is the same one I reach, and the additional concerns about missing numeric comparisons and same-benchmark selection are real but secondary. The dataset artifact may still be valuable, and a controlled regeneration experiment or the original generation logs could resolve the concern; therefore the correct verdict remains CONDITIONAL rather than a full rejection.","tokens_in":30089,"tokens_out":4058,"duration_ms":52702,"concrete_test":"Re-generate the six DGMs in Table 2 and the Q72 subset used in RE-WILD with identical generation settings: same temperature (e.g., 0.7), same context window (e.g., 8,192 tokens), same maximum output token count, and same system prompt, then fine-tune Llama-3.1-8B with the paper's exact recipe and evaluate on the same nine benchmarks. If the average ordering in Table 2 or the RE-WILD-versus-Tulu-3 margin changes by more than the reported 95% confidence intervals, the original attribution is confounded. A cheaper first step is to request the original per-model generation logs (temperature, top-p, context length, max tokens); if these no longer exist, the paper should say so and the comparative claim should be weakened accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claims are that DGM identity strongly affects downstream SFT quality and that RE-WILD outperforms the Tulu-3 SFT mix. Both claims require that differences across generating models reflect model quality rather than generation configuration. Section 2.1 describes per-model context windows that were 'heuristically maximize[d]', ranging from 2,048 to 20,000 tokens, with the largest models run in FP8 quantization, and the paper never reports sampling temperatures. Appendix F explicitly states that temperature settings 'substantially influence output diversity and quality' and recommends that researchers 'Meticulously document all hyperparameters used in generation'. Since response length, style, and diversity are known to shift with context length and temperature, and since Section 3.2 shows that SFT models inherit stylistic properties from their DGMs, the Table 2 DGM ranking and the RE-WILD advantage could partly reflect generation configuration rather than DGM quality. The paper's own context-truncation ablation addresses only one model and one dimension, so it does not resolve the confound. This is the most load-bearing weakness because the paper's headline contribution is the comparative conclusion, not merely the release of a large dataset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WILDCHAT-50M, a large corpus of synthetic chat transcripts generated by 54 open-weight data-generating models (DGMs) on prompts from WildChat. It reports analyses of generation efficiency, intra-LLM response similarity, and the effect of DGM choice on downstream SFT performance, and introduces RE-WILD, a three-source SFT mix that is claimed to outperform the Tulu-3 SFT mixture with only 40% as many samples. The main empirical claims are that DGM identity strongly and unpredictably affects downstream SFT quality, that RE-WILD is a strong SFT mix, and that several secondary findings hold: data scaling helps, blending DGMs does not, style is inherited from the DGM, and on-policy DGMs are more effective teachers. The paper also provides the dataset, samples, and code in a public repository.","tokens_in":30281,"tokens_out":7127,"duration_ms":64637,"significance":"If the claims hold, the paper would provide a valuable public artifact: a chat post-training corpus far larger than prior public chat sets, together with evidence about the sensitivity of SFT to data-generating-model choice. The authors release the dataset, samples, and code, and the evaluation covers nine benchmarks with a standardized framework. The strongest assets are the scale and diversity of the generated corpus and the extensive benchmarking effort. However, the comparative conclusions are currently under-supported because generation settings are neither controlled nor reported, and the headline RE-WILD comparison is not numerically documented in the manuscript itself.","major_comments":[{"comment":"The DGM comparison in Table 2 and the choice of Qwen-2.5-72B as the main RE-WILD component are confounded by unstandardized and unreported generation settings. Section 2.1 reports that context windows were 'heuristically maximize[d]' and range from 2,048 to 20,000 tokens, and that the largest models were run in FP8 quantization, while sampling temperatures are never reported. Appendix F states that temperature settings 'substantially influence output diversity and quality' and recommends that researchers 'Meticulously document all hyperparameters used in generation'—a recommendation the paper itself does not follow. Since Section 3.2 demonstrates that SFT models inherit stylistic properties from their DGMs, differences in response length and style across DGMs could drive part of the Table 2 ranking. The context-truncation ablation in Section 3.2 tests only one model and only the context-length dimension, so it cannot rule out the confound. The central comparative claims require either standardized generation settings or an explicit sensitivity analysis that varies temperature and context length while holding the DGM fixed.","section":"§2.1, Appendix F"},{"comment":"The headline claim that RE-WILD 'outperforms the recent Tulu-3 SFT mixture from Allen AI with only 40% as many samples' is not supported by any numeric table in the manuscript: Figure 1 is a spider chart and the text directs readers to the GitHub repository for exact scores. No confidence intervals are reported for the RE-WILD versus baseline comparison. In addition, Section 3.1 says RE-WILD components were chosen 'heuristically' while Section 3.3 describes the 'empirical process by which we determined which DGMs had high SDQ'; because the same benchmarks are used for both selection and evaluation, the authors should discuss this selection bias and should include the numeric breakdown in the paper itself.","section":"§3.1, Fig. 1"},{"comment":"The claim that fine-tuning on on-policy or same-family DGMs 'produces stronger benchmark results' is contradicted by the paper's own Table 6. For the Llama-3.1-8B target, L8B:L8I achieves Avg 0.36, identical to L8B:L70 and only 0.01 above L8B:Q7; the Table 6 caption's statement that 'Both Llama and Qwen benefit from more similar upstream models' is therefore not supported for Llama. The main-text conclusion should be restricted to the Qwen target or supported with additional runs and significance estimates.","section":"§3.2, App. B, Table 6"},{"comment":"The central quantitative findings rest on single training runs without seeds or run-to-run variance. Table 2 reports 95% confidence intervals for some benchmark columns but not for MTBench or AlpacaEval, and Figure 2's scaling curves and Figure 1's spider chart have no error bars. Because differences in Avg as small as 0.01 (e.g., L8B:L8I vs L8B:L70 in Table 6) are used to draw conclusions, the paper should either add repeated-seed results or explicitly state that differences of this size are within noise.","section":"§2.2, §3.2, Figs. 1-2"}],"minor_comments":[{"comment":"The paper alternates between '50 different open-weight models', '54 DGMs', 'WILDCHAT-50M', and 'over 125 million chat transcripts in aggregate'; please clarify whether the dataset unit is conversations, responses, or turns and make the counts consistent.","section":"§1, §2.2"},{"comment":"There are several typos: 'attactive' should be 'attractive', and 'A vgis' in the Figure 2 caption should be 'Avg is'.","section":"§3.2, Fig. 2"},{"comment":"The symbols '¡' and '¿' before 0.3 and 1.2 should be '<' and '>'.","section":"Appendix F"},{"comment":"The sentence 'On the AlpacaEval leaderboard, Qwen 2.5-72B and Llama 3.3-70B are essentially tied; but Qwen 2.5-72B is a superior DGM (as measured by AlpacaEval performance)' is self-contradictory and should be rephrased.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The main fixable weaknesses are the missing generation-hyperparameter documentation and the absence of in-paper numeric results for the RE-WILD comparison; the dataset artifact itself is potentially very valuable. The on-policy claim inconsistency with Table 6 should be fixed before acceptance. I see no misconduct concerns; the benchmark-selection issue is standard and should be acknowledged rather than ignored."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ben, two things to know before you spend time: the dataset is real, large, and useful, and the paper's headline comparative claim is not yet pinned down with numbers or clean experimental control.\n\nWhat's actually new: WILDCHAT-50M gives the community 50M+ multi-model chat responses to the same prompts, across 54 open-weight DGMs, roughly 125M transcripts total, with code and samples released. That's a real resource, about 50x the size of prior public chat datasets, built at 10k H100-hours. The analyses around it—response similarity across models, style heritability from DGM to SFT, the scaling and blending ablations—are new at this scale and mostly sensible. The authors use a standardized eval harness, report CIs on several tables, and include honest limitations (Section 5) plus Appendix F, which lists best practices for synthetic generation. That's more than many resource papers do.\n\nThe soft spots, in order of seriousness. First, the RE-WILD vs Tulu-3 result is shown as a spider chart with no numeric table in the main text; exact scores are deferred to GitHub. For the headline claim, that's not adequate—readers need numbers and CIs. Second, and more load-bearing, generation settings are not controlled or reported. Context windows vary from 2048 to 20000 tokens, the largest models run in FP8, and sampling temperature is never given—even though Appendix F says temperature 'substantially influence[s] output diversity and quality' and recommends documenting all hyperparameters. The authors' own context-truncation ablation (Qwen 20k→8k) partially addresses context length, but only for one model, and it leaves the temperature confound untouched. So the DGM ranking in Table 2 and the RE-WILD advantage could partly reflect generation config rather than model quality. The stress-test note is right about this; I don't think it's fatal to the dataset, but it is fatal to the strength of the comparative claims as written. Third, RE-WILD's components were chosen using the same benchmark suite on which it is evaluated; some in-sample fitting is likely. A held-out benchmark or two would help.\n\nWho should read it: anyone working on post-training data curation or synthetic data quality; the dataset itself is the deliverable. If a serious referee got this, I would send it out, with a required revision: numeric tables with CIs for RE-WILD, per-model generation configs, and at least one held-out benchmark. The dataset deserves publication even if the comparative claims get softened.","headline":"The dataset is a genuine step-change resource; the RE-WILD superiority claim is plausible but currently rides on a spider chart and unreported generation settings.","tokens_in":30810,"tokens_out":3410,"would_cite":true,"duration_ms":38599,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces WILDCHAT-50M, a 50-million-conversation synthetic chat corpus generated by 54 open-weight model variants, and shows that a three-source SFT mix built from the best generator's responses outperforms the Tulu-3 mixture…","keywords":["synthetic data","post-training","supervised fine-tuning","data generating models","chat transcripts","dataset release","LLM evaluation","data curation"],"falsifier":"Re-run the six-generator comparison of Table 2 with identical generation settings, including the same temperature, the same context truncation, and the same token limits, across all generators; if the ranking of teachers by downstream SFT performance changes materially, the paper's assignment of credit to DGM identity is confounded by generation configuration. A second check is to reproduce RE-WILD's advantage over the Tulu-3 SFT mixture using only the officially released hyperparameters and an independent random seed.","tokens_in":29854,"feed_emoji":"💬","tokens_out":5564,"duration_ms":51353,"temperature":0.7,"pith_summary":"This paper tries to establish that the choice of model generating synthetic training data is a decisive factor in how well a fine-tuned language model performs, and that a large public corpus of such data can beat carefully curated mixes. The authors build WILDCHAT-50M, the largest public chat dataset to date, by having over 50 open-weight models respond to real user prompts from the WildChat corpus. Fine-tuning Llama-3.1-8B on a three-source mix of that data, called RE-WILD, outperforms the larger Tulu-3 SFT mixture on nine post-training benchmarks while using only 40% of its samples. If correct, the paper gives the field a public resource for studying synthetic data quality and evidence that modest, well-chosen synthetic sets can displace larger curated ones.","feed_headline":"50 open models answer 50M chats; a smaller mix beats Tulu-3","feed_subtitle":"Real user prompts re-answered by 54 open-weight models produce SFT data that outperforms a curated blend 40% larger.","key_machinery":"The central object is the dataset itself: WILDCHAT-50M, roughly 50 million chat conversations in which real user prompts from the existing WildChat corpus are re-answered by 54 open-weight model variants ranging from 0.5B to 104B parameters. The comparative method is synthetic data quality (SDQ) measurement: each generating model's responses become a supervised fine-tuning set for a fixed base model (Llama-3.1-8B), and the resulting fine-tunes are scored on a standardized nine-benchmark suite mixing ground-truth and LLM-judge evaluations. The RE-WILD mix then combines 246,750 conversations from the best-performing generator with 99,800 MMLU auxiliary items and 20,000 math items, showing that a small, complementary-skills blend can beat a larger curated mixture.","core_discovery":"The paper's central discovery is that synthetic data quality is driven less by which benchmark a generating model is good at and more by domain-agnostic response traits, and that these traits are inherited during supervised fine-tuning. Across six data-generating models from four model families, ranging from 0.5B to 104B parameters, no single model dominates nine benchmarks, and parameter count does not predict data quality. Fine-tuned models track their teacher's stylistic choices, such as structure, comprehensiveness, and tone, almost exactly, while skills such as world knowledge and mathematics transfer only when the data is specifically curated for them. The paper also reports that blending responses from multiple generating models gives no benefit over single-model data, and that responses from diverse LLMs resemble each other far more than human responses to the same prompts would.","pith_inferences":["A testable extension the paper leaves implicit: running the same generation pipeline at fixed temperature and context settings across all 50 models would separate DGM quality from generation configuration; the paper's own appendix warns that temperature settings substantially influence output diversity and quality, yet never reports the values used.","The finding that response blends benchmark between their parts suggests that the prompt-diversity benefits established for large real datasets do not transfer to response space, implying that curation effort should concentrate on selecting teachers rather than mixing their outputs.","The styling-inheritance result suggests a cheap diagnostic for synthetic data quality: measure structural traits of a candidate generator's outputs before committing compute to full-scale generation.","The near-chance agreement between teacher and student benchmark strengths implies that distillation rankings can shift at inference time; users of such fine-tunes may want to re-evaluate models trained on their own outputs rather than assuming student rankings mirror teachers."],"forward_implications":["Researchers can replace hand-curated SFT mixtures with response data from a single strong open-weight model and match or beat larger blends at lower cost.","DGM selection, not prompt diversity, is the main lever for generalist chat quality; blending multiple response generators adds nothing beyond averaging their performance.","Style traits such as comprehensiveness and formatting are inherited by fine-tunes, so dataset builders can control the stylistic fingerprint of a model by choosing which generator answers the prompts.","Scaling synthetic data improves SFT performance for some generators but tapers off quickly for others, so the value of collecting more data depends heavily on which model generated it.","Fine-tuned models do not inherit their teacher's benchmark-specific strengths, so a good teacher is not necessarily one that scores well on the target benchmark."],"supporting_citations":[{"why":"Supplies the real user prompts that all responses in WILDCHAT-50M answer, holding prompt diversity fixed across generators.","marker":"(Zhao et al., 2024b)"},{"why":"Provides the Tulu-3 SFT baseline that RE-WILD must beat and the Persona Hub Algebra data used in the mix.","marker":"(Lambert et al., 2024)"},{"why":"The inference engine (vLLM) through which all 50 models' responses were generated, enabling controlled efficiency comparisons.","marker":"(Kwon et al., 2023)"},{"why":"Supplies Evalchemy, the standardized evaluation framework that produces the nine-benchmark scores.","marker":"(Guha et al., 2024)"},{"why":"Contributes AlpacaEval2 with length-controlled win rate as one of the LLM-judge benchmarks in the suite.","marker":"(Dubois et al., 2024)"},{"why":"Source of the MMLU auxiliary train data included in RE-WILD for world-knowledge coverage.","marker":"(Hendrycks et al., 2021)"},{"why":"The Qwen2.5 family, whose 72B instruct variant generates the responses that anchor the RE-WILD mix.","marker":"(Qwen Team, 2024)"},{"why":"Supplies the Llama-3.1-8B base model for all SFT runs and the Llama-3.3-70B generator used in comparisons.","marker":"(Dubey et al., 2024)"}],"fun_headline_variants":["50M chats, 50 models: smaller SFT mix beats Tulu-3","Synthetic data's key: response traits, not benchmark scores","50M synthetic chats: smaller mix beats Tulu-3's curated data","Largest chat dataset: response style predicts SFT quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that differences in downstream model performance are caused by which model generated the training data, even though generation settings were not standardized: context windows were tuned per model, and temperature values, which the paper's own appendix says substantially influence output diversity and quality, were never reported.","fun_headline_variants_meta":{"raw":{"variants":["50M chats, 50 models: smaller SFT mix beats Tulu-3","Synthetic data's key: response traits, not benchmark scores","50M synthetic chats: smaller mix beats Tulu-3's curated data","Largest chat dataset: response style predicts SFT quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2524,"prompt_tokens":893,"completion_tokens":1631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1553}},"tokens_in":509,"tokens_out":1631,"duration_ms":13289,"temperature":1.0,"reasoning_tokens":1553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:12:05.786643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six-generator comparison of Table 2 with identical generation settings, including the same temperature, the same context truncation, and the same token limits, across all generators; if the ranking of teachers by downstream SFT performance changes materially, the paper's assignment of credit to DGM identity is confounded by generation configuration. A second check is to reproduce RE-WILD's advantage over the Tulu-3 SFT mixture using only the officially released hyperparameters and an independent random seed.","supporting_citations":[],"review_version":1}