{"id":"ef4c5423-2f11-42b3-bb5f-b34ffa69671c","arxiv_id":"2502.01697","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BARE generates diverse, high-quality synthetic training data from only three seed examples by having a base model draft and an instruction-tuned model refine, improving downstream fine-tuning accuracy in few-shot settings.","lead":"This paper introduces BARE, which uses a base language model to draft diverse synthetic examples from just three seed examples, then an instruction-tuned model to refine them into high-quality training data. It reports that fine-tuning small models on BARE-generated data outperforms training on data from instruction-tuned models alone, and even beats one human-curated math dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run fine-tuning results without error bars leave the key causal gap (35.8 vs 30.8 on GSM8K, 28.1 vs 26.0 on LCB) within plausible noise; the role of base-model diversity is not yet established.","rationale":"I agree with the reader's conditional verdict. The method is well motivated, the Instruct-Instruct ablation is the right control, and the diversity/IR measurements are suggestive. However, the central claim that BARE's downstream gains come from base-model diversity hinges on a few point estimates that are within plausible noise for single runs. The GSM8K 35.8 vs 30.8 gap is the cleanest test of the mechanism; at n=500 the binomial SE is about 2.1 points, so 5 points is ~2.4 SE, and the actual variance is larger once data-generation and fine-tuning randomness are included. The LCB comparison is even weaker. The Figure 9/Table 6 discrepancy on PubMedQA is an independent internal inconsistency that needs a correction. None of this disproves BARE, but it means the paper's strongest claims should not be taken as established until multi-seed estimates and the inconsistency are addressed. Hence CONDITIONAL remains the right verdict.","tokens_in":20413,"tokens_out":12374,"duration_ms":115230,"concrete_test":"Run 5 independent data-generation seeds × 3 LoRA fine-tuning seeds for the four GSM8K conditions that isolate the mechanism — Llama-3.1-70B-Instruct direct, GPT-4o direct, GPT-4o refining Llama-3.1-70B-Instruct, and BARE (Llama-3.1-70B-Base + GPT-4o) — and for the LCB TOP pair BARE(Llama-3.1-8B) vs Llama-3.1-70B-Instruct. Report mean ± 95% CI over seeds. If the BARE-vs-GPT-4o-refine-Instruct GSM8K gap falls below 2 points or the LCB difference changes sign, the causal role of the base model is not established. Also re-extract the PubMedQA result behind Figure 9 from the exact data-generation/refinement run and compare to Table 6 to determine whether 0.79 or 0.72 is the correct value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All downstream numbers are single runs with no error bars, and the most decision-relevant comparison is not the headline 35.8 vs 17.8 (which confounds refinement with generator choice) but the Instruct-Instruct control in Table 12 (§B.4): GPT-4o refining Llama-3.1-70B-Instruct data reaches 30.8% on GSM8K, while BARE with Llama-3.1-70B-Base + GPT-4o reaches 35.8%. That 5-point gap on n=500 is only ~2.4 standard errors for the binomial component alone, before adding variance from data generation and LoRA fine-tuning. On LCB TOP the BARE 8B result (28.1%) versus Llama-3.1-70B-Instruct (26.0%) is ~0.7 SE on n=442. Thus the specific claim that base-model diversity (rather than the extra refinement pass) drives downstream gains is not yet distinguishable from run-to-run noise. Separately, Figure 9 reports PubMedQA accuracy of 0.79 for BARE (Llama-3.1-8B-Base + GPT-4o), but Appendix Table 6 reports 0.72 for the same configuration; if 0.72 is the correct value, BARE does not beat GPT-4o independent prompting (0.71) on that domain. This internal inconsistency must be resolved before the aggregate claim is accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BARE (Base-Refine), a two-stage method for few-shot synthetic data generation: a base language model generates a diverse set of candidate examples from three seed examples, and an instruction-tuned model refines each candidate for quality. The authors report that base-model generations are more diverse, that instruction-tuned generations are higher quality, and that fine-tuning small models on BARE-generated data improves downstream accuracy relative to instruction-tuned-only generation across six domains, including LiveCodeBench TOP, GSM8K, and RAFT-based RAG tasks.","tokens_in":20704,"tokens_out":9286,"duration_ms":80696,"significance":"The proposed setting is well motivated, and the central insight—that base models can supply the diversity that is normally bought with very large seed sets—is cleanly articulated. The Instruct-Instruct ablation in Appendix B.4 is an important control: it shows that replacing the base generator with an instruction-tuned generator lowers downstream accuracy, so the reported effect is not simply an artifact of the two-stage pipeline. If the results are stable under repeated trials, BARE would be a practical contribution to few-shot synthetic data generation. The main obstacles are experimental: the absence of uncertainty quantification and an internal numeric inconsistency in the PubMedQA results.","major_comments":[{"comment":"All downstream fine-tuning results are single runs without error bars or multiple seeds, and the smallest evaluation sets are static n=100 test sets (HotpotQA and PubMedQA). The most decision-relevant comparison is not the headline 35.8% versus 17.8% on GSM8K (which confounds the choice of generator with the extra refinement pass) but the Instruct-Instruct control in Table 12: GPT-4o refining Llama-3.1-70B-Instruct data reaches 30.8%, while BARE with Llama-3.1-70B-Base plus GPT-4o reaches 35.8%; on n=500 that 5-point gap is roughly 1.7 binomial standard errors, before adding variance from data generation and LoRA fine-tuning. On LCB TOP, the 28.1% versus 26.0% comparison on n=442 is about 0.7 standard error. The paper should report repeated runs and confidence intervals, or otherwise demonstrate that the reported gaps are outside run-to-run noise.","section":"Section 5 and Appendix B.2, Tables 5–8"},{"comment":"The same configuration (BARE with Llama-3.1-8B-Base generator and GPT-4o refiner) is reported as 0.79 accuracy on PubMedQA in Figure 9 and 0.72 in Table 6. If the correct value is 0.72, BARE does not outperform GPT-4o independent prompting (0.71) on PubMedQA, and the statements in Section 5 that BARE outperforms all GPT-4o prompting methods and also outperforms existing RAFT pipelines on PubMedQA are not supported. The discrepancy must be resolved before the aggregate claims can be accepted.","section":"Figure 9 versus Appendix Table 6"},{"comment":"The motivating claim that base models are more diverse is not universal: on LCB TOP, Llama-3.1-70B-Base has average pairwise cosine similarity 0.468, which is higher than the 0.389 of Llama-3.1-70B-Instruct, i.e., less diverse. The paper attributes the reversal to the base model repeating phrases from the examples, but no quantitative evidence is provided. Because diversity is the paper's proposed mechanism for downstream gains, this exception should be analyzed explicitly (for example, by recomputing similarity after removing verbatim repeated spans or by showing representative samples), rather than only acknowledged in a short remark.","section":"Section 3.3 and Table 1"}],"minor_comments":[{"comment":"The temperature sweep on PubMedQA shows Llama-3.1-8B-Instruct at temperature 0.5 reaching 75% accuracy, which is above BARE's 73% in the same table; this contradicts the claim in Appendix B.3 that temperature gains are 'small relative to gains by using BARE.'","section":"Appendix B.3, Table 10"},{"comment":"The phrase 'all over methods' should presumably be 'all other methods'; more generally, the paper mixes fractional and percentage reporting (e.g., 0.79 in Figure 9 versus 28.1% in Figure 8), which makes numeric cross-checks harder.","section":"Figure 9 caption"},{"comment":"The statement that 'across all domains training with BARE-generated data leads to the highest model accuracy' is too strong, since Table 3 (Enron, BARE with Llama-3.1-70B: 0.771 F1 versus 0.848 for 70B Instruct) and Table 5 (HotpotQA, BARE with Llama-3.1-70B: 0.54 versus 0.55 for 70B Instruct) show BARE not being best.","section":"Section 5"},{"comment":"There is a typo, 'temperauter' should be 'temperature'.","section":"Appendix A"},{"comment":"The indistinguishability-rate prompt asks the discriminator to identify the 'low quality' example rather than the 'synthetic' example; this conflation should be stated explicitly as a design choice, since it may affect the interpretation of the reported IR values.","section":"Appendix D, IR prompt"},{"comment":"No code or data-generation scripts are provided; for a method whose value is empirical, releasing generation and fine-tuning scripts would materially aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is worth publishing once the experimental rigor is addressed. The main blockers are the absence of repeated trials, the unresolved PubMedQA discrepancy, and the LCB TOP diversity exception; all appear fixable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the core idea—use a base model for diversity, an instruction-tuned model for refinement, with only three seed examples—is genuinely new and worth building on. The authors run the control that matters: swapping the base generator for an instruction-tuned generator (Appendix B.4) drops GSM8K accuracy from 35.8 to 30.8 with GPT-4o refinement. That is the right experiment, and it points where they say it does. The seed-size scaling plot and the temperature ablations are also useful, and the paper is honest about domains where BARE does not help (e.g., 70B family on HotpotQA and LCB).\n\nThe soft spots are real, though. Everything downstream is a single run with no error bars. The key causal gap is 5 points on n=500; the binomial component alone is about 2.4 standard errors, before adding variance from data generation and LoRA fine-tuning. That is suggestive, not conclusive. On LCB TOP, the BARE 8B result (28.1) is only 2 points above Llama-3.1-70B-Instruct (26.0) on n=442, which could easily flip. And there is an internal inconsistency you must fix: Figure 9 shows PubMedQA at 0.79 for BARE with Llama-3.1-8B-Base + GPT-4o, but Table 6 lists 0.72 for that same configuration. If 0.72 is right, BARE does not beat GPT-4o independent prompting (0.71) on that domain, and the abstract's across-the-board claim loses support. The \"comparable to state-of-the-art\" LiveCodeBench wording also overstates things: 28.1 vs DeepSeekCoder's 32.7 is a real gap, even if the seed-data comparison favors BARE.\n\nNone of this means the method is wrong. The Instruct-Instruct ablation is the right control and the effect is in the expected direction. But the paper asks the reader to take the causal story on a few single runs, and one of the displayed numbers contradicts its own appendix. That has to be resolved before publication.\n\nWho is this for: anyone working on synthetic data generation, efficient fine-tuning, or low-resource domain adaptation. It is a solid empirical contribution that deserves a serious referee. My recommendation: send it to peer review, but condition acceptance on multiple seeds, corrected figures, and either error bars or a clearly labeled preliminary-noise discussion.","headline":"A genuinely new few-shot synthetic data recipe with the right control experiment, but single-run numbers and one internal inconsistency make the headline gains look shakier than the prose suggests.","tokens_in":21266,"tokens_out":1735,"would_cite":true,"duration_ms":18489,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Base models beat instruct models at few-shot data generation","keywords":["synthetic data generation","few-shot learning","base language models","instruction-tuned models","data diversity","data quality","Base-Refine","fine-tuning"],"falsifier":"Run the GSM8K comparison, BARE with Llama-3.1-70B-Base plus GPT-4o refinement versus instruct-only generation, for ten independent fine-tuning seeds and check whether the 35.8% versus 17.8% gap persists with non-overlapping confidence intervals; if the gap collapses, the central claim that BARE improves downstream performance is not established.","tokens_in":20206,"feed_emoji":"🤖","tokens_out":5268,"duration_ms":42013,"temperature":0.7,"pith_summary":"The paper claims that the standard practice of generating synthetic training data with instruction-tuned models fails in the few-shot regime because these models produce too little diversity, and that untuned base models, which have been largely ignored for this purpose, are a better source of variety. On this basis it introduces Base-Refine (BARE), a two-stage method that first samples a large, diverse set of candidate examples from a base model using just three seed examples, then has an instruction-tuned model refine each candidate for quality. The paper reports that fine-tuning a Llama 3.1 8B model on 1,000 BARE-generated coding examples reaches accuracy comparable to state-of-the-art similarly sized models on LiveCodeBench, and that BARE-generated math data improves a fine-tuned Llama 3.2 1B on GSM8K by 101% over instruction-model-only data. A sympathetic reader would care because this points to a data-efficient way to create training sets in domains where curated examples are scarce or expensive.","feed_headline":"Base models beat instruct models at few-shot data generation","feed_subtitle":"A two-stage pipeline turns 3 seed examples into datasets that top instruction-only generation on math and code.","key_machinery":"The load-bearing object is the two-stage Base-Refine (BARE) pipeline: a base language model, such as Llama-3.1-70B-Base, independently samples candidate examples conditioned on three seed examples at temperature 0.7, and then an instruction-tuned model, such as Llama-3.1-70B-Instruct or GPT-4o, edits each candidate to improve realism and correctness while keeping its theme. The mechanism is the division of labour: diversity is inherited from the base model's untuned output distribution, and quality is imposed per entry by the refiner, so the final dataset keeps the base model's spread while gaining instruction-tuned fidelity. The paper measures spread with average pairwise embedding cosine similarity and quality with indistinguishability rate (IR), the rate at which a strong LLM discriminator fails to tell a synthetic entry apart from real examples.","core_discovery":"The central discovery is a diversity-quality trade-off between model checkpoints: base models, which have not undergone post-training, generate far more diverse outputs from a few seed examples, while instruction-tuned models generate more realistic, in-distribution examples but collapse toward a narrow mode. BARE exploits this trade-off by separating generation from refinement: the base model supplies spread, and the instruction-tuned model repairs correctness, style, and ambiguity without removing the original diversity. The paper demonstrates that this separation is what matters by ablating the first stage: swapping the base generator for an instruction-tuned generator while keeping the same refine step lowers GSM8K accuracy from 29.8% to 25.4% with a Llama refiner and from 35.8% to 30.8% with GPT-4o, while dataset diversity stays similar. The paper concludes that base-model priors are an effective and largely untapped source of seed-efficient diversity for synthetic data.","pith_inferences":["If the diversity-quality trade-off is general, then model families that have not been post-trained, including older or smaller checkpoints, could be recycled as cheap generators in low-resource domains without requiring a large curated seed pool.","The indistinguishability rate metric could be turned into a filtering criterion inside BARE: instead of refining all base samples, a discriminator could first select the most realistic candidates, potentially improving data efficiency further.","The convergence of BARE and instruct-only methods near 100 seed examples suggests the base model's diversity advantage matters most when seed data is scarcest; in high-resource settings the extra refinement cost may not pay off.","A natural stress test is to apply BARE to specialized domains such as clinical or legal text, where large seed sets are hardest to obtain and instruction-tuned-only methods would be expected to collapse to narrow modes."],"forward_implications":["With only 3 seed examples, BARE-generated data outperforms data from either base-only or instruct-only generation on downstream fine-tuning across all four generative domains tested: HotpotQA, PubMedQA, GSM8K, and LiveCodeBench.","On LiveCodeBench Test Output Prediction, fine-tuning Llama-3.1-8B-Instruct on 1,000 BARE examples reaches 28.1% accuracy, close to top similar-size models that were trained with seed sets of tens of thousands of examples.","BARE improves the RAFT pipeline for RAG data generation by up to 18.4% on HotpotQA with the 8B model family, setting a new state of the art in that pipeline.","On GSM8K, BARE is the only tested generation method whose data improves a fine-tuned Llama-3.2-1B model over its untrained baseline, and with GPT-4o refinement it reaches 35.8%, above the 26.6% obtained from the official human training set.","The advantage of BARE over instruct-only generation persists as the seed set grows, with the two methods converging around 100 seed examples, about 10% of the generated dataset size."],"supporting_citations":[{"why":"Supplies the Llama 3.1 base and instruct models used as generators, refiners, and fine-tuning backbones.","marker":"[Dubey et al., 2024]"},{"why":"Provides the GSM8K seed examples and evaluation set used for math reasoning experiments.","marker":"[Cobbe et al., 2021]"},{"why":"Provides the LiveCodeBench Test Output Prediction benchmark used for code evaluation.","marker":"[Jain et al., 2024]"},{"why":"Defines the RAFT baseline for RAG synthetic data generation that BARE modifies and improves.","marker":"[Zhang et al., 2024b]"},{"why":"Represents the large-seed-set OSS-Instruct baseline (80,000 code snippets) that BARE compares against in the few-shot setting.","marker":"[Wei et al., 2024]"},{"why":"Motivates the concern that low-diversity synthetic data leads to model collapse, framing why diversity matters.","marker":"[Shumailov et al., 2024]"},{"why":"Provides evidence that diversity in training data improves downstream performance.","marker":"[Chen et al., 2024]"},{"why":"Provides the LoRA fine-tuning method used in all downstream evaluation runs.","marker":"[Hu et al., 2021]"},{"why":"Supplies the text-embedding-3-small model used to compute the diversity metric.","marker":"[OpenAI, 2024a]"},{"why":"Inspires the adversarial indistinguishability rate metric used to measure entry-wise data quality.","marker":"[Goodfellow et al., 2014]"}],"fun_headline_variants":["Base models generate more diverse few-shot data","Two-stage BARE turns 3 seeds into high-quality datasets","Few-shot data generation: base models win on diversity","BARE: base model diversity enables few-shot synthetic data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the reported downstream accuracy differences are real and not run-to-run noise: every fine-tuning result is a single run with no error bars, and HotpotQA and PubMedQA are evaluated on static test sets of only 100 items.","fun_headline_variants_meta":{"raw":{"variants":["Base models generate more diverse few-shot data","Two-stage BARE turns 3 seeds into high-quality datasets","Few-shot data generation: base models win on diversity","BARE: base model diversity enables few-shot synthetic data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001905,"raw_usage":{"total_tokens":7515,"prompt_tokens":1044,"completion_tokens":6471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":6407}},"tokens_in":660,"tokens_out":6471,"duration_ms":50083,"temperature":1.0,"reasoning_tokens":6407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:03:37.011566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the GSM8K comparison, BARE with Llama-3.1-70B-Base plus GPT-4o refinement versus instruct-only generation, for ten independent fine-tuning seeds and check whether the 35.8% versus 17.8% gap persists with non-overlapping confidence intervals; if the gap collapses, the central claim that BARE improves downstream performance is not established.","supporting_citations":[{"cited_title":"The Llama 3 herd of models, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the Llama 3.1 base and instruct models used as generators, refiners, and fine-tuning backbones."}],"review_version":1}