{"id":"34600d06-4a00-4b45-b45e-56faae96d9db","arxiv_id":"2412.09807","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using Llama-3.1-8B to generate and score synthetic MCQA data, then distilling those soft labels into DeBERTa-v3-base, improves few-shot MMLU accuracy from 28.9% to 39.3%.","lead":"This paper trains a small DeBERTa model for multiple choice questions by first using Llama-3.1-8B to generate synthetic practice questions and score the answer choices, then fine-tuning DeBERTa on that data. On the MMLU benchmark, this lifts accuracy from 28.9% to 39.3% with only five real examples per topic.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MMLU headline result is not protected against test-set contamination: the only semantic-similarity check (Appendix C.9) is run on ARC, not MMLU, so the reported 10.4-point gain may reflect memorized MMLU content.","rationale":"Good-faith reading: the paper's contribution is a practical recipe — use an LLM to generate 1024 MCQA items per topic from 5 examples, score choices with LLM logits, and fine-tune DeBERTa-v3-base with distillation. The MMLU result is the headline. The most load-bearing assumption is not simply that synthetic data matches the test distribution in style and difficulty; it is that the synthetic MMLU data and teacher scores are free of test-set leakage, because the teacher is a web-scale LLM that has almost certainly seen MMLU. The paper's own contamination check is limited to ARC, and even there the max-similarity numbers are high enough that the check's conclusion is not obviously safe. This is a concrete, checkable gap; it does not require assuming the authors acted improperly. If the check passes, the central claim is supported; if it fails, the 10.4-point improvement is an artifact. The reader's 'transferability' concern is closely related, but I am narrowing it to a specific failure mode that would most directly invalidate the measured accuracy. The verdict stays conditional, with the added condition that the authors provide MMLU-level similarity statistics and ideally release the generated data and code.","tokens_in":25996,"tokens_out":7419,"duration_ms":78356,"concrete_test":"Re-run the Appendix C.9 embedding analysis on the MMLU data: for each MMLU task, embed the generated questions (or a random 200-sample subset) with all-MiniLM-L6-v2 and compute the maximum cosine similarity to all MMLU test questions, exactly as in Figures 3–4. If the average maximum similarity exceeds the ARC values (0.59/0.54) or if any generated question has max cosine similarity greater than 0.9 with a test question, the reported 39.3% is suspect. As a second check, for the top-100 nearest generated–test pairs, verify whether the teacher's soft label agrees with the test answer; agreement on near-duplicates would indicate answer leakage. Also report this statistic separately for the under-generated tasks (college mathematics, formal logic, high school US history).","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Section 4.1, Table 1) is that training DeBERTa-base-v3 on 1024 LLM-generated examples per MMLU task plus LLM soft-label distillation raises MMLU accuracy from 28.9% to 39.3%. This claim requires that the generated training data and teacher scores are not contaminated by the MMLU test set. The teacher, Llama-3.1-8B-Instruct, is a web-scale LLM almost certainly pretrained on MMLU, so it can reproduce near-duplicates of test questions and supply their correct answers. The paper's only contamination analysis (Appendix C.9, Figures 3–4) is performed on ARC-Easy and ARC-Challenge, not on MMLU, the benchmark where the headline claim is made. Notably, even on ARC the maximum cosine similarity between generated and test questions reaches 0.93 (ARC-Easy) and 0.89 (ARC-Challenge); the authors interpret this as safe only by comparison with training–test similarity, but near-duplicates need not be exact. For MMLU no similarity statistic is reported, so the headline improvement may be inflated by the student memorizing test answers through the generated data and teacher scores. Additionally, Appendix B.2 shows several MMLU tasks generated far fewer than 1024 usable points (e.g., college mathematics 512, formal logic 538, high school US history 305) with JSON-parse rates below 15%; these under-generated tasks are nonetheless averaged into the headline, and no per-task contamination check is offered.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a few-shot MCQA pipeline in which an instruction-tuned LLM (Llama-3.1-8B-Instruct) generates synthetic multiple-choice questions from five seed examples, either directly in JSON or through a decomposed question/positive/negative generation scheme; the same LLM then scores the choices by extracting token logits; and the soft scores are used as distillation targets when fine-tuning DeBERTa-v3-base. The central reported result is on MMLU: JSON generation plus distillation raises DeBERTa's accuracy from 28.9% to 39.3%, a 10.4-point absolute gain, with ablations on ARC-Easy/ARC-Challenge covering generated-data size, generation temperature, generator model size, and a paraphrasing baseline.","tokens_in":26305,"tokens_out":6438,"duration_ms":68475,"significance":"If the headline result is robust, the paper offers a practical and inexpensive recipe for building an efficient encoder-only MCQA system from only five labeled examples. The work has real strengths: it ablates the number of generated points, generation temperature, and number of negative choices; it compares against a paraphrasing baseline; it includes a cross-dataset transfer experiment (Appendix C.8); and it reports results averaged over five seeds in the ablation tables. However, the two load-bearing issues below — the absence of an MMLU contamination analysis and the absence of variance on the headline MMLU table — prevent me from endorsing the central claim as stated. Both are empirically addressable, so the paper warrants major revision rather than rejection.","major_comments":[{"comment":"The headline result in Section 4.1 (the 28.9 to 39.3 MMLU gain) is not protected against test-set contamination. Appendix C.9 reports semantic similarity between generated, training, and test questions only for ARC-Easy and ARC-Challenge, not for the MMLU tasks that carry the main claim. Because the teacher model, Llama-3.1-8B-Instruct, is trained on web-scale data and almost certainly has seen MMLU test items, and because Appendix B.2 shows that several MMLU tasks produce fewer than 1024 usable points (e.g., college mathematics 512, formal logic 538, high school US history 305), the reported 10.4-point improvement could in part reflect the student memorizing test-like content through the generated questions and teacher scores. The authors should run the same similarity analysis per MMLU task, or provide an equivalent contamination check, and report the numbers before the MMLU result can be taken at face value.","section":"Section 4.1 / Appendix C.9"},{"comment":"The paper says 'We average the results across 5 different random seeds for all experiments,' but Table 1 gives no standard deviation, standard error, or per-seed values for any MMLU row. The ARC ablations in Table 4 show that the 5-shot baseline has very high seed variance (26.5 ± 13.8 on ARC-Easy), so without error bars on the MMLU numbers the 10.4-point gain over the 5-shot baseline cannot be distinguished from seed noise. Please add variance or significance measures for Table 1 and for the MMLU rows of Table 10.","section":"Section 4, Table 1"},{"comment":"The comparison with LLaMA-7B (35.1) and Flan-T5-250M (35.9) is confounded: those numbers are taken from the original papers, which use different prompting, few-shot selection, and possibly different evaluation splits, as the table footnote admits. The claims in the abstract and Section 4.1 that the distilled DeBERTa 'surpasses' or 'approaches' these models are therefore not established by a controlled experiment. The authors should either run the same evaluation protocol on LLaMA-7B and Flan-T5-250M (or an open checkpoint with the same harness) or rephrase those claims to say 'comparable to published numbers.'","section":"Section 4.1, Table 1"}],"minor_comments":[{"comment":"The phrase 'trained with only a 5-shot examples' is ungrammatical; it should be 'trained with only five examples' or 'with only 5-shot examples.' The same typo appears in Section 4.1.","section":"Abstract and Section 1"},{"comment":"There is a typo: 'We observer that' should be 'We observe that.'","section":"Appendix C.6"},{"comment":"The function name 'vmlDeviceGetMemoryInfo' appears to be a typo for 'nvmlDeviceGetMemoryInfo' (or a similar pyNVML call).","section":"Appendix B.2"},{"comment":"The main-text distillation loss in Section 3.2 uses no temperature parameter, but Appendix C.2 introduces a temperature r and reports results for different values. The paper should state explicitly which distillation temperature was used for the MMLU experiments in Section 4.","section":"Section 3.2 and Appendix C.2"},{"comment":"The manuscript does not include a link to code or the generated datasets, which limits reproducibility of the exact reported numbers. I would encourage the authors to release both.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the MMLU contamination question is the crux of this paper. The authors' own Appendix C.9 shows that even on ARC, maximum cosine similarity between generated and test questions reaches 0.93 (ARC-Easy) and 0.89 (ARC-Challenge) with a MiniLM embedding model; without the analogous MMLU numbers, the 10.4-point improvement cannot be verified as genuine generalization. I would not reject outright, because the methodology is sound in other respects and the missing evidence is empirically obtainable, but the revision should be evaluated with a per-task MMLU contamination analysis and proper variance reporting in hand."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core is real: training DeBERTa-base-v3 on LLM-generated MCQA data with soft-label distillation beats the 5-shot baseline by 10.4 points on MMLU (28.9% to 39.3%), and the ablations on ARC (generate vs. distill, data size, temperature, negative count) are honest and reasonably thorough. The decomposed generation method is a sensible practical contribution—it avoids JSON parsing failures and produces usable data at scale. The cross-dataset experiment (Appendix C.8) is a good check: training on ARC-generated data gives 37.9% on MMLU, showing the method teaches format plus some domain knowledge, not just memorized answers. The comparison to LLaMA-7B and Flan-T5-250M is appropriately hedged in the table footnote, though the introduction overclaims by saying the model 'surpasses' those baselines when the setups are not comparable.\n\nThe soft spots are the usual ones for this line of work, and they are real. First, the headline MMLU result has no error bars in Table 1, despite the paper claiming 5-seed averages. Given that the ARC results show non-trivial variance, this matters. Second, the contamination check in Appendix C.9 is run on ARC, not MMLU—the benchmark where the headline claim lives. The stress-test note is right: the teacher LLM almost certainly saw MMLU in pretraining, and near-duplicate test questions with correct answers could leak through the generated data or the teacher's soft labels. The ARC similarity numbers (max cosine 0.93 on Easy, 0.89 on Challenge) are not reassuring by themselves; the authors interpret them as safe only relative to training-test similarity, which is a weak standard. No per-task similarity statistics are reported for MMLU. Third, the JSON generation method produced far fewer than 1024 usable points on some MMLU tasks (e.g., 305 for high school US history) with parse rates below 15%, and those under-generated tasks are averaged into the headline. That is a data-quality red flag that deserves per-task reporting.\n\nThe paper is honest about its limitations and does not hide the weak spots—the appendix stats are actually informative. The method is not a paradigm shift; it combines known techniques (LLM data generation, probability-score distillation) in a specific recipe. But the recipe is practical, the ablations are instructive, and the result, if it survives a proper contamination check, would be useful for resource-constrained MCQA.\n\nWho is this for? Researchers working on small-model distillation, synthetic data generation, or few-shot QA. It deserves a serious referee: the central claim is specific, the experiments are reproducible in principle (though no code/data is released), and the contamination concern is testable. My recommendation: send it to peer review, but require error bars for the headline MMLU result, a MMLU-specific contamination analysis, and per-task data counts for the generated set. I would not desk-reject it.","headline":"A useful empirical recipe for tiny MCQA models via LLM-generated data plus soft-label distillation, but the headline MMLU number is not yet protected against contamination or verified against a rigorous baseline.","tokens_in":26903,"tokens_out":746,"would_cite":false,"duration_ms":9880,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that an encoder-only student model, DeBERTa-base-v3, improves from 28.9% to 39.3% average MMLU accuracy, a 10.4-point absolute gain, when fine-tuned on LLM-generated MCQA data whose answer choices are scored by the LLM.","keywords":["few-shot learning","multiple choice question answering","LLM distillation","soft labels","MMLU","DeBERTa","synthetic data generation","encoder-only models"],"falsifier":"Evaluate the student on held-out MMLU subjects whose topics and styles were deliberately excluded from the 5-shot prompt and from the generation prompts; if accuracy on those subjects stays near the 28.9% baseline, the synthetic distribution did not transfer. A simpler test: replace the LLM's soft labels with random permutations of the same distribution and check whether the 10.4-point gain collapses.","tokens_in":1678,"feed_emoji":"🤖","tokens_out":2266,"duration_ms":75236,"temperature":0.7,"pith_summary":"The paper asks whether a small encoder-only model can become an effective few-shot multiple-choice question answerer without needing large labeled datasets. It proposes a two-step recipe: use an LLM to generate synthetic MCQA questions and choices, and use the same LLM's probability scores over choices as soft labels when fine-tuning DeBERTa-v3-base. On MMLU this raises accuracy from 28.9% to 39.3%, a 10.4-point gain, and the 184M-parameter student outperforms much larger models reported under similar settings. The authors read this as evidence that LLM-driven data generation plus distillation transfers knowledge efficiently into small models.","feed_headline":"LLM data and soft labels lift a small model by 10.4 points on MMLU","feed_subtitle":"DeBERTa-base reaches 39.3% using only 5-shot examples and an LLM teacher.","key_machinery":"The load-bearing mechanism is the combination of LLM-based synthetic data generation and probability-score distillation. The teacher (Llama-3.1-8B-Instruct) creates MCQA instances either in JSON format or through a decomposed question/positive/negative generation pipeline, then outputs the likelihood of each answer letter, that is, a soft label over the choices. The student (DeBERTa-v3-base) is trained with a cross-entropy loss against these soft probabilities. The softness is what matters: with hard labels from the teacher the reported gain collapses, while with soft labels the student learns a smoother distribution, which the authors link to robustness against noisy generated data.","core_discovery":"On the paper's own terms, the central discovery is that knowledge distillation from LLMs works for few-shot MCQA in a specific, effective way: an LLM generates synthetic questions and choices, scores each choice with a soft probability, and those scores rather than hard labels are used to train a smaller encoder-only model. This recipe lifts DeBERTa-base-v3 from 28.9% to 39.3% on MMLU, surpassing the 5-shot performance of LLaMA-7B and Flan-T5-250M. The authors attribute the gain to the soft labels correcting noisy generated answers and effectively transferring the teacher's answer-choice distribution.","pith_inferences":["Editorial extension: if the synthetic distribution faithfully matches real test data, the recipe could generalize to any task where an LLM can propose candidate outputs and score them, with the student's ceiling bounded by the teacher's domain coverage.","Editorial extension: the large gain from soft labels over hard labels implies that other synthetic-data pipelines with noisy generation should treat teacher confidence as part of the training signal, a design choice the paper validates only for MCQA and ARC.","Editorial extension: the cross-dataset experiment suggests a substantial portion of the improvement is format learning plus broad language understanding; a sharper test would vary topic overlap while holding format constant, which the paper does not perform.","Editorial extension: because generation and scoring still require a multi-GPU LLM, the computational savings materialize at inference time, not training time, so the practical benefit is mainly for deployment on memory-constrained devices."],"forward_implications":["A 184M-parameter student beats models over 30 times larger on few-shot MMLU, so deployment cost for MCQA systems in resource-constrained settings drops sharply.","The distillation step consistently improves accuracy across data sizes, generation methods, and teacher LLMs, indicating that soft-label training stabilizes learning from imperfect synthetic data.","With roughly 1,024 generated items, the student matches the performance of training on 256 to 512 real ARC examples, reducing the need for expensive labeled MCQA data.","The JSON generation variant, despite lower parse success, yields slightly better accuracy and combines well with an existing multi-task DeBERTa baseline, suggesting an implicit quality filter.","The approach transfers across datasets (MMLU to ARC) and extends to binary classification via a logit-threshold heuristic, pointing toward broader applicability."],"supporting_citations":[{"why":"Provides the MMLU benchmark and its 5-shot dev/test setup, which is the evaluation target for the main claim.","marker":"Hendrycks et al. (2020)"},{"why":"Supplies the distillation loss framework used to transfer soft probability targets to the student.","marker":"Hinton et al. (2015)"},{"why":"Establishes the LLM scoring method that converts letter likelihoods into soft labels for MCQA choices.","marker":"Robinson & Wingate (2023)"},{"why":"Provides the Llama-3.1-8B-Instruct model used as the teacher for both data generation and scoring.","marker":"Dubey et al. (2024)"},{"why":"Defines the LLaMA-7B baseline whose few-shot MMLU performance the distilled student surpasses.","marker":"Touvron et al. (2023)"},{"why":"Defines the Flan-T5-250M baseline whose few-shot MMLU performance the distilled student surpasses.","marker":"Chung et al. (2024)"},{"why":"Provides the Tasksource DeBERTa baseline, a strong encoder-only comparison and the model that is further improved by the paper's method.","marker":"Sileo (2024)"},{"why":"Supplies the ARC-Easy and ARC-Challenge datasets used for the ablation studies that isolate the effect of data size, generation method, and distillation.","marker":"Clark et al. (2018)"}],"fun_headline_variants":["LLM teacher boosts small model by 10.4 points on MMLU","Soft labels from LLMs power few-shot MCQ accuracy jump","Distilling LLM knowledge lifts DeBERTa to 39.3% on MMLU","LLM-generated data and soft scores beat direct fine-tuning"],"cache_read_input_tokens":28928,"weakest_assumption_plain":"The method assumes that the LLM-generated questions and choices, together with the LLM's probability scores, are a faithful proxy for the real MMLU test distribution, so that a student trained on this synthetic data learns genuine question-answering ability rather than overfitting to the synthetic data's quirks.","fun_headline_variants_meta":{"raw":{"variants":["LLM teacher boosts small model by 10.4 points on MMLU","Soft labels from LLMs power few-shot MCQ accuracy jump","Distilling LLM knowledge lifts DeBERTa to 39.3% on MMLU","LLM-generated data and soft scores beat direct fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1423,"prompt_tokens":888,"completion_tokens":535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":455}},"tokens_in":504,"tokens_out":535,"duration_ms":5041,"temperature":1.0,"reasoning_tokens":455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:42:40.732181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the student on held-out MMLU subjects whose topics and styles were deliberately excluded from the 5-shot prompt and from the generation prompts; if accuracy on those subjects stays near the 28.9% baseline, the synthetic distribution did not transfer. A simpler test: replace the LLM's soft labels with random permutations of the same distribution and check whether the 10.4-point gain collapses.","supporting_citations":[{"cited_title":"Leveraging large language models for multiple choice question answering","cited_arxiv_id":null,"evidence_quote":"Establishes the LLM scoring method that converts letter likelihoods into soft labels for MCQA choices."},{"cited_title":"tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework","cited_arxiv_id":null,"evidence_quote":"Provides the Tasksource DeBERTa baseline, a strong encoder-only comparison and the model that is further improved by the paper's method."}],"review_version":1}