{"id":"807a4b33-e5f6-4a45-8242-cbac4925c706","arxiv_id":"2506.03541","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A debate-and-reflect distillation pipeline with tree-structured DPO improves small-model accuracy on MMLU Pro and MATH compared with standard distillation baselines.","lead":"This paper proposes a debate-and-reflect (D&R) pipeline in which a small student model argues with stronger teacher models, receives self-reflections and teacher feedback, and is fine-tuned using a tree-structured preference objective. The authors report large accuracy gains over standard distillation baselines on three MMLU Pro categories and MATH.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains rest on single unseeded runs; the T-DPO-specific advantage over RPO is 0.02 on average and negative on MATH, so the central quantitative claims are not yet established.","rationale":"The reader's CONDITIONAL verdict is reasonable, and I do not see a fatal internal flaw. However, the most load-bearing weakness is not the debate-log data-quality assumption, but the statistical fragility of the headline numbers. The SFT-only gap over the best single teacher is small (0.79), and the T-DPO-vs-RPO comparison is essentially a tie (0.02 average), while T-DPO is worse than SFT on MATH. These patterns make the claim that T-DPO is a key contributor and that D&R reliably beats single-teacher distillation depend on run-to-run noise. A multi-seed rerun with paired comparisons is the minimal check that would settle this. The MMLU-Pro train/test split is a genuine external-validity concern as well, but the immediate blocking issue is whether the reported point estimates are reproducible and whether the proposed components are responsible for them.","tokens_in":17799,"tokens_out":12686,"duration_ms":153957,"concrete_test":"Re-run the full D&R pipeline (SFT and T-DPO) plus the key controls—D&R_SFT, Single-Teacher Claude3.5, and D&R with RPO—on the same fixed debate logs, with at least 5 independent training seeds, and report mean±std and paired differences on each benchmark. If the average D&R-minus-Single-Teacher-Claude gap is not consistently ≥2.95, or if the T-DPO-minus-SFT gain is not consistently positive on MATH and CS, the central quantitative claims do not survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—D&R with T-DPO gives an average improvement of 14.18 over Mistral-7B-Instruct and beats the best single-teacher baseline by at least 2.95—rests on one training run per condition, with no error bars, no seed variation, and no significance testing. This is load-bearing because the component attributions are fragile. D&R_SFT beats Single-Teacher Claude3.5 by only 0.79 average points (36.00 vs 35.21), so most of the claimed advantage comes from the T-DPO stage. Yet when T-DPO is replaced by the existing RPO objective on the same preference trees, the average is 38.14 vs 38.16 (Table 4), a 0.02 difference; on MATH, T-DPO is 0.32 worse than D&R_SFT (Table 1). With MMLU-Pro test sizes of only 205–325 per category, the reported 2–4 point category-level gaps are within one binomial standard error. Without repeated seeds or paired significance tests, the headline margins over single-teacher distillation and the specific contribution of T-DPO are not quantitatively secured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents D&R, a multi-agent distillation framework in which a Mistral-7B student model debates three commercial teacher models over up to four rounds, generating self-reflections and teacher feedback, and records the interactions in a Multi-Agent Interaction Graph. From these graphs the authors extract SFT instances and preference trees, then train the student with SFT followed by T-DPO, a DPO variant that conditions on the previous debate round's structured information. Experiments on three MMLU-Pro categories and MATH compare D&R with no distillation, single-teacher SFT distillation, and a reimplemented MAGDI baseline. The reported results give an average gain of 14.18 points over Mistral-7B-Instruct, an advantage of at least 2.95 points over the best single-teacher baseline, and a 2.16-point additional gain from T-DPO over SFT-only D&R.","tokens_in":18096,"tokens_out":11259,"duration_ms":106597,"significance":"If the reported gains hold under replication, the D&R framework is a practically useful contribution: it converts an expensive multi-agent debate process into a one-time training signal, releases the student from live teacher involvement at inference, and provides concrete details (model versions, prompt templates, hyperparameters, token costs) that support reproduction. The conversion of interaction graphs into preference trees and the transfer experiment to a stronger student (Llama-3.1-8B) are useful additions. The data construction uses external gold labels and held-out evaluation, so I do not see a derivation-from-fit circularity. The main unresolved question is quantitative: the central comparisons rest on single runs per condition with no reported seed variation on small MMLU-Pro test splits, and the specific advantage of T-DPO over an existing preference objective (RPO) is at most 0.02 points on average, so the strength of the empirical claims currently exceeds the evidence.","major_comments":[{"comment":"The headline comparisons are reported without error bars, multiple seeds, or significance tests. The MMLU-Pro test splits are small (205, 325, and 239 problems), so the per-category differences of 2.44 to 4.60 points against Single-Teacher Claude3.5 are within roughly one or two binomial standard errors; the 'at least 2.95 average improvement' claim is carried by pooling over categories and MATH. I ask the authors to report multiple training/evaluation seeds or paired bootstrap confidence intervals for the main comparisons, and to state how many test examples each accuracy number is based on.","section":"§4.2, Table 1, Appendix A Table 5"},{"comment":"The conclusion that T-DPO is 'the most suitable objective' for debate distillation is not supported by Table 4. T-DPO exceeds RPO by only 0.02 points on average (38.16 vs. 38.14), and on MATH the T-DPO model is 0.32 points below the SFT-only model (17.32 vs. 17.64). Without variance estimates or a significance test, this cannot separate T-DPO from RPO, and the 2.16-point contribution of T-DPO over SFT may not be robust. The experiment should be replicated with multiple seeds and, ideally, with a small hyperparameter search for RPO rather than a single alpha=1 setting.","section":"§4.3.5, Table 4"},{"comment":"The authors split the MMLU-Pro test set into training and test portions because no dedicated training set exists, but the paper does not justify why this split yields a valid held-out evaluation. Using part of the official test set as training material raises the risk of distributional leakage or contamination; a defense of the split, or an additional evaluation on an independent benchmark, is needed to support the generalization claims drawn from MMLU-Pro.","section":"§4.1, Appendix A"},{"comment":"The paper explicitly limits its evaluation to final-answer correctness rather than the validity of reasoning paths. Because the stated contribution is about transferring and improving reasoning, this limitation is load-bearing: the accuracy gains could in principle come from better answer selection or format changes rather than from the reasoning improvements the framework is claimed to produce. The authors should add process-level evidence (e.g., reasoning-step accuracy, correction quality, or a qualitative analysis of reasoning paths) or explicitly soften the reasoning claims.","section":"Limitations section"},{"comment":"The ablation of self-reflection and teacher feedback is reported under an 'Additional Correction Accuracy' protocol that mixes inference-time correction methods with the training-data ablation. The table does not state which dataset or split it refers to, and the 'without SR/TF' rows are compared against different reference points in different correction columns. To make the contribution of SR and TF transparent, the same ablation should be reported on the main evaluation protocol (the setting used in Table 1) for each of the four benchmarks.","section":"§4.3.2, Table 2"},{"comment":"The claim that student-in-the-loop debate and SR/TF produce higher-quality training data is used to explain the 11.01-point gain over MAGDI, but the only direct evidence is one qualitative example in Appendix B; the ablations in Table 2 measure the effect of removing SR/TF training data under inference-time correction, not data quality directly. A quantitative comparison of learning signals (e.g., training D&R on teacher-only debate logs that lack student participation and SR/TF) would make this explanation load-bearing.","section":"§4.2, Appendix B"}],"minor_comments":[{"comment":"Typographical errors such as 'a novelDebate andReflect' should be corrected throughout the paper.","section":"Abstract and §1"},{"comment":"The header 'Multi-Teaher Distillation' contains a typo; also, the MAGDI row appears to use inconsistent formatting for the model name.","section":"Table 1"},{"comment":"The efficiency comparison reports average token costs without variance or sample sizes; a brief note on stability would help, since the Pareto-frontier claim in Figure 3 depends on these means.","section":"§4.3.1, Table 6"},{"comment":"The prompt template contains a typo ('Fianl answer') in the incorrect-answer placeholder; this should be fixed for reproducibility.","section":"Appendix F, Figure 7"},{"comment":"The preference-tree construction involves several choices (how many previous-round nodes to include, how to segment long structured information, when to create a new tree) that are not ablated; a short discussion of their sensitivity would strengthen the method section.","section":"Appendix E, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the ideas are reasonable, but the empirical core needs strengthening before I can support acceptance. I would particularly encourage the editor to require either additional seeds with significance testing or a clear statement of the reproducibility constraints, and to ask the authors to address the MMLU-Pro train/test split issue explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this is a competently executed paper with a genuinely new integration, but the headline numbers are not yet established. The specific contribution of T-DPO over the existing RPO objective is 0.02 points on average, and every result in the paper comes from a single unseeded run.\n\nWhat's new: D&R puts the student model into the debate loop with self-reflection and teacher feedback, then turns the debate logs into preference trees and trains with DPO. MAGDI used multi-teacher debate graphs but no student-in-the-loop feedback; Eurus and CPO used tree-structured DPO but not on debate logs. That combination is new, and the ablations in Table 2 give real—if thin—evidence that self-reflection and teacher feedback matter: removing them costs 5+ points on physics. I also give credit for the honest limitations section, the attempt with Llama-3.1-8B to test the capacity-bottleneck explanation for the MATH result, and the token-cost comparison.\n\nSoft spots, in order. First, no error bars, no multiple seeds, no significance tests anywhere. MMLU Pro category-level test sizes are 205-325 questions; a 2-4 point gap is within one binomial standard error. The paper says 'significantly outperforms' but nothing in it is statistically tested. Second, splitting the MMLU Pro test set for training is a real benchmark-hygiene issue; even with a held-out part, training on a public test set needs more discussion. Third, the quantitative case for T-DPO collapses under inspection. Table 4 shows T-DPO beats RPO by 0.02 on average (38.16 vs 38.14), and on MATH it is 0.32 below the SFT-only stage. The paper's claim that T-DPO is 'most suitable' is not supported by its own table. Fourth, the core training data—debate logs and preference trees—is not released, so the quality claim in Appendix B cannot be checked beyond the one example. The efficiency result is largely a tautology: a distilled model uses fewer tokens than live debate.\n\nOverall: the method is plausible and the experiments are consistent with 'this works', but the magnitude claims and component attribution are not secured. This deserves a serious referee, but the paper needs seeds, paired significance tests, and data release before the strong claims can stand. Who is this for? Researchers working on distillation of reasoning into small models; they'll read it as a proof-of-concept, not as a benchmark result.","headline":"Useful integration of debate, feedback, and tree-structured DPO, but the headline margins are not yet secured: T-DPO's edge over RPO is 0.02 on average, and the runs are unseeded.","tokens_in":18595,"tokens_out":3167,"would_cite":false,"duration_ms":33601,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small language model that debates stronger teachers, then trains on preference trees extracted from those debates, outperforms single-teacher distillation on knowledge and reasoning benchmarks.","keywords":["knowledge distillation","multi-agent debate","direct preference optimization","tree-structured preference optimization","self-reflection","teacher feedback","small language models","reasoning benchmarks"],"falsifier":"Run the same T-DPO pipeline on a teacher-only debate dataset of equal size (no student in the loop, no self-reflection or teacher feedback) and compare; if that dataset matches or beats D&R's results, the student-in-the-loop and feedback data are not what drives the reported gains.","tokens_in":17584,"feed_emoji":"🤖","tokens_out":5042,"duration_ms":47236,"temperature":0.7,"pith_summary":"This paper claims that a small language model can be made substantially better at knowledge and reasoning tasks by letting it debate stronger teacher models, collecting the debate interactions as structured logs, and distilling those logs back into the student's weights. The key move is to include the student itself in the debate, along with explicit self-reflection from the student and corrective feedback from teachers, so the training signal targets the student's actual mistakes. The paper then converts each debate log into a 'preference tree' and trains with a tree-structured variant of Direct Preference Optimization (T-DPO), so the student learns not just the right answer but why one reasoning path is preferred over another. On the MMLU-Pro and MATH benchmarks, the authors report an average gain of 14.18 points over the original Mistral-7B-Instruct, an edge of at least 2.95 points over the best single-teacher distillation, and an extra 2.16 points from T-DPO over supervised fine-tuning alone. If these results hold, distillation becomes a cheaper and more targeted route to capable small models than static teacher-generated training data.","feed_headline":"Debating teachers lifts a 7B model 14 points on reasoning tests","feed_subtitle":"Student-in-the-loop debates plus tree-structured preference optimization beat single-teacher distillation on MMLU-Pro and MATH.","key_machinery":"The central object is the preference tree extracted from a Multi-Agent Interaction Graph (MAG), a record of the debate. Each tree root is the input prompt plus the structured information from the previous debate round (prior responses, self-reflections, and teacher feedback); its children are the correct and incorrect responses generated in that round. T-DPO applies the standard DPO objective to these trees, treating the correct response as the chosen completion and the incorrect one as the rejected completion. The machinery does two jobs at once: it preserves the debate's error-correction history in the prompt, and it trains the student to prefer the reasoning path that was actually corrected, rather than only imitating final answers.","core_discovery":"The paper's central discovery claim is that the quality of distillation data matters more than the distillation objective alone, and that debate is a way to manufacture high-quality data targeted at a specific student's weaknesses. Concretely, the authors claim that when a student model engages in multi-round debates with stronger teachers, and the debate includes the student's self-reflection and teachers' explicit feedback on the student's errors, the resulting logs support better distillation than teacher-only debate logs. The tree-structured objective T-DPO then leverages the hierarchical structure of these logs: each preference tree has the previous round's structured information as the root, a correct response as the chosen node, and an incorrect response as the rejected node, so the student learns from both the contrast and the reasoning history. The reported experiments show D&R with T-DPO beating single-teacher SFT distillation, the multi-teacher graph-distillation baseline MAGDI, and its own SFT-only variant, while also lowering per-problem token cost at inference.","pith_inferences":["The same preference-tree construction could be applied to logs from other interactive settings, such as code repair, tool-use dialogues, or human-in-the-loop corrections, not only to knowledge and reasoning benchmarks.","The RPO comparison hints that rising chosen-completion log probabilities do not reliably predict downstream accuracy; a cheaper proxy for preference-tree quality could be tested directly.","A direct test of the data-quality premise would be to measure whether later debate rounds produce more correct responses than earlier ones, or whether self-reflections actually identify the specific error that teacher feedback then corrects; the paper reports neither.","The Llama-3.1 result suggests the MATH bottleneck with Mistral is a capacity effect rather than a debate-data effect, but this is only tested on one additional student model and one dataset."],"forward_implications":["Small models can reach accuracy close to the debate-level scores of the API teachers on the tested categories: D&R brings Mistral-7B from a 23.98 average to 38.16, while the debate itself scores 84.29 to 90.38 on MMLU Pro categories.","Including the student in the debate and requiring explicit self-reflection and teacher feedback is what enables the distilled model to correct its own mistakes at inference; ablations that remove this data drop accuracy by up to 6.46 points and the model no longer self-corrects.","T-DPO adds consistent gains over SFT across data scales (100 to 974 debates) and across two student models, so the benefit is not tied to one dataset size or one student architecture.","Distilled D&R models are more token-efficient than running a debate at inference and also than the compared distillation baselines, because the student learns more concise reasoning paths.","A stronger student model continues to benefit from T-DPO on MATH where Mistral plateaus, suggesting the framework transfers to students with larger reasoning capacity."],"supporting_citations":[{"why":"Supplies the MAGDI baseline that D&R is compared against, and the Multi-Agent Interaction Graph construction that D&R reuses to record debates.","marker":"(Chen et al., 2024)"},{"why":"Provides the Direct Preference Optimization objective that T-DPO generalizes from pairs to tree-structured debate logs.","marker":"(Rafailov et al., 2023)"},{"why":"Supplies the preference-tree idea that motivates T-DPO's hierarchical organization of reasoning paths.","marker":"(Yuan et al., 2025)"},{"why":"Provides another tree-structured preference optimization approach that T-DPO builds on for chain-of-thought reasoning.","marker":"(Zhang et al., 2024a)"},{"why":"Defines the MATH benchmark and its training data and evaluation protocol used for the mathematical reasoning experiments.","marker":"(Hendrycks et al., 2021b)"},{"why":"Defines the MMLU-Pro benchmark whose computer science, physics, and biology categories are used for the knowledge and reasoning experiments.","marker":"(Wang et al., 2024b)"},{"why":"Supplies RPO, the alternative DPO-style reasoning objective used in the comparison showing T-DPO is a better fit for debate distillation.","marker":"(Pang et al., 2024)"},{"why":"Provides the LoRA parameter-efficient adaptation method used in all SFT and T-DPO training runs.","marker":"(Hu et al., 2022)"}],"fun_headline_variants":["Debate logs beat teacher-only distillation for small LLMs","Tree-structured preferences boost debate-based model distillation","Student-teacher debates yield better data for efficient LLM training","Multi-agent debate plus tree-structure lifts small-model reasoning","Debate, reflect, distill: 7B model gains 14 points on reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that debate logs that include the student model plus self-reflection and teacher feedback are higher-quality training data than teacher-only debate logs; this is supported by a qualitative example in the appendix, not by a quantitative measure of data quality.","fun_headline_variants_meta":{"raw":{"variants":["Debate logs beat teacher-only distillation for small LLMs","Tree-structured preferences boost debate-based model distillation","Student-teacher debates yield better data for efficient LLM training","Multi-agent debate plus tree-structure lifts small-model reasoning","Debate, reflect, distill: 7B model gains 14 points on reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1326,"prompt_tokens":914,"completion_tokens":412,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":530,"tokens_out":412,"duration_ms":4718,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:00:16.903040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same T-DPO pipeline on a teacher-only debate dataset of equal size (no student in the loop, no self-reflection or teacher feedback) and compare; if that dataset matches or beats D&R's results, the student-in-the-loop and feedback data are not what drives the reported gains.","supporting_citations":[],"review_version":1}