{"id":"7c6a4610-2838-4d5f-90bd-79d81a22bf7e","arxiv_id":"2412.17846","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Prompting a large teacher model to explain itself before knowledge distillation gives a 17-point accuracy gain on GSM8K for an 8B student, but the evidence is weakened by test-set tuning and missing error bars.","lead":"A small language model trained on outputs from a giant one improves on math word problems when the giant is prompted to explain itself, with the best prompt raising accuracy from 31% to 48%. The paper's evidence is weakened by test-set tuning and a single-run comparison.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 55% gain is not a clean measurement: alpha and temperature are selected on the full GSM8K test split (§4.2), so the reported test accuracy is partly an artifact of the search.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: tuning alpha and temperature on the full GSM8K test split invalidates the reported test accuracy as a clean measurement. My stress-test agrees. The paper explicitly says in Section 4.2 that the Bayesian optimization evaluates on the full test split and selects hyperparameters based on the lowest validation loss, then Section 5 uses that same split for the final accuracy comparison. This is not a minor stylistic issue; it is a protocol violation that biases the headline 55% improvement. The claim could still be true, but the reported evidence does not support it as presented. No ad hominem is intended: the concern is purely about the experimental design. A clean holdout validation split and a single untouched test evaluation would settle the question. The attention analysis is also weakened by using a single example and no error bars, but the test-set tuning is the more fundamental problem because it directly affects the central accuracy claim. The reader's REJECT verdict is therefore unchanged by my analysis.","tokens_in":11718,"tokens_out":2362,"duration_ms":22670,"concrete_test":"Re-run the KD experiments with a strict data split: use a subset of the GSM8K train questions for distillation training, hold out a disjoint validation split for Bayesian optimization of alpha and temperature, and touch the official GSM8K test split only once after fixing all hyperparameters. If Ground Truth KD then beats Base KD by a comparable relative margin on the untouched test set, ideally across at least three seeds with mean and confidence interval, the central claim would survive; if the margin drops sharply or reverses, the reported 55% was largely an artifact of test-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (Ground Truth KD 48.14% vs. Base KD 30.62%, a 55% relative gain) rests on the test-set accuracies in Table 1 and Figure 4. Section 4.2 states that the authors conduct a 50-iteration Bayesian optimization over alpha and temperature, evaluating on the full test split and selecting the hyperparameters with the lowest validation loss. The same full test split is then used for the final accuracy table, so the headline result is selected, not measured: the hyperparameter search had direct access to the evaluation labels, and the final numbers are not an unbiased estimate of generalization. This alone is sufficient to compromise the reported 55% improvement; the absence of error bars or repeated seeds makes it impossible to tell whether the gap is signal or selection noise. The paper's own limitations section does not flag this test-set leakage, and the data-size inconsistency (Section 3 says the first 1319 training questions, Section 4.1 says the first 2600) further undercuts the reproducibility of the protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three teacher-side response-priming prompts (Teacher, Ground Truth, Confidence) for knowledge distillation from a quantized Llama 3.1 405B teacher to a Llama 3.1 8B student, evaluated on GSM8K with LoRA fine-tuning. It reports that Ground Truth prompting raises student accuracy from 30.62% (base KD) to 48.14%, a 55% relative gain, and it presents self-attention entropy, self-focus, and cross-layer similarity analyses intended to explain the gain. The central claim is that response-priming prompting improves KD and that the attention metrics reveal the mechanism.","tokens_in":12009,"tokens_out":7813,"duration_ms":71901,"significance":"If the result were clean, response-priming prompting would be a simple and broadly applicable enhancement to knowledge distillation pipelines. The paper has several strengths: the prompt templates are explicit, the experimental setup is described in detail, compute costs are reported, code is released, and the attention-level investigation is a useful attempt to move beyond accuracy numbers. However, the main empirical result is not a clean measurement because the hyperparameters are selected on the test split, the reported KD loss equations are internally inconsistent, the training subset size is contradictory, and the attention analysis rests on a single prompt. These problems affect the central claims, so the contribution is interesting but not established.","major_comments":[{"comment":"The 50-iteration Bayesian optimization over alpha and temperature is performed by training on the custom GSM8K train split and evaluating on the full test split, and the final accuracies in Table 1 and Figure 4 are then reported on that same test split. The paper calls the selection criterion 'validation loss,' but it is computed on the test labels, so the headline 30.62% to 48.14% improvement is a selected result rather than an unbiased measurement. No repeated seeds or error bars are provided, so it is impossible to tell how much of the reported gap is signal and how much is selection noise. The limitations section (Section 9) does not acknowledge this test-set leakage.","section":"Section 4.2"},{"comment":"Section 3 states that the student is fine-tuned on 'the first 1319 questions' of the GSM8K train split, while Section 4.1 states that the authors 'select the first 2600 questions of GSM8K's train split as our training data.' These numbers are materially different, and every reported result depends on the transfer set. The paper must state which subset was actually used and justify the choice.","section":"Section 3 versus Section 4.1"},{"comment":"Equation (1) defines p_i as a log-softmax: p_i = z_i/T - log(sum_j exp(z_j/T)), but Equation (3) uses p_i in the denominator of the reverse-KL ratio q_i log(q_i/p_i). Since p_i is a log-probability (non-positive) and q_i is a probability, the ratio is not a probability ratio, and the logarithm is undefined for negative arguments. As written, the loss is not the reverse KL divergence claimed in the text; the equations or the implementation must be corrected.","section":"Figure 3, Eqs. (1)-(3)"},{"comment":"The attention analysis (entropy, self-attention focus, and cross-layer similarity) is computed on a single prompt, the 'Janet's ducks' example listed in Appendix A, for each model. The discussion then states that the more accurate prompted models 'are better at using their final self-attention layers' and that the observed behaviors 'can be tied to increased accuracy.' With one example, no error bars, and no statistical or interventionist test, these are post hoc descriptions rather than causal evidence. The claims in the abstract and Section 7 go beyond what this analysis can support.","section":"Section 6"}],"minor_comments":[{"comment":"The phrase '55% performance increase' should be specified as a relative increase; the absolute improvement is 17.52 percentage points (30.62% to 48.14%), which is a more informative statement.","section":"Abstract"},{"comment":"The sentence 'A lower alpha value indicates that the teacher model's logits exert a positive influence on the student model, and vice-versa' is ambiguous; the directions for both alpha and temperature should be stated precisely in terms of the loss formula.","section":"Section 4.2"},{"comment":"The statement that Ground Truth KD shows 'a nearly 400% increase' over the non-distilled model is a relative comparison; the absolute accuracies (12.20% vs. 48.14%) should also be given in the text.","section":"Section 5.1"},{"comment":"The criteria for whether a high or low entropy, self-focus, or similarity score is 'positive' or 'negative' are introduced after the results are inspected; an independent, pre-specified criterion would strengthen the interpretation.","section":"Sections 6.1-6.3"},{"comment":"The limitations section mentions that hyperparameter selection is 'hidden and compute/time heavy,' but it does not disclose that the selection used the test split or that the attention analysis used a single prompt; both should be listed as limitations.","section":"Section 9"}],"recommendation":"reject","confidential_remarks":"I agree with the stress-test analysis: the test-set leakage in Section 4.2 is decisive. The reported 55% gain is not a free prediction, and the absence of error bars compounds the problem. The inconsistent loss equations and the contradictory training-subset sizes reinforce my view that the current manuscript does not support its central claims. I would, however, be willing to look at a substantially revised version that redoes the hyperparameter selection on a proper validation split, reports multi-seed results, corrects the loss equations, clarifies the dataset size, and replaces the one-example attention analysis with a more systematic study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline result—Ground Truth prompting giving a 55% relative gain over Base KD on GSM8K—is not supported by the evidence as reported. Section 4.2 says the authors ran 50 iterations of Bayesian optimization over alpha and temperature, evaluating on the full test split, and then report that same split as the final accuracy in Table 1. That is test-set selection, not evaluation. The 30.62% vs. 48.14% gap could be partly an artifact of the hyperparameter search. There are no error bars or repeated seeds, so we can't tell signal from selection noise.\n\nThat said, the paper is not without merit. The idea—prompt the teacher during data generation for knowledge distillation—is a sensible, low-cost lever, and the authors correctly cite the relevant prior work (Magister et al., McDonald and Emami). They also do a few things well: they fix the two-epoch LoRA setup, they report compute resources, and they make code available. The three prompt variants are simple, but the comparison is clean in the sense that only the teacher prompt changes across conditions.\n\nThe soft spots go beyond the test-set tuning. There is an internal inconsistency in the training subset size—Section 3 says the first 1319 questions, Section 4.1 says the first 2600. That alone would complicate replication. The attention analysis (entropy, self-focus, similarity) is computed on a single GSM8K question and is purely post hoc; the paper claims these metrics \"can be tied to\" accuracy, but there is no correlational or causal support beyond one example. The qualitative analysis is also anecdotal. On novelty, the prompts are variations of reasoning-eliciting instructions, and the general finding that prompting the teacher helps KD is already in the cited literature.\n\nWho gets value from this paper? People building practical KD pipelines might pick up the prompt ideas, but they should not rely on the numbers. The paper deserves a serious referee because the question is relevant and the flaws are identifiable and fixable—but as it stands the central claim is unverified. I would send it to review with a demand for a clean evaluation protocol: hold out the test split from any hyperparameter selection, run multiple seeds, and report error bars.","headline":"The reported 55% gain from Ground Truth prompting is not a clean measurement: alpha and temperature were tuned on the GSM8K test split, so the headline accuracy is partly a product of the search.","tokens_in":12452,"tokens_out":1935,"would_cite":false,"duration_ms":17281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompting the teacher before distillation raises a small model's math accuracy by 55 percent.","keywords":["knowledge distillation","response-priming prompting","GSM8K","Llama 3.1","LoRA","self-attention analysis","reverse KL divergence","prompt engineering"],"falsifier":"Re-run the same comparison but select α and temperature on a separate validation subset, then report test accuracy for the chosen hyperparameters; if Ground Truth prompting no longer beats unprompted KD by a comparable margin, the claimed effect is largely an artifact of test-set tuning.","tokens_in":11554,"feed_emoji":"🎯","tokens_out":4393,"duration_ms":35010,"temperature":0.7,"pith_summary":"This paper claims that the way you prompt a large teacher model during knowledge distillation changes how much a smaller student model learns. The authors distill a Llama 3.1 8B model from a 405B teacher on the GSM8K math benchmark and find that telling the teacher its outputs will train smaller models (Ground Truth prompting) boosts student accuracy from 30.62% to 48.14%, a 55% relative increase over unprompted distillation. They argue that this improvement is accompanied by measurable changes in the student's self-attention: lower entropy and lower self-focus in the final layers, plus higher stability between late-layer attention distributions. If the claim holds, it would make large models cheaper to deploy by extracting more teachable behavior from them.","feed_headline":"Teacher prompt lifts small model's math score 55%","feed_subtitle":"Telling the teacher to write for smaller models raises distilled Llama's GSM8K accuracy from 30.6 to 48.1 percent.","key_machinery":"The central intervention is response-priming prompting: a set of short prompt strings inserted into the teacher's input while generating the transfer set, such as instructing the teacher to act as a teacher, to state that its output will train smaller models, or to double-check its answer. The mechanism story is carried by three attention metrics—attention entropy, self-attention focus (the diagonal of the attention matrix), and attention distribution similarity (cosine similarity between consecutive layers)—which the authors use to link prompt choice to internal student behavior. Training itself uses a combined hard and soft loss with reverse KL divergence and LoRA, allowing the teacher's logits to shape the student's confidence during fine-tuning.","core_discovery":"On the paper's own terms, prompting the teacher before it generates the distillation transfer set is not a side detail but a primary lever. The Ground Truth prompt—instructing the teacher to produce answers that are clear and feasible for smaller models to learn from—yields the highest GSM8K accuracy (48.14%) among all student variants, beating unprompted KD (30.62%), confidence prompting (34.04%), and teacher prompting (42.30%). The paper further claims that the more accurate prompted models show systematic differences in their self-attention: lower average entropy in the late layers, lower token self-focus in the final layers, and higher cosine similarity between attention distributions of consecutive late layers. The authors interpret these patterns as evidence that the student has finalized its interpretation of the input and integrates context more effectively, which they tie to the improved reasoning quality.","pith_inferences":["If the effect is causal, similar response-priming prompts may transfer to other math and reasoning benchmarks, because the prompt targets the teachability of the transfer set rather than the specific problems.","A cleaner test of the claim would tune the loss hyperparameters on a held-out validation split instead of the full test split; the reported 55% improvement could shrink if the search overfits the test set.","The attention metrics could be used as a training signal or early-stopping criterion for distillation quality, an application the paper does not explore.","The paper's comparison would be stronger with multiple random seeds and a larger set of benchmarks to rule out dataset-specific effects."],"forward_implications":["Ground Truth prompting, applied only to the teacher, raises a distilled 8B model's GSM8K accuracy from 30.62% to 48.14%, a 55% relative gain over unprompted KD.","Teacher prompting also improves accuracy to 42.30%, showing that several reasoning-eliciting prompts help, not just one specific formulation.","All prompted KD variants beat the unprompted KD baseline, and their accuracy ordering matches the qualitative quality of the reasoning traces the student produces.","The student's late-layer attention becomes measurably more focused and more stable between layers, providing a candidate internal signature for good distillation."],"supporting_citations":[{"why":"provides the GSM8K dataset and the train/test split that grounds every accuracy number.","marker":"(Cobbe et al., 2021)"},{"why":"supplies both the Llama 3.1 405B teacher and the Llama 3.1 8B student model used in the pipeline.","marker":"(Dubey et al., 2024)"},{"why":"introduces LoRA, the parameter-efficient fine-tuning method the student uses.","marker":"(Hu et al., 2021)"},{"why":"establishes chain-of-thought reasoning prompting, which motivates the teacher-prompting strategies.","marker":"(Wei et al., 2023)"},{"why":"prior work on chain-of-thought prompting in distillation, the baseline the paper extends with a modern LLM-specific loss.","marker":"(Magister et al., 2023)"},{"why":"provides the rationale for using a more fine-grained soft loss for LLM knowledge distillation.","marker":"(Gu et al., 2024)"},{"why":"supplies the reverse KL divergence view that underpins the confidence-prompting rationale.","marker":"(Le, 2017)"}],"fun_headline_variants":["Ground Truth prompt boosts distilled LLM's math score by 55%","Teacher prompt before distillation lifts student GSM8K 55%","Priming the teacher with a simple prompt improves KD by 55%","For better small LLMs, prompt the teacher first: +55% GSM8K","Why prompted distillation works: Attention patterns reveal the secret"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on treating the full GSM8K test set as a valid place to pick the distillation hyperparameters (α and temperature); if the test set must be kept untouched for an honest accuracy measurement, the reported 55% gain may be partly an artifact of that search.","fun_headline_variants_meta":{"raw":{"variants":["Ground Truth prompt boosts distilled LLM's math score by 55%","Teacher prompt before distillation lifts student GSM8K 55%","Priming the teacher with a simple prompt improves KD by 55%","For better small LLMs, prompt the teacher first: +55% GSM8K","Why prompted distillation works: Attention patterns reveal the secret"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2901,"prompt_tokens":1001,"completion_tokens":1900,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":1807}},"tokens_in":617,"tokens_out":1900,"duration_ms":13333,"temperature":1.0,"reasoning_tokens":1807,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:20:28.725877+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same comparison but select α and temperature on a separate validation subset, then report test accuracy for the chosen hyperparameters; if Ground Truth prompting no longer beats unprompted KD by a comparable margin, the claimed effect is largely an artifact of test-set tuning.","supporting_citations":[],"review_version":1}