{"id":"5d92d2be-9431-4c3a-8b02-d88e549e879c","arxiv_id":"2502.01491","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Sequence-level knowledge-distilled NMT students memorize more of the original corpus and hallucinate more than same-size baselines trained directly on that corpus, despite never seeing it.","lead":"This paper finds that smaller machine-translation models trained on teacher-generated translations memorize more of the original training corpus and hallucinate more than same-size models trained directly on the original data. The result matters because most production translation systems are distilled, so the distillation step may amplify hidden failure modes instead of reducing them.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline increase in student memorization may be a subset-selection artifact: students only see original targets for the 18.4% of examples the teacher replicated, and no control baseline is evaluated on that subset. A restricted analysis could settle it.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern I would raise: the student and baseline differ in target exposure in a way that is confounded with example memorizability. The paper's headline numbers are real measurements, and the direction may well be correct, but the causal reading 'SeqKD facilitates memorization' is not established without a subset-restricted comparison. My proposed check uses the already-trained baseline, so it is cheap. If the restricted comparison fails to show a student advantage, the main contribution reduces to a descriptive observation about the teacher-replicated subset. If it succeeds, the paper's strongest claim is substantially strengthened. I also note the subgroup denoising analysis uses Comet-QE-22 both to define quality buckets and to evaluate model outputs on those buckets, which invites circularity, but that is a secondary issue. The paper is honest about limitations, provides code, and covers five language pairs; none of this is an integrity concern. Because the reader already made the central concern and issued CONDITIONAL, I do not change the verdict.","tokens_in":18968,"tokens_out":7167,"duration_ms":66368,"concrete_test":"For each language pair, let A={i : θT(x_i)=tC_i} (the teacher-replicated subset, ~18.4%). Recompute θS's ExMem_TC and replication_TC rates restricted to A. Then compute the same restricted rates for the existing baseline θB (trained on the full original corpus) on A only. If ExMem_θS|A ≈ ExMem_θB|A and replication_θS|A ≈ replication_θB|A, the headline 57% relative increase is a subset-selection artifact rather than an effect of SeqKD. To strengthen further, train an additional control θB_A on (x_i, tC_i) for i∈A with the same architecture and comparable training budget; if θS still exceeds θB_A on A, the SeqKD effect survives the selection confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—students extractively memorize 57% more of TC than baselines (§2.2, Figure 3b)—compares two training regimes that differ not only in the use of SeqKD but in which original targets the student ever sees. Because θT replicates only 18.4% of TC on average, the student's training targets equal original TC exactly only on subset A; elsewhere it sees synthetic TT≠TC. ExMem with respect to TC is thus concentrated on A for θS, while θB's ExMem is computed over the full corpus. If A is intrinsically more extractively memorable, the student's higher ExMem rate could reflect selection into A rather than SeqKD. The paper acknowledges 'within that smaller pool' (§2.2) but never reports θB's ExMem or replication rates restricted to A, nor trains a control on A with original targets. The abstract's 'despite not directly seeing' framing assumes exposure count is the only relevant difference, which the subset-selection objection challenges.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies sequence-level knowledge distillation (SeqKD) for neural machine translation, asking whether students inherit instance-level memorization from teachers. Across five WMT20 language pairs, the authors train Transformer-large teachers, Transformer-base students on teacher-generated targets, and same-size baselines on original targets. They report that students replicate original targets at a slightly higher rate than baselines (+3.4% relative), have a 57% higher extractive-memorization (ExMem) rate, and hallucinate more, while also exhibiting 'secondary ExMem' of teacher-generated text. The paper further analyzes data subgroups and finds amplified denoising on low-quality subgroups, and proposes Adaptive-SeqKD, a finetuning intervention that reduces memorization and hallucinations.","tokens_in":19187,"tokens_out":7244,"duration_ms":62896,"significance":"If the central claim survives scrutiny, this is a valuable and timely empirical result: it challenges the default assumption that distillation only transfers quality, and it gives practitioners a concrete reason to monitor memorization-related failures in distilled NMT systems. The strengths are the breadth (five language pairs, multiple memorization and hallucination metrics, additional beam-size and model-size ablations in Appendix C), the open code/data pipeline, and the proposal of a simple mitigation with consistent effects. The main quantitative claim, however, currently rests on a comparison that does not control for the student's restricted exposure to original targets, and the subgroup analysis uses the same metric for stratification and evaluation; both are fixable with additional analysis.","major_comments":[{"comment":"The central claim that students extractively memorize more than same-size baselines is confounded by the fact that the student's training data includes original targets TC only on the subset A where the teacher replicated TC (18.4% on average), whereas the baseline sees TC for the full corpus. If A is a non-random subset (e.g., easier or noisier examples), the higher ExMem rate of θS could reflect selection into A rather than an effect of SeqKD. The text acknowledges 'within that smaller pool' but never reports θB's replication or ExMem rates restricted to A, nor a control baseline trained on A with original targets. The absolute counts in Table 4 (e.g., EN-DE θS primary ExMem 22k vs θB 12k) suggest the effect may survive such a restriction, since primary ExMem for θS can only arise on A; however, this needs to be verified explicitly. I request a restricted analysis of θB on A and, if feasible, a control model trained on A.","section":"§2.2, Figure 3b, Table 4"},{"comment":"The 'amplified denoising' conclusion is drawn from Comet-QE-22 improvements on subgroups that were constructed by bucketing examples on Comet-QE-22 scores of the corpus targets. This creates a circularity risk: the same reference-free metric is used both to stratify and to evaluate, so the gains may reflect regression to the mean or metric-specific bias rather than genuine denoising. The paper reports chrF and MSTTR in the appendix, but the headline subgroup claim is specifically framed in terms of Comet-QE-22. Please validate the amplified-denoising effect with an external measure (e.g., human ratings, chrF against a held-out reference, or a different QE model) or demonstrate that the pattern is robust when subgroups are stratified by an alternative quality signal.","section":"§3, Figure 6, Appendix F.2"}],"minor_comments":[{"comment":"The phrase 'despite not directly seeing the original training data' is inaccurate: the student directly sees TC for the 18.4% of examples that the teacher replicated. Consider rewording to 'despite seeing most original targets only indirectly through synthetic translations' or similar.","section":"Abstract and §2.2"},{"comment":"The headline 57% ExMem increase is a relative increase over a small base rate; please also report the absolute percentage-point differences so readers can calibrate practical significance.","section":"Abstract and Figure 3b"},{"comment":"The column headings 'ExMem TC (#)' and 'ExMem TT (#)' do not specify the denominator; clarify whether the reported percentages are of replicated examples or of the full corpus, and whether the counts are in thousands.","section":"Table 4"},{"comment":"The CM approximation for the four non-EN-DE pairs uses a single OUT model and only 10% of datapoints; while this is acknowledged, the subgroup analysis would benefit from a sensitivity check or from focusing the CM claims on EN-DE where the approximation is stronger.","section":"Appendix F.1"},{"comment":"There are minor typographical errors, e.g., 'sytems' in Appendix E and 'NatHat' in Appendix D; please proofread.","section":"Appendix E and D"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope and likely to be citable if the missing control is added. The authors should be encouraged to provide the restricted analysis; without it, the central claim is not fully established. The use of Comet-QE-22 for both stratification and evaluation in §3 should also be addressed. I do not see grounds for rejection, but the revision is substantive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know: this paper is the first to measure memorization inheritance in SeqKD for NMT, and its central direction is credible—students memorize more than same-size baselines even though they never see the original targets directly. But the headline 57% extractive-memorization increase is not fully clean. The student only sees the 18.4% of original targets that the teacher replicated, and there is no baseline trained on that same subset, so part of the gap could be selection rather than distillation.\n\nWhat's new: the secondary ExMem channel is the real gem. The student literally memorizes teacher-invented strings (the AmarillasLatinas.net example), which is a concrete mechanism for hard-coding errors. That is not in the vision-domain KD literature, and it changes the safety calculus for library-scale distillation. The paper also runs five language pairs, uses standard WMT20 corpora, releases code, and is unusually honest about limitations.\n\nSoft spots: (1) The subset confound I mentioned. The paper acknowledges it ('within that smaller pool') but never runs the control. A restricted analysis comparing the baseline on the replicated subset would settle it. (2) The subgroup analysis in §3 is partly circular: quality buckets are defined by Comet-QE-22, and 'amplified denoising' is demonstrated by Comet-QE-22 gains on those same buckets. It is not damning, but it weakens the causal claim. (3) The absolute rates are small—the 57% increase is from roughly 0.3% to 0.45% ExMem in some pairs—so the practical risk should not be oversold. (4) Adaptive-SeqKD is a nice practical lever, but the finetuning is brief and the evaluation is limited to reductions in niche metrics; it does not address the core mechanism.\n\nNet: the paper's central claim that SeqKD amplifies memorization relative to a same-size baseline holds up in direction, and the secondary-ExMem observation is genuinely new. The magnitude is less certain. This deserves a serious referee; I would ask for the subset-controlled baseline and a sharper separation between the selection effect and the distillation effect.\n\nWho gets value: NMT practitioners doing model compression, safety folks studying data leakage and hallucination, and anyone building distillation pipelines. I'd bring it to reading group—it's a good case study in how a plausible result can be partially confounded.\n\nRecommendation: engage, review, require the control.","headline":"SeqKD students do seem to memorize more than same-size baselines, but the 57% headline is confounded by the student only seeing a teacher-replicated 18.4% subset; the secondary-ExMem finding is the real news.","tokens_in":19743,"tokens_out":2347,"would_cite":true,"duration_ms":20390,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sequence-level knowledge distillation transmits a teacher's memorized phrases and hallucinations to students, who memorize 57% more extractively than equally sized baselines despite never seeing the original corpus.","keywords":["sequence-level knowledge distillation","neural machine translation","extractive memorization","hallucination","counterfactual memorization","denoising","model compression","Adaptive-SeqKD"],"falsifier":"Train a same-size baseline model on exactly the teacher-generated corpus $(S_C, T_T)$ that the student saw, with no other changes; if that control baseline attains the same or higher ExMem and hallucination rates as the student, then the reported amplification is a property of the data subset, not of SeqKD. Alternatively, train students on shuffled or randomly subsetted versions of $T_T$ of the same size and check whether the ExMem increase persists.","tokens_in":1753,"feed_emoji":"🧠","tokens_out":1702,"duration_ms":57251,"temperature":0.7,"pith_summary":"Sequence-level knowledge distillation (SeqKD) is supposed to compress a large translation model into a smaller one by training the student on the teacher's translations rather than on the original parallel corpus. This paper sets out to show that what gets transmitted is not only the teacher's skill but also its faults: students who never see the original training data nevertheless memorize more of it than an equally sized baseline model trained on that data directly, with extractive memorization up 57% on average and oscillatory hallucinations up 31%. The paper also claims students pick up a second layer of memorization, reproducing text that the teacher itself hallucinated. If correct, this means standard quality metrics understate the risks of distillation, and practitioners need to monitor students for memorized fragments and hallucinations, not just translation quality scores.","feed_headline":"Despite seeing no raw data, distilled students memorize 57% more","feed_subtitle":"SeqKD students never see the original corpus, yet they replicate more of it and hallucinate more than the baseline.","key_machinery":"The load-bearing mechanism is the SeqKD pipeline itself: a large teacher $\\theta_T$ trained on the parallel corpus $(S_C, T_C)$ generates synthetic targets $T_T$ for the same sources, and a smaller student $\\theta_S$ trains on $(S_C, T_T)$, so the student's only window into the original data is through the teacher's outputs. The paper's quantitative lens is the ExMem metric of Raunak and Menezes, which flags targets the model emits after seeing at most 75% of the source, together with replication (exact match) rates and two hallucination rates (natural and oscillatory); counterfactual memorization (CM) scores, defined as the probability gap for a target between a model trained with and without that example, are used to subgroup the data. The argument that SeqKD amplifies memorization rests on comparing $\\theta_S$ with a baseline $\\theta_B$ of the same size trained directly on $(S_C, T_C)$, and the argument that denoising explains the effect rests on subgroup analyses showing students translate low-quality and low-confidence examples better than the teacher.","core_discovery":"The central claim is that memorization is inherited through SeqKD and is amplified relative to a same-size baseline. The paper measures replication (exact match) and extractive memorization (ExMem) with respect to the original parallel corpus, and finds that students, despite seeing on average only 18.4% of the corpus through the teacher's outputs, exceed the baseline's ExMem rate by 57.0% (std. 15.4) and its replication rate by 3.4% (std. 0.9). Students also produce more oscillatory and natural hallucinations than the baseline, and they exhibit 'secondary' ExMem: they memorize outputs the teacher generated, including hallucinated fragments such as a website name that appears in the teacher's translation. The paper further argues that SeqKD acts as a denoising filter, and on low-quality corpus subgroups students show amplified denoising, translating more accurately than both teacher and baseline; this creates the paper's central paradox: students memorize more overall yet improve exactly where the teacher filtered noise. Finally, the paper proposes Adaptive-SeqKD, which finetunes the teacher briefly on a high-quality subset before generating student targets, and reports that this reduces student ExMem and oscillatory hallucinations without hurting translation quality.","pith_inferences":["If the underlying mechanism is reduced regularization from denoised targets, then the same memorization amplification should appear in other distillation settings with noisy or misaligned training data, including LLM distillation; this is a testable prediction the paper does not make.","The paper's Adaptive-SeqKD selects high-quality data by teacher confidence and near-memorization (chrF>90); one could replace the teacher's own confidence with an external reference-free quality estimator and apply the same finetuning step when the original training corpus is unavailable, which the paper only sketches.","A direct control baseline trained on the teacher-generated corpus—the missing experiment identified in the limitations—would tell whether the 57% ExMem increase is caused by SeqKD or by the selection of already-memorizable sentences; if the latter, the recommendation should be reframed as data curation rather than a property of distillation."],"forward_implications":["Practitioners using SeqKD should evaluate students for extractive memorization and hallucinations, since average metrics like BLEU and Comet-QE can mask systematically worse failure modes.","The teacher's hallucinated outputs can propagate into students as verbatim memorized text, so cleaning teacher outputs matters even when students do not literally see raw corpora.","Adaptive-SeqKD, which finetunes the teacher on a high-quality subset before distillation, offers a low-cost way to cut student ExMem and oscillatory hallucinations without degrading BLEU or Comet-QE.","Higher beam sizes during SeqKD reduce student hallucinations (especially oscillatory ones) while slightly changing quality, suggesting that the 'do not use greedy search' lesson is about failure modes as much as quality.","Students can simultaneously memorize more and translate low-quality examples better, so memorization and generalization improvements are not opposed in this distillation setting."],"supporting_citations":[{"why":"Defines extractive memorization (ExMem) with respect to the parallel corpus, the central metric of the paper's quantitative claim.","marker":"Raunak and Menezes, 2022"},{"why":"Introduced sequence-level knowledge distillation, the training paradigm whose fault inheritance the paper studies.","marker":"Kim and Rush, 2016"},{"why":"Defines counterfactual memorization scores, which the paper uses to construct data subgroups and test whether high-CM examples are inherited.","marker":"Feldman, 2020"},{"why":"Supplies the natural and oscillatory hallucination metrics that the paper uses to measure amplified failure modes.","marker":"Guerreiro et al., 2023"},{"why":"Connects hallucinations to memorization in NMT and provides the natural hallucination (NatHal) definition used here.","marker":"Raunak et al., 2021"},{"why":"Provides the WMT20 parallel corpora used for all five language pairs in the experiments.","marker":"Barrault et al., 2020"},{"why":"Sets the Transformer architecture and training duration for the teacher, student, and baseline models.","marker":"Vaswani et al., 2017"}],"fun_headline_variants":["Students inherit teacher's memory: 57% more extractive memorization","SeqKD amplifies teacher memorization in student NMT models","Adaptive-SeqKD reduces memorization and hallucinations in distilled NMT","Distilled NMT students mimic teacher's errors and memorize more","Knowledge distillation passes on teacher's memorization faults"],"cache_read_input_tokens":21888,"weakest_assumption_plain":"The paper's comparison assumes that the student's higher memorization and hallucination rates are caused by the SeqKD training procedure; but the student only ever sees the 18.4% of the original corpus that the teacher replicated, and no baseline was trained on that same teacher-generated subset, so the difference could partly reflect the memorizability of that particular selected data rather than an effect of distillation itself.","fun_headline_variants_meta":{"raw":{"variants":["Students inherit teacher's memory: 57% more extractive memorization","SeqKD amplifies teacher memorization in student NMT models","Adaptive-SeqKD reduces memorization and hallucinations in distilled NMT","Distilled NMT students mimic teacher's errors and memorize more","Knowledge distillation passes on teacher's memorization faults"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3516,"prompt_tokens":975,"completion_tokens":2541,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2454}},"tokens_in":591,"tokens_out":2541,"duration_ms":17858,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:08:14.355108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a same-size baseline model on exactly the teacher-generated corpus $(S_C, T_T)$ that the student saw, with no other changes; if that control baseline attains the same or higher ExMem and hallucination rates as the student, then the reported amplification is a property of the data subset, not of SeqKD. Alternatively, train students on shuffled or randomly subsetted versions of $T_T$ of the same size and check whether the ExMem increase persists.","supporting_citations":[],"review_version":1}