{"id":"7b85f6af-855c-417b-9e67-686c9017cd57","arxiv_id":"2412.14528","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"A multi-level optimal transport loss combining sequence-level ranking, top-k truncation, and Sinkhorn sequence distance outperforms earlier cross-tokenizer distillation losses on QA and summarization.","lead":"This paper presents a new training loss that lets a small language model learn from a large one even when the two models split words into different pieces. The method reports consistent score gains on question answering and summarization tasks across several model pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The token-level losses are undefined when teacher and student vocabularies differ in size, which is exactly the cross-tokenizer setting the paper claims to address.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the permutation Q in Eq. (10)-(11) is not defined for unequal vocabulary sizes, and no evidence shows that it aligns semantically related dimensions. This is the most critical point because the token-level losses are the primary proposed components (LHAD and LSL), and even the sequence-level Sinkhorn cost matrix is built from the truncated ranked vectors t^Tr and s^Tr, so every loss in Eq. (21) depends on Q. Since all experiments use mismatched vocabularies, the method as written cannot be instantiated without an unstated procedure for Q. This is an internal-consistency problem, not a disagreement with external consensus, and it directly undermines the reproducibility of the reported outperformance. The concern is concrete and testable by examining the available code. The reader's conditional verdict already reflects this, so I do not change the verdict; the conditionality is appropriate, and the concrete test would determine whether the condition is satisfied.","tokens_in":13116,"tokens_out":8122,"duration_ms":70110,"concrete_test":"Inspect the released code (github.com/2018cx/Multi-Level-OT) to determine the exact construction of Q when the teacher vocabulary size m does not equal the student vocabulary size n. Specifically, check whether Q is a padded permutation (e.g., independent sequence-level ranking and truncation to a common top-k) or a rectangular assignment solved by an OT or linear assignment solver. Then re-run a single QED configuration (e.g., LLaMA2-7B -> OPT-350M) with the paper's stated hyperparameters; if the code requires a different Q construction than Eq. (10)-(11) or if the losses cannot be computed for m≠n, the central claim is unsupported as written. If the code does implement a well-defined Q for unequal sizes and reproduces Table 1, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MultiLevelOT consistently outperforms CTKD baselines rests on the token-level losses LHAD and LSL, which operate on logits after applying the permutation Q in Eq. (10)-(11). But Q is defined as a permutation matrix satisfying Q*=argmin_Q sum_t sum_i |tSR,i(t)-[Qs(t)]_i|, and a permutation matrix exists only when m=n. In every reported experiment the vocabularies differ: LLaMA2-7B (~32k) vs OPT-350M (~50k), Pythia-410M (~50k), and Bloomz-560M (~250k). No procedure is given for computing Q when m≠n, nor for how the top-k truncation in Eq. (12) interacts with an m×n assignment. If Q is instead a non-square assignment, the claim that P*=P^HAD (identity) in Eq. (13)-(14) requires a perfect matching between k teacher dimensions and k student dimensions; when one vocabulary is larger, unmatched dimensions are silently dropped. The paper provides no evidence that such an alignment is semantically meaningful. Without a well-defined Q, the token-level losses—and hence the central method—are underspecified in the exact setting the paper targets, making the reported gains unreproducible from the paper alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MultiLevelOT, a cross-tokenizer knowledge distillation loss for LLMs with different tokenizers. The method combines token-level losses (a holistic absolute difference loss and a sequential logarithmic loss, both applied after sequence-level ranking and top-k truncation) with a sequence-level Sinkhorn distance loss. Experiments on extractive QA (QED), generative QA (FairytaleQA), and summarization (DIALOGSum) use LLaMA2-7B as the teacher and OPT, Pythia, Bloomz, and mT0 students; the reported tables show consistent gains over SFT, SeqKD, MinED, and ULD, together with ablations and hyperparameter sensitivity studies.","tokens_in":13348,"tokens_out":12795,"duration_ms":111342,"significance":"If the method is correctly specified, it is a practically useful contribution: a logit-only, module-free CTKD loss that claims to work across vocabulary sizes, model families, and architectures, with released code. The paper's strengths are the breadth of experiments (multiple tasks, students, and teachers), the ablations that isolate each loss component, and the explicit hyperparameter analysis for the Sinkhorn iterations N and truncation size k. However, the formal specification of the token-level alignment is incomplete in the cross-tokenizer setting, and the main empirical claims are supported only by single-run point estimates. Both issues need to be addressed before the central claims are fully supported.","major_comments":[{"comment":"The token-level OT problem is only well-posed when the teacher and student vocabularies have the same size. Eq. (8) requires a doubly stochastic matrix P of size m x n, forcing m=n, and Eq. (10)-(11) define Q as a permutation matrix, which also requires m=n. Yet every reported experiment uses unequal vocabularies (e.g., LLaMA2-7B has about 32k tokens while OPT/Pythia have about 50k and Bloomz-560M has about 250k). The paper does not state how Q or the top-k truncation in Eq. (12) is computed when m != n, nor whether unmatched dimensions are discarded before or after truncation. Because L_HAD and L_SL in Eqs. (14) and (16) depend on this step, the central loss is underspecified in exactly the cross-tokenizer setting the paper targets and cannot be reproduced from the manuscript alone. Please provide an explicit algorithmic construction for m != n and demonstrate that the identity-P claim in Eqs. (13)-(14) holds under it.","section":"Methods, §3"},{"comment":"The central claim that MultiLevelOT 'consistently outperforms all baseline methods across all datasets and student models' rests on single-run point estimates. No standard deviation, number of random seeds, or significance test is reported, and some margins are small; for example, Table 1, FairytaleQA, OPT-350M: Ours 46.96 vs MinED 46.11. Please report variance across at least three seeds or otherwise justify that the reported differences are not within run-to-run noise.","section":"Experiments, Tables 1-2"},{"comment":"The same symbol P is used for the token-level transport plan (size m x n before truncation, k x k after) and the sequence-level plan (size T x T), and Eq. (8) states one set of constraints that cannot apply to both plans simultaneously. Please introduce separate notation, e.g., P_tok and P_seq, and state their respective marginal constraints; as written, the objective in Eq. (7) is ambiguous and cannot be verified.","section":"Methods, Eqs. (7)-(8)"}],"minor_comments":[{"comment":"The phrase 'eliminating the need for dimensional or token-by-token correspondence' overstates the method: Eqs. (14) and (16) are elementwise after ranking and truncation, so a rank-based dimensional correspondence is still established. Please soften or qualify this claim.","section":"Abstract and §3"},{"comment":"The 'over 71%' gap reduction in the QED task is consistent with the average row of Table 1 (teacher 61.68, ULD 59.30, Ours 60.99), but per-student reductions vary; please state explicitly that the figure refers to the average over the three students.","section":"Experiments, 'Comparison with SOTA'"},{"comment":"The table caption contains a typo: 'FairtaleQA' should be 'FairytaleQA', and 'Rouge-LSUM' should be 'Rouge-LSum' for consistency with the rest of the paper.","section":"Table 5"},{"comment":"The notation 'tSR = t[argsort(...)]' is ambiguous; please clarify that argsort returns the indices that sort the summed logits in descending order and that tSR is the resulting reordered matrix, and define sSR after applying Q in the same way.","section":"Eq. (10)"},{"comment":"The notation s^k_i(t) collides with the truncation index k; consider using a tilde or a superscript '(k)' to denote truncated logits, and define the range of the truncation explicitly.","section":"Eq. (15)"},{"comment":"The exclusion of DSKD is motivated by its added learnable parameters, but since DSKD is a cross-tokenizer baseline, please report its performance as an additional reference or provide a quantitative comparison that isolates the effect of the extra modules.","section":"Baselines"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the core idea is plausible. The main risk is that the token-level losses are not well-defined for unequal vocabularies in the written paper; I would not recommend acceptance until this is resolved with a clear algorithmic specification. I did not run the released code; if the code implements a specific partial matching or independent top-k ranking, the paper should describe that procedure. The empirical comparison would also be substantially stronger with variance reporting or significance tests on the main tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is a genuinely new combination for cross-tokenizer KD. The sequence-level Sinkhorn distance loss over tokens is new, and the paper shows consistent, if small, gains (1-2 F1/ROUGE points) over ULD across tasks, students, and teachers. The token-level losses are not new—they reduce to ranked L1 and truncated cross-entropy—but bundling them with a sequence-level OT loss is a reasonable contribution.\n\nThe experiments are solid in scope: labeled and unlabeled settings, multiple model families, ablations, and sensitivity checks for k and N. One correction to the reader's report: the 71% gap-reduction claim does match the table if you use the average over the three students on QED (ULD 59.30, ours 60.99, teacher 61.68), so that is not an overclaim.\n\nThe real soft spot is the definition of Q in Eqs. (10)-(11). It is called a permutation matrix, but a permutation matrix only exists when teacher and student vocabularies have the same size. Every experiment uses unequal vocabularies. The paper never says how Q is computed in the rectangular case, nor how top-k truncation interacts with the matching. If the intended procedure is to rank each vocabulary independently by its sequence-level sum and then truncate to k, the losses are well-defined—but that is not what the equations say. As written, the token-level losses are underspecified in exactly the setting the paper claims to address. That is a load-bearing reproducibility gap. The absence of error bars is minor but worth fixing; the hyperparameter choice anchored on QED is a minor limitation. The citation pattern is appropriate; they build on ULD and SinKD honestly.\n\nThe paper deserves a serious referee. I would send it to review with a request to clarify Q, add variance across seeds, and make the code publicly available (the link is promised). For a reader working on KD or LLM compression, this is worth engaging with. I would not cite it as-is until the Q issue is resolved.","headline":"Useful cross-tokenizer distillation loss with consistent small gains, but the token-level math is underspecified for unequal vocabularies—fix that before accepting.","tokens_in":13893,"tokens_out":4681,"would_cite":true,"duration_ms":42375,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MultiLevelOT claims that optimal transport at both token and sequence levels lets a student language model absorb a teacher's knowledge even when their tokenizers and vocabularies do not match, and reports consistent gains over prior…","keywords":["knowledge distillation","cross-tokenizer distillation","optimal transport","Sinkhorn distance","large language models","logit alignment","vocabulary mismatch","sequence-level distillation"],"falsifier":"Replace the sequence-level ranking and permutation in Eqs. (10)-(11) with a random permutation of the student's vocabulary dimensions and re-run the three tasks; if scores remain close to the reported ones, the alignment Q is not doing the claimed work, and if they collapse, it is. A second check: on a held-out set, look up the actual vocabulary tokens behind the top-k ranked dimensions of the teacher and student after matching and measure how often they are the same word or a translation or paraphrase.","tokens_in":12901,"feed_emoji":"🧠","tokens_out":7591,"duration_ms":55816,"temperature":0.7,"pith_summary":"Knowledge distillation usually requires teacher and student to share a tokenizer, because divergence losses compare vocabulary dimensions one by one. This paper claims that constraint is unnecessary: a loss built on optimal transport can align the two models' output distributions even when their vocabularies and tokenizations differ. The proposed MultiLevelOT computes optimal transport distances at two levels, across vocabulary dimensions within tokens and across tokens within a sequence, and combines an absolute-difference cost, a logarithmic cost, and a Sinkhorn distance cost. On extractive QA, generative QA, and summarization, it reports consistent gains over prior cross-tokenizer methods such as ULD across several student models. If the claim holds, practitioners can distill across model families and architectures without retraining aligners or standardizing tokenizers.","feed_headline":"Multi-level OT distills LLMs across mismatched tokenizers","feed_subtitle":"New loss ranks and truncates logits, then applies token- and sequence-level transport, beating ULD on QA and summarization.","key_machinery":"The machinery is a two-level optimal transport objective. At the token level, the paper reconstructs ULD's token-wise OT as an equivalent sorted elementwise loss and generalizes it by ranking all tokens in the sequence jointly, truncating to the top-k logit dimensions, and using two cost matrices: absolute difference $C^{tok}_{ij}(t)=|t_i(t)-s_j(t)|$ and logarithmic $C^{tok}_{ij}(t)=-t_i(t)\\log s_j(t)$. This yields the holistic absolute difference loss $L_{HAD}$ and sequential logarithmic loss $L_{SL}$, after which a single optimal transport plan, diagonal in form, applies to all tokens. At the sequence level, the cost between token rows is fed into a Sinkhorn distance, $P_{\\lambda} = \\arg\\min_P \\sum_{ij} P_{ij}C_{ij} - \\lambda h(P)$, approximated by iterative Sinkhorn normalization, giving $L_{SD}$; the total loss weights these terms with cross-entropy: $L = \\sum_t L_{CE}(y(t),s(t)) + \\alpha(L_{HAD}+\\beta L_{SL}+\\gamma L_{SD})$. The design claim is that diverse cost matrices capture both local logit geometry and global sequence structure while avoiding dimensional or token-by-token correspondence.","core_discovery":"The paper's central claim is that cross-tokenizer knowledge distillation can be made universal by replacing dimension-wise or token-wise divergence with multi-level optimal transport on logit distributions. After temperature-scaling both models' logits, MultiLevelOT ranks each model's vocabulary dimensions by their summed logits over the whole sequence, applies a permutation to the student so its dimensions are matched to the teacher's, truncates both to the top k dimensions, and then applies two token-level losses, a holistic absolute difference loss and a sequential logarithmic loss, plus a sequence-level Sinkhorn distance loss that measures the transport cost between the teacher's and student's token rows. The paper claims this consistently outperforms ULD and other baselines across all datasets and student models, cuts the teacher-student performance gap on QED by over 71% relative to ULD, and generalizes across model families, architectures, and scales without extra modules.","pith_inferences":["A plausible extension the paper only gestures at is multi-teacher distillation: because the loss needs no vocabulary correspondence, several teachers with different tokenizers could supervise one student by averaging their OT losses, provided the sequence-level costs are comparable.","The sequence-level Sinkhorn cost is built from pairwise token distances, so its memory and time grow with sequence length squared; for long-document or long-context tasks, chunking or a more approximate OT plan would likely be needed.","The semantic content of the learned permutation is testable: if the top-k ranked teacher and student dimensions after matching frequently correspond to the same word or morpheme, the elementwise losses are transferring lexical knowledge; if they do not, the reported gains may come from distributional shaping rather than lexical alignment."],"forward_implications":["A single distillation recipe works for teachers and students from different model families, such as LLaMA, Mistral, and Qwen, and for student scales from 160M to 1B, with no extra trainable modules.","The method improves on ULD on every dataset and student model reported, including the unlabeled setting where only teacher-generated text is available.","Sequence-level transport removes the need for rigid token-by-token alignment, so words split differently by different tokenizers can still be matched.","Because the total loss is a weighted sum of cross-entropy with three OT-based terms, it can be dropped into an existing supervised fine-tuning loop with three scalar weights."],"supporting_citations":[{"why":"The prior cross-tokenizer distillation loss that MultiLevelOT extends, reconstructs, and uses as its main baseline.","marker":"ULD (Boizard et al. 2024)"},{"why":"Supplies the Sinkhorn distance formulation and the argument that traditional divergence measures suffer from mode issues.","marker":"SinKD (Cui et al. 2024b,a)"},{"why":"Introduces Sinkhorn distances with entropy regularization, the approximation used for the sequence-level transport cost.","marker":"Cuturi 2013"},{"why":"A dynamic-programming logit alignment baseline compared in both labeled and unlabeled settings.","marker":"MinED (Wan et al. 2024)"},{"why":"A sequence-level knowledge distillation baseline that supervises the student with teacher outputs.","marker":"SeqKD (Kim and Rush 2016)"},{"why":"The extractive QA dataset used for the main experiments and the reported 71% gap-reduction figure.","marker":"QED (Lamm et al. 2021)"},{"why":"The generative QA dataset used to evaluate the method across student scales and teachers.","marker":"FairytaleQA (Xu et al. 2022)"},{"why":"The summarization dataset used to test the method beyond extractive and generative QA.","marker":"DIALOGSum (Chen et al. 2021)"}],"fun_headline_variants":["Cross-tokenizer KD via multi-level optimal transport","Distill any LLM pair with OT, no shared vocab needed","Multi-level OT beats ULD for cross-tokenizer distillation","Optimal transport unifies tokenizers for LLM distillation","Token- and sequence-level OT for universal KD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The token-level losses assume that after ranking and truncating the vocabulary dimensions, the i-th surviving dimension of the teacher and the i-th surviving dimension of the student are close enough in meaning that elementwise comparison is informative, yet the paper gives no procedure for computing the matching permutation when the vocabularies differ in size.","fun_headline_variants_meta":{"raw":{"variants":["Cross-tokenizer KD via multi-level optimal transport","Distill any LLM pair with OT, no shared vocab needed","Multi-level OT beats ULD for cross-tokenizer distillation","Optimal transport unifies tokenizers for LLM distillation","Token- and sequence-level OT for universal KD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000557,"raw_usage":{"total_tokens":2661,"prompt_tokens":969,"completion_tokens":1692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":1611}},"tokens_in":585,"tokens_out":1692,"duration_ms":12223,"temperature":1.0,"reasoning_tokens":1611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:09:04.440454+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the sequence-level ranking and permutation in Eqs. (10)-(11) with a random permutation of the student's vocabulary dimensions and re-run the three tasks; if scores remain close to the reported ones, the alignment Q is not doing the claimed work, and if they collapse, it is. A second check: on a held-out set, look up the actual vocabulary tokens behind the top-k ranked dimensions of the teacher and student after matching and measure how often they are the same word or a translation or paraphrase.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Sinkhorn distances with entropy regularization, the approximation used for the sequence-level transport cost."}],"review_version":1}