Pith. sign in

REVIEW 2 major objections 5 minor 60 references

Byte-Prefix Marginalization re-expresses a teacher's next-token distribution on a different student vocabulary via shared byte prefixes, beating existing cross-tokenizer distillation methods by 3.7–6.6 points on six benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:03 UTC pith:QNWW6I6K

load-bearing objection BPM is a genuinely new construction for cross-tokenizer distillation with honest, broad experiments, but the exactness claim leans on an unproven idealization of BPE that the paper itself flags. the 2 major comments →

arxiv 2607.22334 v1 pith:QNWW6I6K submitted 2026-07-24 cs.LG cs.AIcs.CL

Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

Hao Wang , Kun Yuan , Wenlin Zhong , Minglei Zhang , Han Xiao , Ming Sun , Honggang Qi This is my paper
classification cs.LG cs.AIcs.CL
keywords knowledge distillationon-policy distillationcross-tokenizer distillationbyte-prefix marginalizationtokenizer alignmentlanguage modelsgeneralized Jensen-Shannon divergencewhitespace collapse
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that on-policy distillation across different tokenizers can be made exact and mass-preserving by working in the shared byte space. It defines Byte-Prefix Marginalization (BPM): each teacher token's probability is assigned to the longest student token whose byte representation is a prefix of the teacher token's bytes; masses landing on the same student token are summed, and unassignable mass goes into an explicit residual cell. This gives a target over the full student vocabulary that preserves the teacher's total probability mass and that, at more than 99% of training positions, exactly equals the teacher-induced byte-prefix marginal. If correct, the result matters because it removes the shared-tokenizer restriction on full-vocabulary on-policy distillation, letting a compact student learn from any byte-level BPE teacher. The paper reports consistent improvements of 3.7–6.6 avg@8 points over the strongest baselines across three teacher–student pairs and six math/code benchmarks, and it identifies a whitespace-only failure mode that a precomputed mask fixes.

Core claim

BPM re-expresses the teacher's next-token distribution on the student vocabulary. For each teacher token w, the routing map φ(w) returns the longest student token whose bytes prefix bytes(w), or a residual symbol ⊥ if none exists. The target mass on student token u is ti(u)=Σ_{w: φ(w)=u} q(w), and the residual collects the rest, so the target sums to one and is defined for every student token. The paper proves that under greedy longest-prefix segmentation this scatter form equals the exact byte-prefix marginal whenever the prefix does not span multiple teacher tokens—more than 99% of training positions—and uses a mass-preserving, chain-factorized lower bound for the remaining spanning positi

What carries the argument

The load-bearing object is the routing map φ from teacher tokens to the student vocabulary plus a residual: φ(w) is the longest student token whose byte string is a prefix of bytes(w). φ is built once per tokenizer pair by walking a byte trie of the student vocabulary, so at training time most positions reduce to a batched gather-scatter of teacher probabilities. The map does three jobs at once: it partitions the teacher vocabulary (so mass is preserved), it respects byte-level content (so teacher mass does not land on unrelated student tokens), and it covers the full student vocabulary, with ⊥ absorbing unmatched mass. Exactness of the scatter form rests on the greedy longest-prefix segment

Load-bearing premise

The proof of exactness assumes the student tokenizer always emits the longest token whose bytes prefix the remaining stream; real BPE merge order only approximates that rule, so if the two disagree often, the 'exact' target is only approximate at those positions.

What would settle it

Compute the BPM target and the true byte-prefix marginal—obtained by running the student tokenizer's actual BPE merge order over every teacher-token continuation—at all aligned positions on a held-out set of rollouts for a tokenizer pair. If the probability mass differs by more than numerical tolerance at a non-negligible fraction of positions (particularly where greedy longest-prefix and real merge order diverge), the exactness claim is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Full-vocabulary on-policy distillation no longer requires teacher and student to share a tokenizer; any byte-level BPE teacher can supervise a student with a different vocabulary.
  • Because no teacher probability mass is discarded—unmatched mass is kept in an explicit residual—the student receives signal across its entire vocabulary, not just on shared or rank-matched tokens.
  • At more than 99% of training positions the target is the exact byte-prefix marginal; at the remaining spanning positions a mass-preserving chain lower bound keeps the total mass equal to one.
  • Across three teacher–student pairs and six mathematics/code benchmarks, the best BPM variant beats the strongest baseline by 3.7–6.6 avg@8 points and closes 33.5–43.9% of the teacher–student gap.
  • The target also transfers the teacher's stopping decision, cutting the fraction of contest responses that hit the length cap on one pair from 97% to 34%; the whitespace-row mask prevents a code-indentation collapse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same static routing map could be reused for inference-time ensemble projection across vocabularies, since φ depends only on the two tokenizers and can be applied at nearly zero marginal cost.
  • A direct check of the greedy-segmentation idealization—comparing the BPM target with the byte-prefix marginal computed under real BPE merge order—would quantify how often and how much the exactness claim is approximate; this is the paper's own acknowledged gap.
  • The whitespace-collapse finding suggests a general principle for byte-exact distillation: mask positions where the target mass reflects tokenizer segmentation rather than content, which could extend to other low-content rows such as repeated formatting tokens.
  • Combining BPM with a span-level likelihood target could capture multi-token dependencies that per-position marginal targets cannot express, possibly closing more of the teacher–student gap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Byte-Prefix Marginalization (BPM) for full-vocabulary on-policy distillation when teacher and student use different tokenizers. The target is constructed by routing each teacher token's probability mass to the longest student token whose byte representation is a prefix of the teacher token's bytes, aggregating mass per student token, and placing unmatched mass in an explicit residual ∅. Training then minimizes a generalized Jensen-Shannon divergence (or endpoint KL variants) between the BPM target and the student distribution, with additional handling for stop tokens and a mask for all-whitespace rows. The paper claims that under a greedy longest-prefix tokenization assumption, this target exactly recovers the teacher-induced byte-prefix marginal when the relevant student token lies inside a single teacher token, a condition reported at >99% of training positions; otherwise a chain-factorized lower bound is used. Empirically, BPM is evaluated by distilling Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 into Qwen3.5-2B on six mathematics/code benchmarks, reporting avg@8 improvements of 3.7-6.6 points over the strongest baselines, plus ablations of divergence choice, data mixture, whitespace masking, and distributional closeness.

Significance. Cross-tokenizer distillation is practically important, and BPM is an elegant and well-motivated construction: it is vocabulary-complete, byte-aligned, mass-preserving, and depends only on the two tokenizers, with a cheap precomputed routing map. The paper gives a clean formal development (Lemma C.1, Proposition C.2) under clearly stated assumptions, a complete training algorithm, and an unusually careful empirical study with three teacher-student pairs at different tokenizer distances, multiple baselines, paired bootstrap intervals, and a documented failure mode with a fix. If the exactness claim were fully supported, this would be a clear advance over rank-based, overlap-based, and sequence-level methods. However, the central theoretical claim currently rests on an idealized tokenization assumption that the paper itself concedes is not proven to match real BPE, and the headline empirical margins are selected on the test benchmarks; both points need to be addressed before the claims can be taken at face value.

major comments (2)
  1. [Appendix C, Proposition C.2; §3.1, Assumption (A2)] The exactness claim is load-bearing and is not established for real BPE. Proposition C.2 assumes (A2), and the immediately following paragraph concedes that A2 'idealizes BPE merge order,' that no bound is proven, and that only informal position-by-position cross-checks were run. The >99% statistic in the abstract and §1 refers to the segmentation-relation frequencies in Table D.2 (1:1 and 1:N cases), not to whether A2 holds at those positions. Since actual BPE merge order can emit a first token different from the longest byte-prefix match, the statement that the target 'exactly recovers the teacher-induced byte-prefix marginal' is currently an overstatement. Please add a direct quantitative test: at training-rollout positions, compare the greedy longest-prefix first token with the first token produced by the real student tokenizer on the same continuation, and report the mismatch rate a
  2. [§4.2, Table 2 and §H.4] The headline margins are reported for the best of three BPM divergence variants (FKL), and the checkpoint for each arm is selected by its six-benchmark average on the same test benchmarks. The paired bootstrap intervals in §H.4 resample problems for fixed selected checkpoints; they do not cover training-run variability or checkpoint/divergence selection. Because the abstract's 3.7-6.6-point margins are post-selection quantities, the quantitative claim is somewhat stronger than the evidence supports. This is not fatal: the default β=1/2 arm also beats the strongest baseline on every pair. However, please present the default arm as the primary result, or use a held-out validation split for divergence and checkpoint selection, and state the default-arm margins alongside the best-arm margins.
minor comments (5)
  1. [§3.4 vs. Algorithm 1] The main text says 'If stopping has already collapsed, an explicit bridge target supplies a restoring signal,' but Algorithm 1 always includes the stop bridge in R′ and trains it under D0. Please clarify whether the bridge is always active or is triggered by a collapse detector.
  2. [§A, Eq. (A.8)] The skew parameter λ=0.1 in the reverse-KL endpoint is stated without justification or ablation. A brief note on its choice would be helpful, since the endpoint is part of the reported divergence variants.
  3. [Table D.2] The table reports segmentation-relation frequencies on 120-240 responses per pair. It would be useful to clarify whether these responses come from a final BPM student or earlier checkpoints, and whether the 'exact' rows already incorporate verification that A2 held at those positions.
  4. [§F, SeqKD budget] SeqKD is described as following its own offline recipe with 'no smaller budget,' but the actual compute/FLOP comparison is not quantified. Since SeqKD uses three epochs at a different learning rate, please state the compute budget explicitly.
  5. [General] No code or trained checkpoints are provided, despite a mention of a 'reference implementation.' For reproducibility, please include a public code release or detailed artifact link.

Circularity Check

0 steps flagged

No circularity: BPM is a deterministic transform of teacher distribution and vocabularies; exactness is an explicitly stated reformulation, and the admitted (A2) idealization is a correctness gap, not a circular reduction.

full rationale

The paper's derivation chain is self-contained rather than circular. The BPM target is defined directly as the byte-prefix marginal (Eq. 2: t_i(u) := Pr_{w~q_j(i)}[first student token of bytes(w)=u]), and Proposition C.2 proves that the closed-form scatter of Eq. (4) equals this definition under explicit assumptions (A1)-(A2). This is a mathematical equivalence between a definition and a computable aggregation, not a fitted 'prediction' of a separate quantity: the target depends only on the teacher distribution and the two vocabularies, with no constants fitted to benchmarks. The >99% exactness statistic is an empirical alignment measurement, not an input to the derivation. The paper explicitly flags the main limitation in Appendix C: '(A2) idealizes BPE merge order; we argue its gap is second-order without proving a bound'. That is a genuine correctness risk for the exactness claim, but it is an unvalidated idealization rather than a circular step, because the proof does not assume the conclusion. Self-citations (Skill-SD, KAT-Coder, UI-Genie, etc.) appear only as related-work examples and are not load-bearing for the BPM construction. Divergence and mask choices are ablations selected on observed results, which is a selection issue, not equation-level circularity. Overall, the central claim has independent content and the empirical comparison is not forced by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The target definition itself has no fitted constants; it is a deterministic function of the teacher distribution and the two vocabularies. The load-bearing burden sits in behavioral assumptions about tokenization and teacher access, plus hand-set divergence hyperparameters.

free parameters (2)
  • beta (divergence mixing weight) = beta=0 (FKL) in reported best; default beta=1/2
    The generalized Jensen-Shannon divergence has a free mixing weight beta; the main results report FKL because it scored highest on the same six benchmarks (Table 2). This is a post-hoc choice, not a fitted constant in the target construction.
  • lambda (skew in RKL endpoint) = 0.1
    Hand-chosen skew in D1(t||p) = KL(p || (1-lambda)t + lambda p) (Section A); no independent justification is given for the value.
axioms (3)
  • domain assumption A1: every token w decodes to a byte string bytes(w), and decoding a token sequence concatenates the byte strings losslessly (byte-level BPE).
    Section C. True for the byte-level BPE tokenizers used, but stated as an assumption on which the byte-space formulation rests.
  • domain assumption A2: at any byte position, the student tokenizer emits the longest token in V_s whose bytes prefix the remaining byte stream (greedy longest-prefix matching).
    Section C. This is the load-bearing idealization for Proposition C.2 and the '>99% exact' claim. Real BPE merge order is not exactly greedy longest-prefix, and the paper provides no bound on the gap.
  • domain assumption The teacher's next-token distribution q_j is accessible at aligned positions when scoring the student-generated response re-rendered in the teacher's chat template.
    Section 4.1 and Algorithm 1. If teacher scoring under a different chat template changes token boundaries or adds/removes content, the sync-point alignment and targets are not well-defined.

pith-pipeline@v1.3.0-alltime-deepseek · 28969 in / 12466 out tokens · 120349 ms · 2026-08-01T05:03:44.179797+00:00 · methodology

0 comments
read the original abstract

Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD). However, full-vocabulary OPD typically assumes a shared tokenizer, while existing cross-tokenizer methods may discard teacher probability mass or assign it to student tokens with unrelated content. We introduce Byte-Prefix Marginalization (BPM), which re-expresses the teacher's next-token distribution over the student vocabulary in a shared byte space. Specifically, BPM assigns each teacher token's probability to the longest student token whose byte representation is a prefix of the teacher token's bytes, aggregates mass mapped to the same student token, and places otherwise unmatched mass in an explicit residual category. This produces a vocabulary-complete, byte-aligned, and mass-preserving target for dense OPD. The target exactly recovers the teacher-induced byte-prefix marginal when the relevant prefix does not span multiple teacher tokens (a condition satisfied at more than 99% of training positions) and uses a mass-preserving, chain-factorized lower bound otherwise. Across Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 as teachers, BPM consistently outperforms current cross-tokenizer methods on six mathematics and programming benchmarks, improving six-benchmark avg@8 by 3.7-6.6 points over the strongest baselines.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 34 linked inside Pith

  1. [1]

    DeepSeek-V4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026

    DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026

    arXiv 2026

  2. [2]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Kimi K2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Kimi Team. Kimi K2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    GLM-5: From vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    GLM-5 Team. GLM-5: From vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    The MiniMax-M2 series: Mini activations unleashing max real-world intelligence.arXiv preprint arXiv:2605.26494, 2026

    MiniMax. The MiniMax-M2 series: Mini activations unleashing max real-world intelligence.arXiv preprint arXiv:2605.26494, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561, 2023

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.arXiv preprint arXiv:2306.02561, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  8. [8]

    Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1317–1327, 2016

    2016

  9. [9]

    A survey on knowledge distillation of large language models.arXiv preprint arXiv:2402.13116, 2024

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models.arXiv preprint arXiv:2402.13116, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, ...

    Pith/arXiv arXiv 2025

  11. [11]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  12. [12]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations (ICLR), 2024

    2024

  13. [13]

    Why exposure bias matters: An imitation learning perspective of error accumulation in language generation.arXiv preprint arXiv:2204.01171, 2023

    Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation.arXiv preprint arXiv:2204.01171, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents.arXiv preprint arXiv:2604.10674, 2026

    HaoWang, GuozhiWang, HanXiao, YufengZhou, YuePan, JichaoWang, KeXu, YafeiWen, XiaohuRuan, Xiaoxin Chen, et al. Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents.arXiv preprint arXiv:2604.10674, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    On-policy distillation

    Kevin Lu and Thinking Machines Lab. On-policy distillation. https://thinkingmachines.ai/blog/ on-policy-distillation/, 2025

    2025

  16. [16]

    Bild: Bi-directional logits difference loss for large language model distillation.arXiv preprint arXiv:2406.13555, 2025

    Minchong Li, Feng Zhou, and Xiaohui Song. Bild: Bi-directional logits difference loss for large language model distillation.arXiv preprint arXiv:2406.13555, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Sparse logit sampling: Accelerating knowledge distillation in llms.arXiv preprint arXiv:2503.16870, 2025

    Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, and Joohyung Lee. Sparse logit sampling: Accelerating knowledge distillation in llms.arXiv preprint arXiv:2503.16870, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Neural machine translation of rare words with subword units.arXiv preprint arXiv:1508.07909, 2016

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units.arXiv preprint arXiv:1508.07909, 2016

    Pith/arXiv arXiv 2016

  19. [19]

    How good is your tokenizer? on the monolingual performance of multilingual language models.arXiv preprint arXiv:2012.15613, 2021

    Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models.arXiv preprint arXiv:2012.15613, 2021. 10

    Pith/arXiv arXiv 2012

  20. [20]

    Knowledge fusion of large language models

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  21. [21]

    Towards cross-tokenizer distillation: the universal logit distillation loss for llms.Transactions on Machine Learning Research (TMLR), 2024

    Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. Towards cross-tokenizer distillation: the universal logit distillation loss for llms.Transactions on Machine Learning Research (TMLR), 2024

    2024

  22. [22]

    Dual-space knowledge distillation for large language models

    Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. Dual-space knowledge distillation for large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 18164–18181, 2024

    2024

  23. [23]

    Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models

    Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, and Houqiang Li. Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23724–23732, 2025

    2025

  24. [24]

    Universal cross-tokenizer distillation via approximate likelihood matching

    Benjamin Minixhofer, Ivan Vulić, and Edoardo Maria Ponti. Universal cross-tokenizer distillation via approximate likelihood matching. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  25. [25]

    Unlocking on-policy distillation for any model family

    Carlos Miguel Patiño, Kashif Rasul, Quentin Gallouédec, Ben Burtenshaw, Sergio Paniego, Vaibhav Srivastav, Thibaud Frere, Ed Beeching, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Unlocking on-policy distillation for any model family. Hugging Face research article,https://huggingface.co/spaces/HuggingFaceH4/ on-policy-distillation, 2025. Introduces...

    2025

  26. [26]

    Simct: Recovering lost supervision for cross-tokenizer on-policy distillation.arXiv preprint arXiv:2605.07711, 2026

    Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, Yilin Cheng, Bichuan Feng, Pengfei Liu, Junfeng Fang, and Xiang Wang. Simct: Recovering lost supervision for cross-tokenizer on-policy distillation.arXiv preprint arXiv:2605.07711, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Clark, Dan Garrette, Iulia Turc, and John Wieting

    Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. Canine: Pre-training an efficient tokenization-free encoder for language representation.arXiv preprint arXiv:2103.06874, 2022

    Pith/arXiv arXiv 2022

  28. [28]

    Byt5: Towards a token-free future with pre-trained byte-to-byte models.arXiv preprint arXiv:2105.13626, 2022

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models.arXiv preprint arXiv:2105.13626, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    Megabyte: Predicting million-byte sequences with multiscale transformers.arXiv preprint arXiv:2305.07185, 2023

    Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers.arXiv preprint arXiv:2305.07185, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Byte latent transformer: Patches scale better than tokens.arXiv preprint arXiv:2412.09871, 2024

    Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. Byte latent transformer: Patches scale better than tokens.arXiv preprint arXiv:2412.09871, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    FitNets: Hints for thin deep nets

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. InInternational Conference on Learning Representations (ICLR), 2015

    2015

  32. [32]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. InFindings of the Association for Computational Linguistics: ACL 2023, pages 8003–8017, 2023

    2023

  33. [33]

    Teaching small language models to reason.arXiv preprint arXiv:2212.08410, 2023

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason.arXiv preprint arXiv:2212.08410, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Miniplm: Knowledge distillation for pre-training language models.arXiv preprint arXiv:2410.17215, 2025

    Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Miniplm: Knowledge distillation for pre-training language models.arXiv preprint arXiv:2410.17215, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    UI-Genie: A self-improving approach for iteratively boosting MLLM-based mobile GUI agents.Advances in Neural Information Processing Systems, 38:150376–150411, 2025

    Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, et al. UI-Genie: A self-improving approach for iteratively boosting MLLM-based mobile GUI agents.Advances in Neural Information Processing Systems, 38:150376–150411, 2025

    2025

  36. [36]

    UI-Mem: Self-evolving experience memory for online reinforcement learning in mobile GUI agents.arXiv preprint arXiv:2602.05832, 2026

    Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yuxiang Chai, Yue Pan, Yufeng Zhou, Xiaoxin Chen, Yafei Wen, and Hongsheng Li. UI-Mem: Self-evolving experience memory for online reinforcement learning in mobile GUI agents.arXiv preprint arXiv:2602.05832, 2026

    arXiv 2026

  37. [37]

    Autoregressive knowledge distillation through imitation learning.arXiv preprint arXiv:2009.07253, 2020

    Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning.arXiv preprint arXiv:2009.07253, 2020. 11

    Pith/arXiv arXiv 2009

  38. [38]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Pith/arXiv arXiv 2026

  39. [39]

    f-divergence minimization for sequence-level knowledge distillation.arXiv preprint arXiv:2307.15190, 2023

    Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation.arXiv preprint arXiv:2307.15190, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Distillm: Towards streamlined distillation for large language models.arXiv preprint arXiv:2402.03898, 2024

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models.arXiv preprint arXiv:2402.03898, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Distillm-2: A contrastive approach boosts the distillation of llms.arXiv preprint arXiv:2503.07067, 2025

    Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. Distillm-2: A contrastive approach boosts the distillation of llms.arXiv preprint arXiv:2503.07067, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Kat-coder-v2 technical report.arXiv preprint arXiv:2603.27703, 2026

    KwaiKAT Team. Kat-coder-v2 technical report.arXiv preprint arXiv:2603.27703, 2026

    arXiv 2026

  43. [43]

    Kat-coder-v2.5 technical report.arXiv preprint arXiv:2607.05471, 2026

    KwaiKAT Team. Kat-coder-v2.5 technical report.arXiv preprint arXiv:2607.05471, 2026

    Pith/arXiv arXiv 2026

  44. [44]

    Zero-shot tokenizer transfer.arXiv preprint arXiv:2405.07883, 2025

    Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. Zero-shot tokenizer transfer.arXiv preprint arXiv:2405.07883, 2025

    arXiv 2025

  45. [45]

    Breaking the ceiling of the LLM community by treating token generation as a classification for ensembling

    Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Chang, and Yueh-Se Li. Breaking the ceiling of the LLM community by treating token generation as a classification for ensembling. InFindings of the Association for Computational Linguistics: EMNLP 2024, 2024

    2024

  46. [46]

    Bridging the gap between different vocabularies for LLM ensemble

    Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. Bridging the gap between different vocabularies for LLM ensemble. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 7140–7152, 2024

    2024

  47. [47]

    Determine- then-ensemble: Necessity of top-k union for large language model ensembling.arXiv preprint arXiv:2410.03777, 2025

    Yuxuan Yao, Han Wu, Mingyang Liu, Sichun Luo, Xiongwei Han, Jie Liu, Zhijiang Guo, and Linqi Song. Determine- then-ensemble: Necessity of top-k union for large language model ensembling.arXiv preprint arXiv:2410.03777, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    Qwen3.5-2b.https://huggingface.co/Qwen/Qwen3.5-2B, 2026

    Qwen Team. Qwen3.5-2b.https://huggingface.co/Qwen/Qwen3.5-2B, 2026

    2026

  49. [49]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    GLM Team. Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    GLM-Z1-9B-0414: Model card

    Z.ai. GLM-Z1-9B-0414: Model card. Hugging Face model card, https://huggingface.co/zai-org/ GLM-Z1-9B-0414, 2025

    2025

  51. [51]

    DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  52. [52]

    TACO: Topics in algorithmic COde generation dataset.arXiv preprint arXiv:2312.14852, 2023

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. TACO: Topics in algorithmic COde generation dataset.arXiv preprint arXiv:2312.14852, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021

    2021

  54. [54]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  55. [55]

    MathArena: Evaluating LLMs on uncontaminated math competitions.https://matharena.ai/, 2025

    Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. MathArena: Evaluating LLMs on uncontaminated math competitions.https://matharena.ai/, 2025

    2025

  56. [56]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  57. [57]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Representations (ICLR), 2025. 12

    2025

  58. [58]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  59. [59]

    DeepMath-103K: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. DeepMath-103K: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    Pith/arXiv arXiv 2025

  60. [60]

    Realized-path

    Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. KodCode: A diverse, challenging, and verifiable synthetic dataset for coding.arXiv preprint arXiv:2503.02951, 2025. 13 The appendix follows the dependency chain behind the main result. We first restate the notation and target definitions (Section A), turn them into a complete trainin...

    Pith/arXiv arXiv 2025