Pith. sign in

REVIEW 5 major objections 4 minor 27 references

Bit-level BPE: Below the byte boundary

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a simple bit-level re-encoding of UTF-8 bytes—one 6-bit prefix token plus two 9-bit tokens per three-byte character—losslessly reduces byte-level BPE sequence length for CJK text and lowers decoding errors.

desk verdict A genuinely new sub-byte codec for CJK byte fallbacks, but the lossless claim overreaches, Equation (3) has a typo, and the empirical support is too weak to confirm the throughput story. read the letter →

arxiv 2506.07541 v1 pith:5J2ZX75S submitted 2025-06-09 cs.CL

Sangwhan Moon , Tatsuya Hiraoka , Naoaki Okazaki This is my paper
classification cs.CL
keywords subwordtokenizationbyte-levelBPEUTF-8losslesscompressionCJKlanguagessequencelengthRényiefficiencyneuralmachinetranslation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Byte-level fallback in BPE guarantees no out-of-vocabulary tokens but inflates sequence length, especially for Chinese, Japanese, and Korean, where each character becomes three UTF-8 bytes. This paper claims that most of the redundant information lives in the shared leading bits of those bytes, and proposes a lossless re-encoding that splits each three-byte character into one 6-bit prefix token and two 9-bit tokens, then deduplicates repeated prefixes. The result is shorter token sequences for CJK-heavy text, and in the paper's translation experiments, fewer invalid UTF-8 decodes from a small model. The cost is a modestly larger token vocabulary and a drop in tokenization entropy, which the authors accept in exchange for sequence length.

What carries the argument

The load-bearing mechanism is the 6/9/9 bit re-split of a three-byte UTF-8 character. For bytes $b_1,b_2,b_3$, the encoder computes $\hat{b}_1 = (b_1 \wedge 127) \gg 2$, $\hat{b}_2 = (((b_1 \wedge 3) \ll 7) \vee ((b_2 \wedge 254) \gg 1))$, and $\hat{b}_3 = ((b_2 \wedge 1) \ll 8) \vee b_3$. The 6-bit prefix $\hat{b}_1$ is shared across the CJK blocks ($0xE4$--$0xEF$), so after the first occurrence it can be omitted until the prefix changes; decoding relies on the deterministic rule that each encoded character is exactly one prefix token followed by two 9-bit tokens, so the decoder re-emits the current prefix after every two 9-bit tokens. This carries the entire compression: it removes roughly one third of the bytes' worth of tokens per character while remaining exactly invertible.

What would settle it

Take a UTF-8 string containing a 4-byte character (for example an emoji) or any string that mixes 3-byte and 4-byte characters, apply the paper's encoder exactly as defined (one 6-bit prefix and two 9-bit tokens per character), then apply the decoder. If the output is not the original byte sequence, or if the boundary between characters is misaligned, the fixed 6/9/9 shape is not lossless in general. A simpler check: compute the token count for a corpus dominated by such characters; if the 22.22% per-character reduction disappears or becomes negative, the method's scope is limited to 3-byte CJK runs.

Watch

Extended reading notes

Core claim

The central discovery is that UTF-8's self-synchronizing structure, which repeats a common prefix across every byte of a character, can be factored out once the byte boundary is no longer treated as fixed at eight bits. For a three-byte CJK character like 召, whose bytes are E4 BC 97, the paper shifts boundaries to form a 6-bit prefix p1 (0b111001) and two 9-bit tokens 5E and 97, via the bit operations in equations (3)--(5). Repeating p1 for each character is then unnecessary: the encoder emits the prefix only when it changes, so 召唤众 (E4 BC 97 E5 94 A4 E4 BC 97) becomes p1 5E 97 CA A4 5E 97, a 22.22% reduction in tokens. Decoding re-inserts the current prefix after every two 9-bit tokens and reverses the bit shifts with equations (6)--(8) to recover original UTF-8 bytes exactly. The paper argues this is possible because the model sees tokens as logits, so a 'byte' token may hold any number of bits.

Load-bearing premise

The decoder's correctness rests on every target character being exactly three UTF-8 bytes, so each encoded character is one 6-bit prefix followed by two 9-bit tokens; four-byte characters like emoji or misaligned byte streams break the re-emit-after-two rule, and the paper provides no general fallback for them.

Editorial extensions

If this is right

  • On CJK-heavy training corpora, total tokenized sequence length drops by 0.83% to 6.41% in the paper's experiments, with larger gains when byte fallback is more frequent (Chinese 48.8%, Korean 52%).
  • The number of undecodable outputs from a 65M-parameter translation model falls sharply: from 136 to 0 for Japanese, 1522 to 121 for Korean, and 33 to 14 for Chinese across 5,000 test samples.
  • Throughput expressed as perceived tokens per second improves by a relative gain factor of up to 1.0334 for English-Chinese, after accounting for the reduced sequence length of the same reference text.
  • The tokenizer needs 256 extra 9-bit tokens plus three prefix tokens, increasing the embedding size; unreachable byte tokens could be recycled to offset this parameter growth.
  • Tokenization entropy, measured by Rényi efficiency, decreases (e.g., from 0.764 to 0.634 for English-Chinese), so the method trades a more peaked byte-token distribution for shorter sequences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same prefix-deduplication idea should extend to 4-byte UTF-8 characters (e.g., emoji) with a 6/9/9/9 split, but the decoder's re-emit-after-two rule would need to become re-emit-after-three; until that is specified, applying this method to emoji-rich text is unsafe.
  • Beyond the paper: the 6-bit prefix length and 9-bit trailing tokens were chosen for CJK; one could optimize these boundaries per script or per corpus to balance sequence length, extra vocabulary, and entropy, as the authors note is future work.
  • Beyond the paper: the perceived-TPS measure could compare standard tokenizers with very different vocabulary sizes, such as a 32K-token and a 128K-token subword model, where the sequence-length delta is larger than in these experiments.
  • Beyond the paper: because the method is tokenizer-agnostic and lossless, it could be applied before BPE merging rather than only as a fallback representation, potentially compounding compression; the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes a byte-level tokenization compression scheme that re-encodes UTF-8 byte sequences at sub-byte granularity. The core idea is to split each 3-byte CJK character into a 6-bit prefix token and two 9-bit tokens, then deduplicate repeated prefixes across consecutive characters. The authors report deterministic sequence-length reductions on Chinese, Japanese, and Korean translation datasets, and introduce a 'perceived TPS' metric to factor tokenization length into throughput comparisons. The paper also reports translation quality and decoding-error experiments for a 65M-parameter transformer, plus an abandoned fine-tuning experiment.

Significance. If the scheme is fully specified and scoped correctly, it is a simple and original contribution to byte-level tokenization: the 6/9/9 split is a concrete, implementable idea for reducing the token-count overhead of CJK byte fallbacks, and the sequence-length reductions in Table 3 are direct measurements rather than fitted results. The paper honestly discusses limitations and negative results, including the failed LoRA experiment and the fixed-boundary restriction. However, the central lossless-compression claim is currently stated more broadly than the algorithm supports, and there are specification errors and ambiguities in the worked example that prevent reproduction from the equations alone.

major comments (5)
  1. [§3.3, Eq. (3)] Equation (3) as printed gives the wrong prefix value for the worked example. For b1 = 0xE4, (b1 ∧ 127) >> 2 = (0x64) >> 2 = 0x19, but the text and example use p1 = 0x39 and state that p1 is 0b111001, which would be obtained by b1 >> 2. This is a load-bearing error: a reader cannot implement the codec from the equations alone. Please correct Eq. (3) to b1 >> 2, or equivalently (b1 ∧ 0xFF) >> 2, and verify all subsequent equations against the corrected prefix.
  2. [§3.3, worked example] The decoding example contains an inconsistent reconstruction. The input 召唤众認 has UTF-8 bytes E4 BC 97 E5 94 A4 E4 BC 97 E8 AA 8D. Using the stated tokens p1 5E 97 p1 CA A4 p1 5E 97 p2 55 8D and Eqs. (6)–(8), the reconstruction is exactly those bytes. The paper instead prints 'E4BB BDE5 8187 E7AE 80E8 94B5', which is a different sequence. Please correct the reconstructed byte sequence and check the surrounding explanation for typographical spacing, since tokens such as 'p15E' and 'p255' are ambiguous as typeset.
  3. [§3.3 and Abstract/Ethical Statement] The lossless-compression claim is only established for runs of 3-byte CJK characters, because decoding relies on a deterministic shape of exactly one 6-bit prefix token followed by two 9-bit tokens per character ('the specific Unicode blocks our method targets have a deterministic length of three bytes per character'). No encoding or decoding rule is given for ASCII bytes, 2-byte or 4-byte UTF-8 characters, or mixed-length byte streams, which are ubiquitous in actual byte-level BPE output from tokenizers such as Llama2. Consequently, the abstract's claim of a general 'lossless' sequence-compression technique and the Ethical Statement's description of the method as 'data and task-agnostic' overstate the method's scope. The paper should either restrict all claims to the 3-byte CJK setting, or provide a complete fallback mechanism for other byte lengths and mixed streams, with a decoding rule that does not require a priori knowledge of character boundaries.
  4. [§3.3, footnote on prefix tokens] The handling of the prefix token value is ambiguous and appears contradictory. The text says a special token is used 'instead of the byte representation (0x39) to disambiguate between naturally occurring bytes and incremental decoder triggers,' but the footnote immediately says 'all prefix tokens can usurp existing byte tokens, as they are unreachable. For example, 0x39 is the character "9".' In UTF-8 byte-level BPE, byte 0x39 is a perfectly reachable ASCII token for the digit '9', so it cannot simply be usurped without defining how a literal '9' is encoded. If prefix tokens share values with ordinary byte tokens, the decoder cannot distinguish a prefix from a literal byte; if they do not share values, the paper should state the actual token values. This ambiguity must be resolved for the lossless claim to hold even within the CJK-restricted setting.
  5. [§5, Table 6] The empirical support for the stated motivation of 'eventually saving compute time' is weak and partially contradicted by the reported wall-clock measurements. In Table 6, the en-zh 'Ours' total inference time is 201.65s versus 72.41s for the byte baseline, and the ja-ko times are nearly identical (41.59s vs 41.10s), while en-ja is faster. The 'perceived TPS' metric rescales TPS by the ratio of reference-token counts, which assumes a linear relationship between sequence length and throughput that is not supported by these measured times. Please clarify whether the method's claimed benefit is purely token-count reduction (which Table 3 supports) or actual compute savings, and if the latter, present evidence that the reduced sequence length translates into wall-clock gains in at least the primary setting, or explicitly identify the settings where it does not.
minor comments (4)
  1. [§2.3] There is a typo in 'Rényi effiency' (should be 'efficiency'), and the paper should specify the value of α used when reporting the entropy values in Table 3, since Eq. (2) is parameterized by α.
  2. [§4, Table 5] The column heading 'Size' is not defined; it appears to be the number of training examples, but this should be stated explicitly. Also, the Chinese and Japanese BLEU results are acknowledged as not meaningful for quantitative comparison, so the paper should avoid presenting them as evidence of quality gains without a clear caveat in the text.
  3. [§A.3] The artifacts section says the reference implementation and pretrained models will be distributed at '[To be populated after CR]'. For a methods paper, the reference implementation should be available or an anonymized repository should be linked so the equations can be checked against actual code.
  4. [§3.3] The notation for tokens is confusing: sequences such as 'p15E 97' and 'p255 8D' should be typeset with explicit separators, for example 'p1 5E 97' and 'p2 55 8D', so that prefix tokens are not accidentally read as hexadecimal numbers or as a single token 'p255'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the bit-level codec is self-contained, parameter-free, and its lossless round-trip is demonstrated by explicit inverse operations; the CJK-only scope is a stated limitation rather than a circular derivation.

full rationale

The paper's central claim is a concrete codec: 3-byte CJK UTF-8 characters are re-partitioned into one 6-bit prefix and two 9-bit tokens (§3.3), repeated prefixes are deduplicated, and decoding re-inserts the prefix after every two 9-bit tokens. Losslessness is established by the mutually inverse bit operations (3)–(5) and (6)–(8), and it is not obtained by fitting a parameter to the data whose reduction is later reported. The sequence-length reductions in Table 3 are deterministic applications of this fixed codec to the training corpora; no parameter was optimized against the test set or against the measured reductions. The perceived TPS metric is explicitly defined as TPS multiplied by the relative sequence-length ratio |Tc|/|Te|, with its assumptions stated, so no result is being relabeled as a prediction. There are no load-bearing self-citations: the reference list contains no author-overlapping citations used to justify the method's core operations. The paper itself flags the scope limitation in Section 7 and the Limitations section, noting the fixed bit boundaries are restricted to CJK and that emoji are excluded; Section 3.3 states that the decoder relies on the deterministic three-byte length of the targeted Unicode blocks. That is a limitation of the abstract's unqualified 'lossless' phrasing and a correctness concern, not a circular step, because the codec's validity is checked against the UTF-8 byte stream rather than assumed from its own output. Accordingly, no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The central claim rests on two design choices: the fixed 6/9/9 bit split and the added token set, both hand-derived from UTF-8's 3-byte CJK structure. The method assumes the deterministic 3-byte length for targeted blocks and assumes that a standard Transformer can assimilate extra non-byte tokens. The perceived TPS metric is an additional ad-hoc assumption about how sequence length relates to expressive power. No hidden fitted parameters are used to obtain the lossless reduction numbers.

free parameters (2)
  • bit boundaries (6, 9, 9) = 6, 9, 9
    Chosen by hand to match UTF-8 CJK 3-byte structure (Table 2); not optimized. Section 3.3 says 'the bit boundaries were set to be optimal for CJK scripts' but no search is described.
  • added tokens (256 9-bit tokens + 3 prefix tokens) = 259
    Derived from bit redistribution: 2^9 - 2^8 = 256 extra tokens plus p1, p2, p3. This is part of the codec design, not fitted to data.
assumptions (4)
  • domain assumption Each CJK Unified Ideograph and Hangul syllable is encoded in exactly 3 bytes in UTF-8.
    Section 3.3 relies on 'the specific Unicode blocks our method targets have a deterministic length of three bytes per character.' True for the BMP CJK blocks cited, but not for supplementary characters or emoji.
  • domain assumption The model's vocabulary can contain tokens 0x100-0x1FF and the three prefix tokens without violating tokenizer constraints.
    Section 3.3 states 'the model sees the individual bytes as logits, which does not require them to be aligned to eight bits.' This assumes arbitrary token IDs are usable in a standard BPE vocabulary.
  • standard math Rényi efficiency (renamed 'entropy') is a valid proxy for tokenization quality, and lower entropy implies quality degradation.
    Section 2.3 cites Zouhar et al. (2023) and uses this to interpret Table 3 as showing a trade-off between entropy and sequence length.
  • ad hoc to paper Relative sequence length is a valid multiplier for throughput (the perceived TPS metric).
    Section 5 states 'the assumption made here is that given the same text, a tokenizer that is capable of encoding it with less tokens has higher expressive power.' This is the paper's own proposed metric, not externally validated.
invented entities (2)
  • 9-bit byte tokens (0x100-0x1FF)
    purpose: Carry residual bits when re-cutting 3-byte UTF-8 characters into 6+9+9 bit fields.
    Engineering construct added to the vocabulary; no external falsifiable handle.
  • Prefix tokens p1, p2, p3
    purpose: Signal shared 6-bit prefixes so repeated prefixes can be omitted from the sequence.
    New token IDs introduced by the method; no external evidence beyond the paper's own experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bit-level BPE: Below the byte boundary." pith.science (2026). https://pith.science/paper/5J2ZX75S

@misc{pith2026250607541,
  author       = {Pith},
  title        = {Pith review of: Bit-level BPE: Below the byte boundary},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5J2ZX75S}},
  note         = {Machine review of arXiv:2506.07541}
}
read the original abstract

Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragmatic solution for preventing OOV, especially in the context of larger models. However, breaking a character down to individual bytes significantly increases the sequence length for long-tail tokens in languages such as Chinese, Japanese, and Korean (CJK) and other character-diverse contexts such as emoji. The increased sequence length results in longer computation during both training and inference. In this work, we propose a simple compression technique that reduces the sequence length losslessly.

Figures

Figures reproduced from arXiv: 2506.07541 by the authors.

Figure 1
Figure 1. High-level overview of byte-level text rep [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Byte-level frequency distribution on the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Byte-level distribution shift, with and without our method on the Japanese-Korean subtitle translation task. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Byte embeddings in Llama2, dimensionality reduced with PCA ( [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 6 canonical work pages

  1. [1]

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.614 Do all languages cost the same? tokenization in the era of commercial language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904--9923, ...

    doi:10.18653/v1/2023.emnlp-main.614 2023

  2. [2]

    Lo \" c Barrault, Magdalena Biesialska, Ond r ej Bojar, Marta R. Costa-juss \`a , Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljube s i \'c , Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 202...

    work page 2020

  3. [3]

    Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun ' ichi Tsujii. 2020. https://doi.org/10.18653/v1/2020.coling-main.609 C haracter BERT : Reconciling ELM o and BERT for word-level open-vocabulary representations from characters . In Proceedings of the 28th International Conference on Computational Linguistics,...

    doi:10.18653/v1/2020.coling-main.609 2020

  4. [4]

    Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. arXiv preprint arXiv:2403.06265

    arXiv 2024

  5. [5]

    Thamme Gowda and Jonathan May. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.352 Finding the optimal vocabulary size for neural machine translation . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3955--3964, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.352 2020

  6. [6]

    Masanori Hirano, Masahiro Suzuki, and Hiroki Sakaji. 2023. https://doi.org/10.48550/arXiv.2305.12720 llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology

    work page Pith review arXiv doi:10.48550/arxiv.2305.12720 2023

  7. [7]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre...

    arXiv 2022

  8. [8]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

Show all 27 references
  1. [9]

    Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andr \'e F. T. Martins, and Alexandra Birch. 2018. https://doi.org/10.18653/v1/P18-4020 M arian: Fast neura...

    2018 doi

  2. [10]

    Sander Land and Max Bartolo. 2024. https://arxiv.org/abs/2405.05417 Fishing for magikarp: Automatically detecting under-trained tokens in large language models . Preprint, arXiv:2405.05417

    2024 arXiv

  3. [11]

    Jind r ich Libovick \'y and Alexander Fraser. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.203 Towards reasonably-sized character-level transformer NMT by finetuning subword systems . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processin...

    2020 doi

  4. [12]

    Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gall \'e , Arun Raja, Chenglei Si, Wilson Y Lee, Beno \^ t Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. arXiv preprin...

    2021 arXiv

  5. [13]

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2024. Language model tokenizers introduce unfairness between languages. Advances in Neural Information Processing Systems, 36

    2024

  6. [14]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

    2018 doi

  7. [15]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners

    2019

  8. [16]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. 2021. https://doi.org/10.18653/v1/2021.acl-long.243 How good is your tokenizer? on the monolingual performance of multilingual language models . In Proceedings of the 59th Annual Meeting of the ...

    2021 doi

  9. [17]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  10. [18]

    Uri Shaham and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.naacl-main.17 Neural machine translation without embeddings . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag...

    2021 doi

  11. [19]

    Makesh Narsimhan Sreedhar, Xiangpeng Wan, Yu Cheng, and Junjie Hu. 2023. https://doi.org/10.18653/v1/2023.acl-long.397 Local byte fusion for neural machine translation . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2023 doi

  12. [20]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  13. [21]

    Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2019. https://arxiv.org/abs/1909.03341 Neural machine translation with byte-level subwords . Preprint, arXiv:1909.03341

    2019 arXiv

  14. [22]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  15. [23]

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. https://doi.org/10.1162/tacl_a_00461 B y T 5: Towards a token-free future with pre-trained byte-to-byte models . Transactions of the Association for Computa...

    2022 doi

  16. [24]

    Xiang Zhang and Yann LeCun. 2017. Which encoding is the best for text classification in chinese, english, japanese and korean? arXiv preprint arXiv:1708.02657

    2017 arXiv

  17. [25]

    Vil \'e m Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. https://doi.org/10.18653/v1/2023.acl-long.284 Tokenization and the noiseless channel . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (...

    2023 doi

  18. [26]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  19. [27]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.