■ 26More generally, Lemma 3 holds for any tokenization model with a bigram-based canonicality test (i.e., δ ∈ D ⇐ ⇒BIGRAMS (δ) ⊆ B

=⇒ since the bigrams of δ are all canonical, by Theorem 1, δ ∈ D Therefore, − →D = D

1 Pith paper cite this work. Polarity classification is still indexing.

1 Pith paper citing it

browse 1 citing papers

representative citing papers

cs.CL · 2025-06-09 · conditional · novelty 6.0

Enforcing canonical BPE tokenizations through conditioning or architectural constraints improves held-out likelihood for GPT-2 and Llama models.

Showing 1 of 1 citing paper.

Language Models over Canonical Byte-Pair Encodings cs.CL · 2025-06-09 · conditional · none · ref 5
Enforcing canonical BPE tokenizations through conditioning or architectural constraints improves held-out likelihood for GPT-2 and Llama models.