On chemistry SMILES with a fixed 165-token base, BPE and Unigram-LM produce near-disjoint vocabularies (Jaccard ≤0.161) and Unigram-LM emits 29–41% more tokens across 22 matched conditions.
Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
years
2026 4representative citing papers
SVD on the lm_head weight matrix of transformers reveals interpretable vocabulary clusters that indicate training data composition, model differences, and ethical concerns in models like GPT-OSS, Gemma, and Qwen.
citing papers explorer
-
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
On chemistry SMILES with a fixed 165-token base, BPE and Unigram-LM produce near-disjoint vocabularies (Jaccard ≤0.161) and Unigram-LM emits 29–41% more tokens across 22 matched conditions.
-
Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)
SVD on the lm_head weight matrix of transformers reveals interpretable vocabulary clusters that indicate training data composition, model differences, and ethical concerns in models like GPT-OSS, Gemma, and Qwen.
- Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
- Faster Superword Tokenization