BrahmicTokenizer-131K is a 131K-vocab tokenizer constructed via script-prune crop and linear-programming retrofit to o200k_base, achieving 26.7% fewer tokens on Indic text while matching o200k_base on English fertility and outperforming alternatives on code/math benchmarks.
Efficient softmax approximation for GPUs
6 Pith papers cite this work. Polarity classification is still indexing.
6
Pith papers citing it
abstract
We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computation time. Our approach further reduces the computational time by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax. The code of our method is available at https://github.com/facebookresearch/adaptive-softmax.
representative citing papers
SoftWater, a KL-divergence-based quantizer for LLM softmax heads, allocates bit rate by class frequency and variance and beats WaterSIC at matched head rates on 59 of 60 test points.
SpeLLM converts a standard token-based LLM into a character-spelling model with multiple parallel output heads, achieving competitive downstream performance with a 5.1% average decoding speedup.
Compressive Transformer sets new records on WikiText-103 (17.1 ppl) and Enwik8 (0.97 bpc) via memory compression and introduces the PG-19 long-range language benchmark.
A recurrent-Transformer hybrid that compresses long-range context into a single RNN state vector matches segment-based Transformer baselines with fewer FLOPs on two language benchmarks and one video benchmark.
DETM's test-set word prediction is robust to loss reweighting and prior recomputation, but prefers larger vocabularies and fewer time windows across five diachronic corpora.
citing papers explorer
-
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
BrahmicTokenizer-131K is a 131K-vocab tokenizer constructed via script-prune crop and linear-programming retrofit to o200k_base, achieving 26.7% fewer tokens on Indic text while matching o200k_base on English fertility and outperforming alternatives on code/math benchmarks.
-
SoftWater: Class-Aware Rate Allocation for Softmax Quantization
SoftWater, a KL-divergence-based quantizer for LLM softmax heads, allocates bit rate by class frequency and variance and beats WaterSIC at matched head rates on 59 of 60 test points.
-
SpeLLM: Character-Level Multi-Head Decoding
SpeLLM converts a standard token-based LLM into a character-spelling model with multiple parallel output heads, achieving competitive downstream performance with a 5.1% average decoding speedup.
-
Compressive Transformers for Long-Range Sequence Modelling
Compressive Transformer sets new records on WikiText-103 (17.1 ppl) and Enwik8 (0.97 bpc) via memory compression and introduces the PG-19 long-range language benchmark.
-
Compact Recurrent Transformer with Persistent Memory
A recurrent-Transformer hybrid that compresses long-range context into a single RNN state vector matches segment-based Transformer baselines with fewer FLOPs on two language benchmarks and one video benchmark.
-
Dynamic Embedded Topic Models: properties and recommendations based on diverse corpora
DETM's test-set word prediction is robust to loss reweighting and prior recomputation, but prefers larger vocabularies and fewer time windows across five diachronic corpora.