Pith. sign in

REVIEW 6 cited by

Getting the most out of your tokenizer for pre-training and domain adaptation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.01035 v2 pith:3B5LZOX3 submitted 2024-02-01 cs.CL

Gautier Dagan , Gabriel Synnaeve , Baptiste Rozière This is my paper
classification cs.CL
keywords tokenizergenerationmodelpre-trainedsizeablationscodecontext
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Tokenization is an understudied and often neglected component of modern LLMs. Most published works use a single tokenizer for all experiments, often borrowed from another model, without performing ablations or analysis to optimize tokenization. Moreover, the tokenizer is generally kept unchanged when fine-tuning a base model. In this paper, we show that the size, pre-tokenization regular expression, and training data of a tokenizer can significantly impact the model's generation speed, effective context size, memory usage, and downstream performance. We train specialized Byte-Pair Encoding code tokenizers, and conduct extensive ablations on the impact of tokenizer design on the performance of LLMs for code generation tasks such as HumanEval and MBPP, and provide recommendations for tokenizer hyper-parameters selection and switching the tokenizer in a pre-trained LLM. We perform our experiments on models trained from scratch and from pre-trained models, verifying their applicability to a wide range of use-cases. We find that when fine-tuning on more than 50 billion tokens, we can specialize the tokenizer of a pre-trained LLM to obtain large gains in generation speed and effective context size.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models

    cs.CL 2026-05 conditional novelty 7.0 of 10

    Scratchpad Patching decouples compute from patch size in byte-level language models by inserting entropy-triggered scratchpads to update patch context dynamically.

  2. BMdataset: A Musicologically Curated LilyPond Dataset

    cs.SD 2026-04 unverdicted novelty 7.0 of 10

    A musicologically curated LilyPond dataset of 393 Baroque scores enables LilyBERT to outperform large-scale pre-training on composer and style classification when used alone for fine-tuning.

  3. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

    cs.LG 2025-02 unverdicted novelty 7.0 of 10

    A recurrent-depth architecture enables language models to improve reasoning performance by iterating computation in latent space, achieving gains equivalent to much larger models on benchmarks.

  4. In-Place Tokenizer Expansion for Pre-trained LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Continuing a model's own BPE merges and training only new embedding rows preserves quality while cutting token counts 2.4–4× for previously under-tokenized languages.

  5. BMdataset: A Musicologically Curated LilyPond Dataset

    cs.SD 2026-04 conditional novelty 6.0 of 10

    Expert-curated Baroque LilyPond scores plus LilyBERT beat large noisy MIDI-style pretraining on composer and style classification, with best results from combining both.

  6. Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis

    cs.CL 2025-07 reject novelty 5.0 of 10

    The authors release eight Bengali benchmarks translated from English and report that models with more fragmented Bengali tokenization tend to score lower.

Pith tools