Pith. sign in

REVIEW 1 cited by

Tokenisation is NP-Complete

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.15210 v1 pith:VCKOJDW7 submitted 2024-12-19 cs.DS cs.CLcs.FL

Tokenisation is NP-Complete

Philip Whittington , Gregor Bachmann , Tiago Pimentel This is my paper
classification cs.DS cs.CLcs.FL
keywords tokenisationbottom-upcompressingdatasetdefineddeltadirectdirectly
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
abstract

In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most $\delta$ symbols by either finding a vocabulary directly (direct tokenisation), or selecting a sequence of merge operations (bottom-up tokenisation).

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MultimodalHugs: Enabling Sign Language Processing in Hugging Face

    cs.CL 2025-09 conditional novelty 5.0

    MultimodalHugs provides a standardized TSV-based dataset format, modular processors, and Hugging Face integration to enable reproducible sign language and multimodal translation experiments.