Paper Citation Record · LEDGER

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

As of 8 August 2026, this Paper Citation Record lists 19 of 19 outbound references and 4 inbound Pith citation observations for arXiv:2505.24689.

A citation records a reference. It does not transfer a finding from one paper to another.

pith.paper-citation-record.v1
2505.24689 v1

Coverage vector

measured 19 of 19 reference resolution

Typed states for the displayed outbound observations.

Source: paper_references, paper_reference_links, observed 2026-08-07T12:35:32.474459Z

measured 23 of 23 standing notices

One-hop event checks from named stored sources.

Source: scholarly_work_events, retraction_status_cache, observed 2026-08-07T06:34:17.273281+00:00

measured 4 of 4 inbound itemization

Pith citing papers itemized under the disclosed page cap.

Source: paper_references, paper_reference_links, observed 2026-06-26T04:59:05.057552Z

measured 0 of 1 external citation measurements

A source-named dated measurement, never combined with another source.

Source: arxiv_reference, observed 2026-07-04T13:39:51.284729Z

Reference resolution

19 of 19 outbound references displayed

verified exact1
verified fuzzy7
unresolved11
parse uncertain0
malformed identifier0
metadata mismatch0

External citation measurements

No source-named external measurement is stored.

Outbound references

Observation 5be79990-594a-4478-8015-eac45e8ea966 · outbound

This paper cites wHy DoNt YoU jUsT uSe ThE lLaMa ToKeNiZeR?? Blog Post, 2024.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization wHy DoNt YoU jUsT uSe ThE lLaMa ToKeNiZeR?? Blog Post, 2024

Reference 1

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T12:35:34.054036Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation f773043d-85f6-4efb-99b4-ccc543d3f8e9 · outbound

This paper cites A., and Bergen, B.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization A., and Bergen, B

Reference 2

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T12:35:33.913134Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation 5565fcc0-4489-454f-a936-9bacc3db0cd8 · outbound

This paper cites Goldfish: Monolingual Language Models for 350 Languages.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization Goldfish: Monolingual Language Models for 350 Languages

Reference 3

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:31.015765Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation b38b5f1a-a44d-4e1d-8741-cdcee7495e3c · outbound

This paper cites Getting the most out of your tokenizer for pre-training and domain adaptation.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization Getting the most out of your tokenizer for pre-training and domain adaptation

Reference 4

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T12:35:33.746255Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation 347ba6a1-3b47-4d95-9413-fdd8b9dbe24b · outbound

This paper cites and Richardson, J.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization and Richardson, J

Reference 5

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:31.176037Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ffdf01f4-2f88-4f96-812a-2f9fc00b3dfc · outbound

This paper cites Pre-tokenization on punctuation in GPT-4.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization Pre-tokenization on punctuation in GPT-4

Reference 6

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T12:35:33.549888Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation 695861cb-3e5c-40d0-be88-d0be71ce415f · outbound

This paper cites and Bartolo, M.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization and Bartolo, M

Reference 7

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:31.346827Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 86811bd3-5338-4fc9-8c91-85a671863659 · outbound

This paper cites Jamo-level subword tokenization in low-resource K orean machine translation.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization Jamo-level subword tokenization in low-resource K orean machine translation

Reference 8

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T12:35:33.424934Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation 04a8363d-9535-4fe4-b6db-7d737cc6d8b8 · outbound

This paper cites MYTE : Morphology-driven byte encoding for better and fairer multilingual language modeling.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization MYTE : Morphology-driven byte encoding for better and fairer multilingual language modeling

Reference 9

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:31.489268Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 6acc0685-ed55-4527-9244-d5d224f8072a · outbound

This paper cites SuperBPE: Space Travel for Language Models.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization SuperBPE: Space Travel for Language Models

Reference 10

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:31.585952Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 35cf48fb-1851-44aa-8ba6-f35f1941364b · outbound

This paper cites Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks

Reference 11

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:31.632279Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 8503dc19-4009-46da-8a76-2c49283237d5 · outbound

This paper cites CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages

Reference 12

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:31.714800Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation a325186d-7afe-4afa-b405-9a59de9cc8b8 · outbound

This paper cites tiktoken: a fast BPE tokeniser for use with OpenAI 's models., 2024.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization tiktoken: a fast BPE tokeniser for use with OpenAI 's models., 2024

Reference 13

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T12:35:33.296193Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation 604f266a-a7ed-4c85-bebb-055b730458f7 · outbound

This paper cites W., Reddy, V., Zhang, H., Alameddine, A., Uzan, O., Pinter, Y., and Tanner, C.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization W., Reddy, V., Zhang, H., Alameddine, A., Uzan, O., Pinter, Y., and Tanner, C

Reference 14

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:31.950493Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation eb6e5d96-f3b0-4391-9da3-508daa9aea51 · outbound

This paper cites W., Reddy, V., Tanner, C., and Pinter, Y.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization W., Reddy, V., Tanner, C., and Pinter, Y

Reference 15

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:32.081870Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 3579a6d8-e900-442e-a668-957d4445eaa4 · outbound

This paper cites Neural Machine Translation of Rare Words with Subword Units.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization Neural Machine Translation of Rare Words with Subword Units

Reference 16

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:32.199547Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 9cf2b8b7-b08a-427a-b738-aa698ae8fc7f · outbound

This paper cites Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

Reference 17

Resolution: unresolved; no resolver link, observed 2026-08-07T12:35:32.293186Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation faeebed8-e5c5-43f3-b969-cfe30ae3249d · outbound

This paper cites and Sarveswaran, K.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization and Sarveswaran, K

Reference 18

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T12:35:33.100922Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation e497e40d-f5e0-4bc2-891a-9938e1d9faf5 · outbound

This paper cites Tokenization is Sensitive to Language Variation.

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization Tokenization is Sensitive to Language Variation

Reference 19

Resolution: verified exact; local_arxiv, observed 2026-08-07T12:35:32.734006Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Pith citing papers

Observation b68c0b5c-de20-4d1d-bbf6-9cd03a316395 · inbound

The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices? cites this paper.

The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices? BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

Reference 6

Resolution: verified exact; arxiv_id, observed 2026-05-16T15:01:03.843494Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation b68a1932-1dc9-4a72-ad55-2add4357d734 · inbound

Faster Superword Tokenization cites this paper.

Faster Superword Tokenization BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

Reference 8

Resolution: metadata mismatch; arxiv_id, observed 2026-05-10T23:50:52.391000Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation 618faa9c-7d08-47b9-95d7-9ac0d9b94422 · inbound

Tokenization with Split Trees cites this paper.

Tokenization with Split Trees BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

Reference 55

Resolution: verified exact; arxiv_id, observed 2026-05-22T05:51:08.813261Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

Observation f703090f-7e4b-4f8c-8fda-69f47db258d3 · inbound

MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment cites this paper.

MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

Reference 18

Resolution: metadata mismatch; arxiv_id, observed 2026-07-04T13:39:51.286390Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.