Pith. sign in

Paper Citation Record · LEDGER

Language Models over Canonical Byte-Pair Encodings

As of 8 August 2026, this Paper Citation Record lists 30 of 30 outbound references and 2 inbound Pith citation observations for arXiv:2506.07956.

A citation records a reference. It does not transfer a finding from one paper to another.

pith.paper-citation-record.v1
2506.07956 v1

Coverage vector

measured 30 of 30 reference resolution

Typed states for the displayed outbound observations.

Source: paper_references, paper_reference_links, observed 2026-08-07T05:36:42.077825Z

measured 32 of 32 standing notices

One-hop event checks from named stored sources.

Source: scholarly_work_events, retraction_status_cache, observed 2026-08-07T06:34:17.273281+00:00

measured 2 of 2 inbound itemization

Pith citing papers itemized under the disclosed page cap.

Source: paper_references, paper_reference_links, observed 2026-07-30T19:48:50.620495Z

measured 0 of 1 external citation measurements

A source-named dated measurement, never combined with another source.

Source: arxiv_reference, observed 2026-05-19T09:53:02.095964Z

Reference resolution

30 of 30 outbound references displayed

  • verified exact0
  • verified fuzzy9
  • unresolved20
  • parse uncertain1
  • malformed identifier0
  • metadata mismatch0

External citation measurements

No source-named external measurement is stored.

Outbound references

Observation d69b3726-fcff-4f5b-b2cc-d522723ad1e8 · outbound

This paper cites Consider the following subcases characterizing the possible positions for this merge: (a) The merge is in a(t).

Language Models over Canonical Byte-Pair Encodings Consider the following subcases characterizing the possible positions for this merge: (a) The merge is in a(t)

Reference 1

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.360469Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.006459Z digest=sha256:f41c2c6e6ef725a1a06069f6c7b0743095346e9e663fd6e986f334e33e337c82

Observation d64012d2-7a44-41fd-bda8-e5df4a12f749 · outbound

This paper cites No merges exist.

Language Models over Canonical Byte-Pair Encodings No merges exist

Reference 2

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.351778Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.009559Z digest=sha256:36973c7b2504c4a1a8c7868da44b6cf9231579c28fe049c7745ccaa97e7c168c

Observation ec56f423-7f56-42bc-a8dd-53f808724f8f · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 3

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.336289Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.015630Z digest=sha256:c027dc02342714ad0caa10e5e49bb944adba998b9b91bfc1f6705c9fc9ba0652

Observation 98db2f51-1f62-473c-9aa2-5d356a747daf · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 4

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.328165Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.018617Z digest=sha256:fabe1b894a33827de24fe01ed015a69c9612d6b623f4d1aafc8f641c7fc10736

Observation 323f9d98-d41c-40a8-8677-b7255eb8798a · outbound

This paper cites ■ 26More generally, Lemma 3 holds for any tokenization model with a bigram-based canonicality test (i.e., δ ∈ D ⇐ ⇒BIGRAMS (δ) ⊆ B.

Language Models over Canonical Byte-Pair Encodings ■ 26More generally, Lemma 3 holds for any tokenization model with a bigram-based canonicality test (i.e., δ ∈ D ⇐ ⇒BIGRAMS (δ) ⊆ B

Reference 5

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.320160Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.021466Z digest=sha256:dd0068a381c31dbe2f848cef6f82b78368c161d128302824f5b2114fde68326e

Observation 8e947fbb-3320-4d33-a1da-eff1ec089f92 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 6

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.344068Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.012691Z digest=sha256:41472e8ef0a36989103b1ffae088d29573319abf9029095ca8d9432c784bbd7c

Observation 088aea48-9230-4d60-b020-a4ef03e34465 · outbound

This paper cites • Part 2 (δ ∈ D ⇐= φ(δ) = δ):.

Language Models over Canonical Byte-Pair Encodings • Part 2 (δ ∈ D ⇐= φ(δ) = δ):

Reference 7

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.219074Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.053754Z digest=sha256:37ba3b127ffd895f268df92e0bd51ea2779b6a23ba409a84e49ba7dc0a1153a3

Observation 0baac850-30ce-4e1c-aa99-bbfe8c43f67d · outbound

This paper cites Then, there exists a conflicting pair ⟨s, s′⟩ = find_conflict(δ, δ′), i.e., ⟨s, s′⟩ satisfies π(s) > ⟨s, s′⟩ ≤π(s′).

Language Models over Canonical Byte-Pair Encodings Then, there exists a conflicting pair ⟨s, s′⟩ = find_conflict(δ, δ′), i.e., ⟨s, s′⟩ satisfies π(s) > ⟨s, s′⟩ ≤π(s′)

Reference 10

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.311754Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.024698Z digest=sha256:6944d262fc3800d2f143a8fd52a144211427c4a6a85398ab52ac11e164b1c110

Observation c7136bcd-a23c-424e-b961-e5277a59f296 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 11

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.303772Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.027157Z digest=sha256:da167d2647029583d97e1071381c75886644ce3149260a0ccfc25f6917397ed6

Observation e2e2e39e-e4d1-406d-8a35-3fb2b4c12d58 · outbound

This paper cites Part 2 (¬Φ(δ, δ′) ⇐= δ·δ′ /∈ B).

Language Models over Canonical Byte-Pair Encodings Part 2 (¬Φ(δ, δ′) ⇐= δ·δ′ /∈ B)

Reference 12

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.295966Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.029722Z digest=sha256:4be49aa655343d04340b7882854f1a715b46183e726892b004c3ea0282f1b6eb

Observation 258ec995-ea48-491e-93cc-c384e25d16e9 · outbound

This paper cites Then, φ(δ·δ′) ̸= δ·δ′, by definition.

Language Models over Canonical Byte-Pair Encodings Then, φ(δ·δ′) ̸= δ·δ′, by definition

Reference 13

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.287711Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.032706Z digest=sha256:76575e759ed8ed3145ea098ea50f1617ad36f9f3e130947b94af87f7a3a7e665

Observation aa535417-c14b-47eb-9c0d-24b52eac1535 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 14

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.279381Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.035485Z digest=sha256:0706acc36065b3f9b66f6e1dbd23e28bfff0a7bb91c2936e9cbc5455f68ce178

Observation e9616f6e-8513-4ccd-ac04-c982f822b3a2 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 15

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.271055Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.037964Z digest=sha256:6d62608fcbbc955708bad129ec6348617316638574f0fdaf6d2ffd5f02f11f37

Observation 9bc800d9-1ae8-4d35-aa6a-511e665e52ca · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 16

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.262149Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.040720Z digest=sha256:18a7b4b56ec53ea2f8df6a9b8fd0c83b755c4df1e042c1590b02049d0476e934

Observation 93890bf3-38bd-4eef-91b6-297a5ac2d1c3 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 17

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.253669Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.043120Z digest=sha256:bfab59b0dfbb426453f8fae08f853b87be588e860acb20a34bbed62116ed6c7e

Observation 0c2cce99-26e6-45a6-a7d6-2007d30076cb · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 18

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.244295Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.045576Z digest=sha256:bc97263ce04c2541a20ef62fbad86ad6b37213acac5231105f40ad306bfa3d57

Observation 424d92fc-0288-4071-88f8-3b717be65d70 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 19

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.235871Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.048373Z digest=sha256:9cd6c1157c108009e712bd631060c0a9a2b1c3846b5f89b39711c7e8927a7d40

Observation 7a47f279-bc82-4985-a80e-715adbb0c799 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 20

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.227773Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.050933Z digest=sha256:fb432e22c31c7650af4020bce24271b25930e6807ad9335aecb6a232341fb313

Observation 5ec32428-114b-4091-b776-5edec938b875 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 22

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.210679Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.056308Z digest=sha256:1c2377b9bf24c15022361d334d74c62da40c043ebc81bdd271f8fe12af660ef6

Observation 3dc8ccd7-f107-474d-a2aa-780cfc123c6f · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 23

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.202372Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.058957Z digest=sha256:936e015310140d615107a6b868ed74b12e83b22c69dd42e8f502f7ee47d031ec

Observation baad7f25-b74d-4a40-b5a9-d0f08847b95a · outbound

This paper cites Since each directions have been proven, the proposition holds.

Language Models over Canonical Byte-Pair Encodings Since each directions have been proven, the proposition holds

Reference 24

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.193337Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.061603Z digest=sha256:c1e0c3620c988f68d9b9473552a4e7734b4fa734b74e7c10b991c304264cf3b9

Observation d4859792-f7a8-4a0c-b5ff-1afdd2602f44 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 25

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.183326Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.064616Z digest=sha256:705d4ec5e45e3722eafe803ccaca8f75c37a88793f25b92047c9759f714e5a7c

Observation fc63fc23-7f0b-4cbe-bd25-ae5701047201 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 26

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.173668Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.067384Z digest=sha256:bba50fb95e9d453edc32c99f86574fe025a2f48e9bf6dba9df16b58e102b6b39

Observation cf7b10b2-15b4-47c2-9c0a-8db3fc9184ae · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 27

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.164263Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.069894Z digest=sha256:6d7072529ae34f6c5e049abd18b44d5a0417d42331587102f536edc50247d4b2

Observation ac05a37f-89a8-446b-bb27-9251c129473f · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 28

Resolution
parse uncertain
raw_fallback, observed 2026-08-07T05:36:42.154881Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.072356Z digest=sha256:552c9e2d53b5f62b8b6d3d552b337983eaffc0109624280a0f794a79f899eada

Observation 68cab206-e974-4f79-a210-e0fe98a3e233 · outbound

This paper cites an unresolved cited work.

Language Models over Canonical Byte-Pair Encodings Unresolved cited work

Reference 29

Resolution
unresolved
raw_fallback, observed 2026-08-07T05:36:42.145479Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.075100Z digest=sha256:4a40fd26abd0c77a77a48e282dc4d4f1fc663bec5966985f974fc7c3cd3e6708

Observation 4a1a0b35-f6bb-4789-98d8-b937e57e79bd · outbound

This paper cites Proofs for Section 4 (Canonicality by Conditioning) Proposition 1.

Language Models over Canonical Byte-Pair Encodings Proofs for Section 4 (Canonicality by Conditioning) Proposition 1

Reference 30

Resolution
verified fuzzy
raw_fallback, observed 2026-08-07T05:36:42.135960Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-08-07T05:36:42.077825Z digest=sha256:066bdb45e3f1121f071cab2a9c3a8e077832b2fd3a2f51558c0274f449960c0b

Observation 45315309-3fcd-48f2-a71c-3cc99735163f · outbound

This paper cites GPT-4 Technical Report.

Language Models over Canonical Byte-Pair Encodings GPT-4 Technical Report

Reference 1993

Resolution
unresolved
no resolver link, observed 2026-08-07T05:36:42.002656Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

source=pdf_text observed=2026-08-07T05:36:42.002656Z digest=sha256:09b0613c075fe82feca83745d58bd1ca4a6d123383242f9de917f8d241a340ef

Observation ffd0c972-4fdf-4d97-a0be-31cabbadaf43 · outbound

This paper cites GPT-NeoX-20B: An Open-Source Autoregressive Language Model.

Language Models over Canonical Byte-Pair Encodings GPT-NeoX-20B: An Open-Source Autoregressive Language Model

Reference 2023

Resolution
unresolved
no resolver link, observed 2026-08-07T05:36:41.994894Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

source=pdf_text observed=2026-08-07T05:36:41.994894Z digest=sha256:69e5301b67f3a31252832e26660cd117cf42fe0b36af3733c81512ad669909cd

Observation 26f880b2-5f88-47c1-a8d9-602f65fbcc2a · outbound

This paper cites The Foundations of Tokenization: Statistical and Computational Concerns.

Language Models over Canonical Byte-Pair Encodings The Foundations of Tokenization: Statistical and Computational Concerns

Reference 2025

Resolution
unresolved
no resolver link, observed 2026-08-07T05:36:41.998910Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

source=pdf_text observed=2026-08-07T05:36:41.998910Z digest=sha256:ded8845a7807a0e0adbf9ac1385d5486ced531b9021e18b428f7283c1dc57111

Pith citing papers

Observation aaa57ec2-8b24-4ed7-b9e6-5b44db1a3168 · inbound

Sampling from Your Language Model One Byte at a Time cites this paper.

Sampling from Your Language Model One Byte at a Time Language Models over Canonical Byte-Pair Encodings

Reference 76

Resolution
verified exact
arxiv_id, observed 2026-05-19T09:53:02.097339Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.

source=pdf_text observed=2026-05-19T09:52:20.923710Z digest=sha256:ef84ad449dcc3db68e4c0828f1522e8451b2260988d61d2c22df6f75ecbaed71

Observation 8e524b21-f2da-4c30-af03-30efb6174699 · inbound

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages cites this paper.

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages Language Models over Canonical Byte-Pair Encodings

Reference 19

Resolution
unresolved
no resolver link, observed 2026-07-30T19:48:50.620495Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

source=arxiv_source observed=2026-07-30T19:48:50.620495Z digest=sha256:c303122f00d123bcf838f43d8cb0f33edcc02750140d85002484ba01fcad0baa