Pith. sign in

REVIEW 4 major objections 4 minor 14 references

Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Current vision-language and text models cannot decipher rare scripts that lack Unicode encoding, and Unicode representation improves performance only for common languages.

desk verdict Small but genuinely novel probe of LVLMs on non-Unicode script puzzles; the tokenization is subjective and evidence is thin, but the core difficulty finding is plausible and the paper deserves a chance to grow through review. read the letter →

arxiv 2501.17785 v1 pith:3JFE4PHP submitted 2025-01-29 cs.CL cs.LG

classification cs.CLcs.LG
keywords rarescriptsdeciphermentglyphtokenizationvision-languagemodelslinguisticpuzzleslow-resourcelanguagesUnicodeencodingvisualreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether current vision-language and text-only models can decipher scripts that cannot be typed in Unicode, and answers with a mostly negative result. It builds a small set of linguistic puzzles from olympiad materials, replaces each glyph with a placeholder token, and then feeds the puzzles to GPT-4o, Gemini, and Claude 3.5 Sonnet in three forms: images of the glyphs, written descriptions of the glyphs, or placeholders alone. Across the non-Unicode puzzles, the models rarely reconstruct the scripts' sound systems and frequently misdescribe or misread the geometry of the glyphs. On the encoded side, Unicode helps the models use memorized knowledge for a common language (Malayalam) but does not rescue a low-resource language (Meroitic). The upshot is that current models can do pattern matching but not true decipherment, and that placeholders offer a way to test reasoning without pretraining shortcuts.

What carries the argument

The central object is the glyph token, defined as the fundamental unit of visual information in an unknown script; two adjacent glyphs are split into separate tokens when a vertical white line can pass through their gap, with a subjective exception for horizontal extensions bridging the gap. After segmentation, each glyph token is replaced by a placeholder such as <token_i>, so reasoning can be tested without exposing the model to real characters. That machinery supports two input methods: the Picture Method, which shows an image of all glyphs labeled with placeholders to a vision-language model, and the Description Method, which supplies a table of verbal descriptions of each glyph's appearance and relations to a text-only model.

What would settle it

Give the same five puzzles to expert annotators to segment glyphs according to the orthographies and rerun the models on those versions. If expert-segmented puzzles yield substantially different model accuracies, or if the vertical-white-line rule is shown to split single letters or merge distinct ones in Meroitic or Mandombe, the central performance claims are artifacts of the tokenization rather than a measure of model capability.

Watch

Extended reading notes

Core claim

The paper's central claim is that contemporary LVLMs and LLMs cannot reliably decipher rare scripts that lack Unicode encoding, and that their failures follow a regular pattern: token descriptions are imprecise or geometrically wrong, direction is confused, and reasoning from images often misfires. For glyphs made of indivisible tokens (Avoiuli), the models can sometimes start down the right path by counting and matching tokens, but they cannot complete a syllable-token mapping. For glyphs whose subcomponents carry phonological information (Mandombe, Ditema tsa Dinoko), no model performs phonological analysis and progress stalls. The paper further claims that Unicode encoding is not a cure: it unlocks pre-trained knowledge for a common language but, for Meroitic, even correct Unicode characters and directionality leave performance poor because the models lack the underlying language data.

Load-bearing premise

The load-bearing premise is that the glyph-token segmentation rule, which treats any gap a vertical white line can pass through as a token boundary and applies a subjective exception for horizontal extensions, produces the correct and consistent units for all five puzzles; if the segmentation does not match the scripts' real orthographic units, the measured accuracy says nothing about deciphering the scripts as written.

Editorial extensions

If this is right

  • Token-description accuracy is a bottleneck: if descriptions cannot reliably distinguish glyphs, then the Description Method cannot support decipherment even when the underlying language rules are learnable.
  • Unicode encoding is a proxy for pretraining exposure, not a substitute for it; adding Unicode to low-resource scripts does not make models reason better.
  • Placeholder-based puzzles can serve as a diagnostic that separates genuine linguistic reasoning from retrieval of memorized script knowledge.
  • For scripts with compositional glyphs, models need phonological and structural analysis of subcomponents; current vision-language models instead fixate on surface geometry.
  • Mirrored glyphs and other non-orthographic visual conventions remain undetected by models, so future datasets or prompts must either make these conventions explicit or test for their discovery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to feed Unicode-encoded common languages through the same placeholder pipeline; if performance collapses, it would confirm that placeholders remove retrieval without requiring visual decipherment.
  • The 35-word description limit may itself cause failures; models might succeed with structured or relational description formats such as coordinates or component graphs, so the paper's negative result may overstate the ceiling of description-based decipherment.
  • Connecting the glyph-description task to existing spatial-reasoning benchmarks would isolate whether the failure is about language, vision, or the mapping between the two.
  • If the segmentation rule is applied at scale, it could become a curriculum for testing whether a model detects visual token boundaries, since the same image should yield stable splits under expert annotation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a visual glyph-tokenization method for representing scripts that are not Unicode-encodable, then evaluates GPT-4o, Gemini, and Claude 3.5 Sonnet on five linguistic puzzles drawn from UKLO and NACLO. Two prompting strategies are introduced: the Picture Method for vision-language models and the Description Method for text-only models. Experiments compare token-description generation, puzzle-solving under placeholders/images/descriptions, and the effect of Unicode encoding for a common language (Malayalam) versus a low-resource language (Meroitic). The main reported findings are that current models struggle with non-Unicode scripts, especially in token description and geometric reasoning, and that Unicode encoding helps only for common languages.

Significance. If the method and measurements are reliable, the paper addresses a genuine gap by constructing multimodal puzzles for scripts outside Unicode coverage and by proposing a tokenization scheme that lets both LVLMs and LLMs work with non-encodable scripts. The use of externally sourced UKLO and NACLO puzzles is a strength, as is the explicit acknowledgment of limited scope in Section 5.4. The qualitative observations about model behavior and the demonstration that Unicode support alone does not help low-resource languages are useful for the community. However, the significance is currently constrained by the small number of puzzles, the single-run nature of the experiments, and the lack of validation for the central tokenization step.

major comments (4)
  1. [Section 3] The glyph-tokenization rule is not validated against the scripts' actual orthographic units, and the exception for horizontal extensions is explicitly subjective ('The criteria for identifying and applying this exception may vary among individuals'). This rule is the foundation of both the Picture Method and the Description Method, since all placeholders are derived from it. If the segmentation does not match real grapheme or syllable boundaries, then the reported accuracies, including the description-pairing figures in Section 5.1 and the 'below 15%' reverse-mapping rates in Section 5.2, do not measure decipherment of the scripts as written. The authors should provide inter-annotator agreement, linguistic validation of the token boundaries, or an ablation showing that results are robust across reasonable alternative segmentations.
  2. [Section 5.2] For Avoiuli, the paper states that mirrored glyphs should be considered the same token, yet it explicitly treats them as distinct tokens during tokenization and does not test whether the models can recover the equivalence. Since the paper lists Avoiuli as a case where token identification is the primary challenge, this mismatch means the evaluation does not fully reflect the linguistic property that the authors themselves identify. The reported failure to infer mirror equivalence is therefore not a fair test of the models' decipherment ability. Please either align the tokenization with the stated orthographic property or present mirror handling as a separate controlled experiment.
  3. [Section 5.1 and 5.2] The quantitative evidence is reported inconsistently and at too coarse a grain. The description-pairing accuracies (40.0%, 13.4%, 31.3%) are given without the number of tokens, the number of trials, or any variance measure, and the reverse-mapping success for Mandombe is only described as 'below 15%' without exact values or per-model breakdowns. With only five puzzles and single runs, these numbers cannot be compared across models or across puzzles. The authors should report exact per-puzzle, per-model results with counts and, ideally, multiple runs, and should state which numbers are averages over which items.
  4. [Section 5.4] The paper's own limitation statement acknowledges that the same prompt was used for every model and that only five puzzles were tested, yet the abstract and conclusion make general claims about LVLM and LLM capabilities in deciphering rare scripts. This mismatch between scope and generalization is load-bearing for the paper's framing. A discussion of how the fixed prompt and small puzzle set might bias the results, together with a restriction of the central claims to the five tested puzzles, would make the conclusions more defensible.
minor comments (4)
  1. [Section 3, Figure 1] Figure 1 is not described in enough detail in the text; readers cannot see the exact token boundaries or the horizontal-extension exception being applied. Adding an annotated example with boxes or arrows around the relevant gaps would make the tokenization rule reproducible.
  2. [Section 4] The term 'preliminary experiments' is used in Section 4, but this hedged framing is not carried through the abstract and conclusion, which present the findings as established strengths and limitations. The wording should be made consistent throughout.
  3. [Appendix 7.1] Appendix 7.1 references Figure 3 and Table description examples, but the appendix text does not include the actual description table or the full set of model-generated descriptions. Including the complete table would let readers judge the qualitative claims about 'Lack of Details', 'Direction Confusion', and 'Misleading Elaboration'.
  4. [Section 5.1] The models are referred to inconsistently as GPT-4o, Gemini, and Claude 3.5 Sonnet, while Figure 2 and the text later use 'Claude-3.5'; please standardize the notation. Also, the paper does not provide the specific prompt templates, model version numbers, or decoding parameters, which are needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical results against external UKLO/NACLO puzzles; no fitted input is renamed as prediction and the sole self-citation is not load-bearing.

full rationale

The paper's central claims are empirical measurements of LVLM and LLM performance on existing UKLO and NACLO puzzles, rather than results derived from its own fitted parameters or from its own definitions. The visual tokenization rule in Section 3 is an input assumption used to construct placeholder-based tasks; the paper does not claim to derive that rule from the model outputs, nor does it fit a parameter to a subset of data and then relabel that fit as a prediction. The performance figures, such as the below-15% reverse-mapping accuracy for Mandombe and the 40.0%/13.4%/31.3% description-pairing scores, are measurements on externally sourced puzzles, so they do not reduce to the construction by definition. The only self-citation, Lin et al. (2023), appears in the related-work survey and is not used to justify any load-bearing premise or to forbid alternative methods. The Avoiuli mirrored-glyph discussion is a ground-truth statement about the script, and the paper explicitly notes that it treated mirrored glyphs as distinct tokens during tokenization, so there is no circular step in which the tokenization is assumed to equal the correct answer. The Section 3 caveat that the horizontal-extension exception 'may vary among individuals' is a reproducibility and validity limitation, not evidence that a result is equivalent to its input. The Section 5.4 limitation that only five puzzles were analyzed is an honest scope statement. No equation in the paper is shown to equal another by construction, and no fitted quantity is renamed as an outcome. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central evaluation rests on the segmentation rule, the representativeness of the five selected puzzles, and the assumption that placeholder inputs isolate reasoning from pre-trained knowledge; these are domain assumptions stated or implied by the paper rather than independently validated facts. No fitted numerical parameters are used; the 35-word description limit is a minor hand-set detail that does not carry the central claims.

assumptions (4)
  • domain assumption Glyphs can be reliably segmented into tokens by the vertical-white-line test described in Section 3.
    All placeholder-based tests depend on this segmentation being correct and consistent; the paper itself notes the exception is subjective and may vary among individuals, so it is an unverified assumption.
  • domain assumption Five puzzles from UKLO and NACLO are sufficient to characterize LVLM behavior on non-Unicode script decipherment.
    Section 5.4 explicitly acknowledges this limitation, yet the conclusions are framed broadly, so the analysis relies on the representativeness of this small sample.
  • domain assumption Replacing Unicode text with placeholders removes pre-trained knowledge and isolates true reasoning ability.
    Section 5.3 interprets performance drops as evidence that models rely on pre-trained knowledge; this assumes placeholders do not introduce unrelated difficulties such as loss of orthographic cues.
  • domain assumption Text descriptions of tokens can capture enough visual structure for an LLM to solve the puzzles.
    The Description Method in Section 3 assumes visual tokens can be represented adequately in words; the paper's own experiments show that model-generated descriptions are often poor, which makes this assumption load-bearing for one of the two proposed methods.
invented entities (1)
  • Glyph token
    purpose: A unit of visual information representing a segment of a non-Unicode glyph, replaced by <token_i> placeholders for model input.
    Introduced in Section 3 as the basis for the dataset; there is no external validation that these tokens match the scripts' true orthographic units, and the segmentation rule is acknowledged to be subjective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts." pith.science (2026). https://pith.science/paper/3JFE4PHP

@misc{pith2026250117785,
  author       = {Pith},
  title        = {Pith review of: Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3JFE4PHP}},
  note         = {Machine review of arXiv:2501.17785}
}
read the original abstract

We explore the capabilities of LVLMs and LLMs in deciphering rare scripts not encoded in Unicode. We introduce a novel approach to construct a multimodal dataset of linguistic puzzles involving such scripts, utilizing a tokenization method for language glyphs. Our methods include the Picture Method for LVLMs and the Description Method for LLMs, enabling these models to tackle these challenges. We conduct experiments using prominent models, GPT-4o, Gemini, and Claude 3.5 Sonnet, on linguistic puzzles. Our findings reveal the strengths and limitations of current AI methods in linguistic decipherment, highlighting the impact of Unicode encoding on model performance and the challenges of modeling visual language tokens through descriptions. Our study advances understanding of AI's potential in linguistic decipherment and underscores the need for further research.

Figures

Figures reproduced from arXiv: 2501.17785 by the authors.

Figure 1
Figure 1. Example of Tokenization of Glyphs addresses aspects that previous benchmarks have overlooked. Our approach enables both LVLMs and LLMs to tackle these challenges effectively. To enable models to understand scripts that are not Unicode-encodable, we introduce the concept of a glyph token (The tokens mentioned below all refer to glyph tokens.), representing the fundamen￾tal unit of visual information extracted from an… view at source ↗
Figure 2
Figure 2. Incorrect geometric interpretations example. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example for Token’s Description on Meroitic [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

  4. [4]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet . Accessed: 2024-09-17

  5. [5]

    Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A

    Andrew M. Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A. Chi, Ryan Chi, Scott A. Hale, and Hannah Rose Kirk. 2024. https://arxiv.org/abs/2406.06196 Lingoly: A benchmark of olympiad-level linguistic reasoning puzzles in low-resource and extinct languages . Preprint, arXiv:2406.06196

  6. [6]

    Bozhidar Bozhanov and Ivan Derzhanski. 2013. https://aclanthology.org/W13-3401 Rosetta stone linguistic problems . In Proceedings of the Fourth Workshop on Teaching NLP and CL , pages 1--8, Sofia, Bulgaria. Association for Computational Linguistics

  7. [7]

    McCoy, and Dragomir Radev

    Nathan Chi, Teodor Malchev, Riley Kong, Ryan Chi, Lucas Huang, Ethan Chi, R. McCoy, and Dragomir Radev. 2024. https://aclanthology.org/2024.sigtyp-1.14 M ode L ing: A novel dataset for testing linguistic reasoning in language models . In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 113--119, ...

  8. [8]

    Haisu Guan, Huanxin Yang, Xinyu Wang, Shengwei Han, Yongge Liu, Lianwen Jin, Xiang Bai, and Yuliang Liu. 2024. https://arxiv.org/abs/2406.00684 Deciphering oracle bone language with diffusion models . Preprint, arXiv:2406.00684

Show all 14 references
  1. [9]

    Zheng-Lin Lin, Chiao-Han Yen, Jia-Cheng Xu, Deborah Watty, and Shu-Kai Hsieh. 2023. https://aclanthology.org/2023.rocling-1.33 Solving linguistic olympiad problems with tree-of-thought prompting . In Proceedings of the 35th Conference on Computational Linguistics and Speech Pr...

  2. [10]

    Jiaming Luo, Yuan Cao, and Regina Barzilay. 2019. https://arxiv.org/abs/1906.06718 Neural decipherment via minimum-cost flow: from ugaritic to linear b . Preprint, arXiv:1906.06718

  3. [11]

    Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, et al

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, et al. 2024. https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal mo...

  4. [12]

    Saujas Vaduguru, Aalok Sathe, Monojit Choudhury, and Dipti Sharma. 2021. https://doi.org/10.18653/v1/2021.sigmorphon-1.7 Sample-efficient linguistic generalizations through program synthesis: Experiments with phonology problems . In Proceedings of the 18th SIGMORPHON Workshop ...

  5. [13]

    Pengjie Wang, Kaile Zhang, Xinyu Wang, Shengwei Han, Yongge Liu, Jinpeng Wan, Haisu Guan, Zhebin Kuang, Lianwen Jin, Xiang Bai, and Yuliang Liu. 2024. https://arxiv.org/abs/2401.15365 An open dataset for oracle bone script recognition and decipherment . Preprint, arXiv:2401.15365

  6. [14]

    Hengyuan Zhang, Peng Chang, and Zongcheng Ji. 2023. https://arxiv.org/abs/2308.11891 Bridging the gap: Deciphering tabular data using large language model . Preprint, arXiv:2308.11891

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.