REVIEW 4 major objections 4 minor 14 references
Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Current vision-language and text models cannot decipher rare scripts that lack Unicode encoding, and Unicode representation improves performance only for common languages.
desk verdict Small but genuinely novel probe of LVLMs on non-Unicode script puzzles; the tokenization is subjective and evidence is thin, but the core difficulty finding is plausible and the paper deserves a chance to grow through review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the glyph token, defined as the fundamental unit of visual information in an unknown script; two adjacent glyphs are split into separate tokens when a vertical white line can pass through their gap, with a subjective exception for horizontal extensions bridging the gap. After segmentation, each glyph token is replaced by a placeholder such as <token_i>, so reasoning can be tested without exposing the model to real characters. That machinery supports two input methods: the Picture Method, which shows an image of all glyphs labeled with placeholders to a vision-language model, and the Description Method, which supplies a table of verbal descriptions of each glyph's appearance and relations to a text-only model.
What would settle it
Give the same five puzzles to expert annotators to segment glyphs according to the orthographies and rerun the models on those versions. If expert-segmented puzzles yield substantially different model accuracies, or if the vertical-white-line rule is shown to split single letters or merge distinct ones in Meroitic or Mandombe, the central performance claims are artifacts of the tokenization rather than a measure of model capability.
Extended reading notes
Core claim
The paper's central claim is that contemporary LVLMs and LLMs cannot reliably decipher rare scripts that lack Unicode encoding, and that their failures follow a regular pattern: token descriptions are imprecise or geometrically wrong, direction is confused, and reasoning from images often misfires. For glyphs made of indivisible tokens (Avoiuli), the models can sometimes start down the right path by counting and matching tokens, but they cannot complete a syllable-token mapping. For glyphs whose subcomponents carry phonological information (Mandombe, Ditema tsa Dinoko), no model performs phonological analysis and progress stalls. The paper further claims that Unicode encoding is not a cure: it unlocks pre-trained knowledge for a common language but, for Meroitic, even correct Unicode characters and directionality leave performance poor because the models lack the underlying language data.
Load-bearing premise
The load-bearing premise is that the glyph-token segmentation rule, which treats any gap a vertical white line can pass through as a token boundary and applies a subjective exception for horizontal extensions, produces the correct and consistent units for all five puzzles; if the segmentation does not match the scripts' real orthographic units, the measured accuracy says nothing about deciphering the scripts as written.
Editorial extensions
If this is right
- Token-description accuracy is a bottleneck: if descriptions cannot reliably distinguish glyphs, then the Description Method cannot support decipherment even when the underlying language rules are learnable.
- Unicode encoding is a proxy for pretraining exposure, not a substitute for it; adding Unicode to low-resource scripts does not make models reason better.
- Placeholder-based puzzles can serve as a diagnostic that separates genuine linguistic reasoning from retrieval of memorized script knowledge.
- For scripts with compositional glyphs, models need phonological and structural analysis of subcomponents; current vision-language models instead fixate on surface geometry.
- Mirrored glyphs and other non-orthographic visual conventions remain undetected by models, so future datasets or prompts must either make these conventions explicit or test for their discovery.
Reading between the lines
- A natural next experiment is to feed Unicode-encoded common languages through the same placeholder pipeline; if performance collapses, it would confirm that placeholders remove retrieval without requiring visual decipherment.
- The 35-word description limit may itself cause failures; models might succeed with structured or relational description formats such as coordinates or component graphs, so the paper's negative result may overstate the ceiling of description-based decipherment.
- Connecting the glyph-description task to existing spatial-reasoning benchmarks would isolate whether the failure is about language, vision, or the mapping between the two.
- If the segmentation rule is applied at scale, it could become a curriculum for testing whether a model detects visual token boundaries, since the same image should yield stable splits under expert annotation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a visual glyph-tokenization method for representing scripts that are not Unicode-encodable, then evaluates GPT-4o, Gemini, and Claude 3.5 Sonnet on five linguistic puzzles drawn from UKLO and NACLO. Two prompting strategies are introduced: the Picture Method for vision-language models and the Description Method for text-only models. Experiments compare token-description generation, puzzle-solving under placeholders/images/descriptions, and the effect of Unicode encoding for a common language (Malayalam) versus a low-resource language (Meroitic). The main reported findings are that current models struggle with non-Unicode scripts, especially in token description and geometric reasoning, and that Unicode encoding helps only for common languages.
Significance. If the method and measurements are reliable, the paper addresses a genuine gap by constructing multimodal puzzles for scripts outside Unicode coverage and by proposing a tokenization scheme that lets both LVLMs and LLMs work with non-encodable scripts. The use of externally sourced UKLO and NACLO puzzles is a strength, as is the explicit acknowledgment of limited scope in Section 5.4. The qualitative observations about model behavior and the demonstration that Unicode support alone does not help low-resource languages are useful for the community. However, the significance is currently constrained by the small number of puzzles, the single-run nature of the experiments, and the lack of validation for the central tokenization step.
major comments (4)
- [Section 3] The glyph-tokenization rule is not validated against the scripts' actual orthographic units, and the exception for horizontal extensions is explicitly subjective ('The criteria for identifying and applying this exception may vary among individuals'). This rule is the foundation of both the Picture Method and the Description Method, since all placeholders are derived from it. If the segmentation does not match real grapheme or syllable boundaries, then the reported accuracies, including the description-pairing figures in Section 5.1 and the 'below 15%' reverse-mapping rates in Section 5.2, do not measure decipherment of the scripts as written. The authors should provide inter-annotator agreement, linguistic validation of the token boundaries, or an ablation showing that results are robust across reasonable alternative segmentations.
- [Section 5.2] For Avoiuli, the paper states that mirrored glyphs should be considered the same token, yet it explicitly treats them as distinct tokens during tokenization and does not test whether the models can recover the equivalence. Since the paper lists Avoiuli as a case where token identification is the primary challenge, this mismatch means the evaluation does not fully reflect the linguistic property that the authors themselves identify. The reported failure to infer mirror equivalence is therefore not a fair test of the models' decipherment ability. Please either align the tokenization with the stated orthographic property or present mirror handling as a separate controlled experiment.
- [Section 5.1 and 5.2] The quantitative evidence is reported inconsistently and at too coarse a grain. The description-pairing accuracies (40.0%, 13.4%, 31.3%) are given without the number of tokens, the number of trials, or any variance measure, and the reverse-mapping success for Mandombe is only described as 'below 15%' without exact values or per-model breakdowns. With only five puzzles and single runs, these numbers cannot be compared across models or across puzzles. The authors should report exact per-puzzle, per-model results with counts and, ideally, multiple runs, and should state which numbers are averages over which items.
- [Section 5.4] The paper's own limitation statement acknowledges that the same prompt was used for every model and that only five puzzles were tested, yet the abstract and conclusion make general claims about LVLM and LLM capabilities in deciphering rare scripts. This mismatch between scope and generalization is load-bearing for the paper's framing. A discussion of how the fixed prompt and small puzzle set might bias the results, together with a restriction of the central claims to the five tested puzzles, would make the conclusions more defensible.
minor comments (4)
- [Section 3, Figure 1] Figure 1 is not described in enough detail in the text; readers cannot see the exact token boundaries or the horizontal-extension exception being applied. Adding an annotated example with boxes or arrows around the relevant gaps would make the tokenization rule reproducible.
- [Section 4] The term 'preliminary experiments' is used in Section 4, but this hedged framing is not carried through the abstract and conclusion, which present the findings as established strengths and limitations. The wording should be made consistent throughout.
- [Appendix 7.1] Appendix 7.1 references Figure 3 and Table description examples, but the appendix text does not include the actual description table or the full set of model-generated descriptions. Including the complete table would let readers judge the qualitative claims about 'Lack of Details', 'Direction Confusion', and 'Misleading Elaboration'.
- [Section 5.1] The models are referred to inconsistently as GPT-4o, Gemini, and Claude 3.5 Sonnet, while Figure 2 and the text later use 'Claude-3.5'; please standardize the notation. Also, the paper does not provide the specific prompt templates, model version numbers, or decoding parameters, which are needed for reproducibility.
Circularity Check
No significant circularity: empirical results against external UKLO/NACLO puzzles; no fitted input is renamed as prediction and the sole self-citation is not load-bearing.
full rationale
The paper's central claims are empirical measurements of LVLM and LLM performance on existing UKLO and NACLO puzzles, rather than results derived from its own fitted parameters or from its own definitions. The visual tokenization rule in Section 3 is an input assumption used to construct placeholder-based tasks; the paper does not claim to derive that rule from the model outputs, nor does it fit a parameter to a subset of data and then relabel that fit as a prediction. The performance figures, such as the below-15% reverse-mapping accuracy for Mandombe and the 40.0%/13.4%/31.3% description-pairing scores, are measurements on externally sourced puzzles, so they do not reduce to the construction by definition. The only self-citation, Lin et al. (2023), appears in the related-work survey and is not used to justify any load-bearing premise or to forbid alternative methods. The Avoiuli mirrored-glyph discussion is a ground-truth statement about the script, and the paper explicitly notes that it treated mirrored glyphs as distinct tokens during tokenization, so there is no circular step in which the tokenization is assumed to equal the correct answer. The Section 3 caveat that the horizontal-extension exception 'may vary among individuals' is a reproducibility and validity limitation, not evidence that a result is equivalent to its input. The Section 5.4 limitation that only five puzzles were analyzed is an honest scope statement. No equation in the paper is shown to equal another by construction, and no fitted quantity is renamed as an outcome. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Glyphs can be reliably segmented into tokens by the vertical-white-line test described in Section 3.
- domain assumption Five puzzles from UKLO and NACLO are sufficient to characterize LVLM behavior on non-Unicode script decipherment.
- domain assumption Replacing Unicode text with placeholders removes pre-trained knowledge and isolates true reasoning ability.
- domain assumption Text descriptions of tokens can capture enough visual structure for an LLM to solve the puzzles.
invented entities (1)
-
Glyph token
Cite this review
Pith. "Pith review of Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts." pith.science (2026). https://pith.science/paper/3JFE4PHP
@misc{pith2026250117785,
author = {Pith},
title = {Pith review of: Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts},
year = {2026},
howpublished = {\url{https://pith.science/paper/3JFE4PHP}},
note = {Machine review of arXiv:2501.17785}
}
read the original abstract
We explore the capabilities of LVLMs and LLMs in deciphering rare scripts not encoded in Unicode. We introduce a novel approach to construct a multimodal dataset of linguistic puzzles involving such scripts, utilizing a tokenization method for language glyphs. Our methods include the Picture Method for LVLMs and the Description Method for LLMs, enabling these models to tackle these challenges. We conduct experiments using prominent models, GPT-4o, Gemini, and Claude 3.5 Sonnet, on linguistic puzzles. Our findings reveal the strengths and limitations of current AI methods in linguistic decipherment, highlighting the impact of Unicode encoding on model performance and the challenges of modeling visual language tokens through descriptions. Our study advances understanding of AI's potential in linguistic decipherment and underscores the need for further research.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[4]
Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet . Accessed: 2024-09-17
work page 2024
-
[5]
Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A
Andrew M. Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A. Chi, Ryan Chi, Scott A. Hale, and Hannah Rose Kirk. 2024. https://arxiv.org/abs/2406.06196 Lingoly: A benchmark of olympiad-level linguistic reasoning puzzles in low-resource and extinct languages . Preprint, arXiv:2406.06196
arXiv 2024
-
[6]
Bozhidar Bozhanov and Ivan Derzhanski. 2013. https://aclanthology.org/W13-3401 Rosetta stone linguistic problems . In Proceedings of the Fourth Workshop on Teaching NLP and CL , pages 1--8, Sofia, Bulgaria. Association for Computational Linguistics
work page 2013
-
[7]
Nathan Chi, Teodor Malchev, Riley Kong, Ryan Chi, Lucas Huang, Ethan Chi, R. McCoy, and Dragomir Radev. 2024. https://aclanthology.org/2024.sigtyp-1.14 M ode L ing: A novel dataset for testing linguistic reasoning in language models . In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 113--119, ...
work page 2024
-
[8]
Haisu Guan, Huanxin Yang, Xinyu Wang, Shengwei Han, Yongge Liu, Lianwen Jin, Xiang Bai, and Yuliang Liu. 2024. https://arxiv.org/abs/2406.00684 Deciphering oracle bone language with diffusion models . Preprint, arXiv:2406.00684
arXiv 2024
Show all 14 references
-
[9]
Zheng-Lin Lin, Chiao-Han Yen, Jia-Cheng Xu, Deborah Watty, and Shu-Kai Hsieh. 2023. https://aclanthology.org/2023.rocling-1.33 Solving linguistic olympiad problems with tree-of-thought prompting . In Proceedings of the 35th Conference on Computational Linguistics and Speech Pr...
2023
-
[10]
Jiaming Luo, Yuan Cao, and Regina Barzilay. 2019. https://arxiv.org/abs/1906.06718 Neural decipherment via minimum-cost flow: from ugaritic to linear b . Preprint, arXiv:1906.06718
2019 arXiv
-
[11]
Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, et al
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, et al. 2024. https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal mo...
2024 arXiv
-
[12]
Saujas Vaduguru, Aalok Sathe, Monojit Choudhury, and Dipti Sharma. 2021. https://doi.org/10.18653/v1/2021.sigmorphon-1.7 Sample-efficient linguistic generalizations through program synthesis: Experiments with phonology problems . In Proceedings of the 18th SIGMORPHON Workshop ...
2021 doi
-
[13]
Pengjie Wang, Kaile Zhang, Xinyu Wang, Shengwei Han, Yongge Liu, Jinpeng Wan, Haisu Guan, Zhebin Kuang, Lianwen Jin, Xiang Bai, and Yuliang Liu. 2024. https://arxiv.org/abs/2401.15365 An open dataset for oracle bone script recognition and decipherment . Preprint, arXiv:2401.15365
2024 arXiv
-
[14]
Hengyuan Zhang, Peng Chang, and Zongcheng Ji. 2023. https://arxiv.org/abs/2308.11891 Bridging the gap: Deciphering tabular data using large language model . Preprint, arXiv:2308.11891
2023 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.