Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

PictOBI-20k: Unveiling Large Multimodal Models in Visual Decipherment for Pictographic Oracle Bone Characters

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large multimodal models show only preliminary skill at matching oracle-bone pictographs to objects, and a direct readout of their vision encoders suggests the image is not driving their answers.

desk verdict PictOBI-20k is a real dataset contribution, but the paper's central claim about LMMs ignoring visual information is not supported by the experiment as designed. read the letter →

arxiv 2509.05773 v1 pith:PDKPLJGP submitted 2025-09-06 cs.CV

Zijian Chen , Wenjie Hua , Jinhao Li , Lirong Deng , Fan Du , Tingzhu Chen , Guangtao Zhai This is my paper
classification cs.CV
keywords largemultimodalmodelsoraclebonecharacterancientscriptinterpretationvisualdeciphermentevaluationpictographicvisionencoderlanguageprior
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PictOBI-20k asks whether large multimodal models can look at an ancient Chinese oracle-bone pictograph and pick the real-world object it depicts. The paper builds a benchmark of 15,175 four-choice questions from 80 pictographic character types, pairing 15,175 oracle-bone images with 4,833 object photos, plus 240 human-annotated reference-point pairs. Across 11 large models, the best (Gemini 2.5 Pro) reaches 53.66% accuracy and InternVL3-38B reaches 51.40%, well above the 25% random baseline but far from solved. The core finding is diagnostic: when the models' vision encoders are read out directly, a standalone DINOv2 encoder scores above several full multimodal models, which the authors take as evidence that LMMs are not effectively using visual information and are mostly leaning on language priors. If correct, this shifts the bottleneck from model scale to visual attention.

What carries the argument

The benchmark itself is the mechanism: PictOBI-20k, a set of 15,175 four-choice visual decipherment questions built from 80 categories of pictographic oracle-bone characters, with cyclical option rotation to remove position bias and 240 human-annotated reference-point pairs for alignment studies. The diagnosis rests on a comparison technique drawn from the paper's reference [20]: computing cosine similarity between the patch features of a vision encoder and the object image, treating the encoder's highest-scoring option as its prediction, and contrasting that readout with the LMM's answer. Attention maps and reference-point consistency with humans round out the diagnostic.

What would settle it

Run the same 15,175 multiple-choice prompts twice: once with the oracle-bone image, once with a blank or heavily scrambled version of the same image. If top LMM accuracy stays near 53.7% in both runs, the language-prior diagnosis is supported; if accuracy drops to near 25% random chance when the image is removed, the models are using the visual input and the neglect claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that general large multimodal models possess preliminary but unreliable visual decipherment skills for pictographic oracle-bone characters, and that their mistakes trace to a specific cause: they are not effectively using the image. On PictOBI-20k, no model reaches human-level alignment; Gemini 2.5 Pro matches human reference points 76.25% of the time, and the best overall accuracy is 53.7%. When the vision encoder is evaluated in isolation via cosine similarity of patch features to object images, DINOv2 outperforms several full LMMs, including Claude 4 Sonnet and InternVL3-78B, even though the LMMs contain the same visual backbone plus language layers. The autho

Load-bearing premise

The diagnosis that models ignore the image depends on the assumption that a similarity score between the vision encoder's image features and the object photo measures the visual information that matters for decipherment; if that proxy is invalid, the head-to-head comparison does not establish visual neglect.

Editorial extensions

If this is right

  • If the diagnosis is right, adding a vision-focused training signal that pushes the model to attend to the oracle-bone character's shape should raise accuracy above the current 53.7% ceiling.
  • Scaling model size alone will not solve visual decipherment: the observed increase-then-saturate pattern across Qwen2.5-VL and InternVL3 families implies a practical limit on language-side growth.
  • Vision-encoder readouts can serve as a cheap diagnostic for whether any LMM is ignoring its image input, not just in this benchmark.
  • The 240 reference-point annotations give a concrete human-alignment target that future models can be evaluated against, rather than relying on text-only ground truth.
  • The dataset's separation of normal and complex pictographs offers a way to measure how much contextual reasoning is needed beyond simple shape matching.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the language-prior diagnosis generalizes, the same benchmark design could be ported to other pictographic or logographic scripts, such as Egyptian hieroglyphs or Maya glyphs, where the glyph's visual shape is the main clue.
  • A decisive ablation is directly testable: run the same 15,175 prompts with the oracle-bone image replaced by a blank or scrambled image; if accuracy stays near 53.7%, the language-prior explanation is confirmed, and if it crashes toward 25%, the neglect claim is wrong.
  • The 'golden ratio' between vision and language capacities is suggestive but not demonstrated; it could be tested by pairing fixed vision encoders with language models of varying size under controlled training.
  • The paper's use of human reference points implies a future training objective: supervise LMMs to produce attention maps aligned with human reference points, which may improve both decipherment and interpretability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces PictOBI-20k, a dataset for evaluating large multimodal models (LMMs) on the visual decipherment of pictographic oracle bone characters (OBCs). The dataset contains 15,175 OBC images, 4,833 real-object images, 15,175 four-choice visual-matching questions, and 240 expert-annotated reference-point image pairs. The authors evaluate 11 LMMs and 3 standalone vision encoders. They report that most LMMs perform near or below 35% accuracy while Gemini 2.5 Pro and InternVL3-38B reach roughly 50%, and they claim that LMMs possess preliminary decipherment ability but do not effectively use visual information, instead being limited by language priors. The authors also compare human and machine visual-reference-point consistency.

Significance. If the claims are substantiated, PictOBI-20k would be a useful benchmark for a culturally and scientifically interesting task, and the observation that LMMs underuse their visual representations would have implications for multimodal model design in specialized visual domains. The dataset contribution is concrete: 80 pictographic OBC classes, 15k+ questions, expert reference-point annotations, and planned public release. The paper also evaluates a reasonable spread of proprietary and open LMMs. However, the headline interpretive claim about LMMs neglecting visual information rests on a comparison that is not controlled, and one reported accuracy number is arithmetically impossible. These issues must be addressed before the central conclusion can be accepted.

major comments (3)
  1. [Table 2] The row for Qwen2.5-VL-32B reports Normal accuracy 26.77%, Complex accuracy 29.11%, and Overall accuracy 32.56%. Since Overall must be a weighted average of the Normal and Complex subsets, it cannot exceed both component accuracies. This is an internal inconsistency. Because the reported Overall value is used in the scaling-law discussion ('increase-first, saturate-later') and in the ranking of models, the table needs to be corrected and the affected conclusions rechecked.
  2. [§3.2, Fig. 6] The claim that 'LMMs are not effectively using visual information' is supported by comparing LMM multiple-choice accuracy with a cosine-similarity readout from standalone DINOv2, CLIP, and InternViT encoders. This is not a controlled comparison: the standalone encoders differ in architecture and pretraining from the vision towers inside the evaluated LMMs, and the readout rule (patch-feature cosine similarity to the answer image) is different from the generative multiple-choice decision rule used by an LMM. A higher readout accuracy could reflect encoder choice or decision-rule advantage rather than LMM neglect of vision. To support the claim, the readout should be computed from the same vision encoder used by each LMM and compared with that LMM's accuracy; otherwise the conclusion in the abstract and §3.2 is not established.
  3. [§3.2, Abstract] The paper attributes LMM errors to 'language priors' but reports no text-only (image-ablated) condition and no analysis of answer-choice distribution. Without measuring performance when the OBC image is withheld or when options are manipulated, the specific claim that models are limited by language priors rather than by visual encoding or decision-rule issues is unsupported. A text-only baseline or a controlled intervention on the visual input is needed to separate these factors.
minor comments (4)
  1. [Author affiliation] The affiliation for East China Normal University contains the typo 'Unversity'.
  2. [Fig. 6 caption] 'DIVOv2' should be 'DINOv2'. Also, the caption says the figure shows attention maps, but the method for producing these attention maps is not described in the main text; please specify how they were obtained.
  3. [§2.2] The human-annotation section reports five experts and 240 annotated pairs but gives no inter-annotator agreement measure. Since the reference-point consistency comparison in Fig. 5 treats human annotations as ground truth, a brief agreement statistic would strengthen the reliability of the reference standard.
  4. [Fig. 4] The figure reports average accuracy per class, but the specific classes with highest and lowest accuracy are not labeled in the main text. Adding explicit labels would make the point about structural similarity more concrete.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: benchmark evaluation is self-contained; minor self-citations in protocol choices do not force the central visual-decipherment findings.

full rationale

PictOBI-20k is an empirical benchmark paper, not a derivation from first principles. The dataset is constructed from external OBC and object-image sources, the Qiu taxonomy for class selection, and human annotations; no fitted parameter or defining equation makes a prediction equal to its input. The headline claim that LMMs underuse visual information is inferred by comparing LMM accuracy with standalone vision-encoder readouts. That comparison is an uncontrolled experiment (different encoder architectures, training objectives, and decision rules) and Table 2 contains an arithmetically impossible overall accuracy for Qwen2.5-VL-32B (32.56 cannot be a weighted average of 26.77 and 29.11). These are validity/reliability concerns, not circularity: the encoder readout is computed via the external cosine-similarity rule from [20] and is not defined in terms of the LMM accuracy, nor is any result forced by construction. Self-citations [1] and [25] justify adopting multi-choice instead of yes/no questions to avoid response bias; they support protocol choice but do not load-bear the experimental outcome. Other self-citations ([2], [3]) are background references and are not used as premises for the central claim. Thus no circular step is present; the score of 2 reflects only minor, non-load-bearing self-citation in the evaluation design.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted parameters or invented entities. The benchmark's ground truth rests on external taxonomy, expert annotations, and a borrowed vision-similarity proxy.

assumptions (4)
  • domain assumption The 80 pictographic OBC categories and their real-object correspondences follow Xigui Qiu's taxonomy.
    Ground truth for the multiple-choice labels depends on this external classification; no independent verification of each pairing is provided.
  • domain assumption Visual decipherment can be operationalized as matching an OBC image to one of four real-object images.
    The task assumes the object-referential method is a valid proxy for character decipherment, rather than phonetic or contextual meaning.
  • domain assumption Expert annotations from five OBC specialists on 240 pairs are correct ground truth for visual reference points.
    Used as the gold label for the consistency experiment; inter-annotator agreement is not reported.
  • domain assumption Cosine similarity between patch features of vision encoders measures visual matching ability (per reference [20]).
    The diagnosis that LMMs undervalue visual information compares LMM accuracy to this encoder-only cosine score; the validity of this proxy is not validated on this dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PictOBI-20k: Unveiling Large Multimodal Models in Visual Decipherment for Pictographic Oracle Bone Characters." pith.science (2026). https://pith.science/paper/PDKPLJGP

@misc{pith2026250905773,
  author       = {Pith},
  title        = {Pith review of: PictOBI-20k: Unveiling Large Multimodal Models in Visual Decipherment for Pictographic Oracle Bone Characters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDKPLJGP}},
  note         = {Machine review of arXiv:2509.05773}
}
read the original abstract

Deciphering oracle bone characters (OBCs), the oldest attested form of written Chinese, has remained the ultimate, unwavering goal of scholars, offering an irreplaceable key to understanding humanity's early modes of production. Current decipherment methodologies of OBC are primarily constrained by the sporadic nature of archaeological excavations and the limited corpus of inscriptions. With the powerful visual perception capability of large multimodal models (LMMs), the potential of using LMMs for visually deciphering OBCs has increased. In this paper, we introduce PictOBI-20k, a dataset designed to evaluate LMMs on the visual decipherment tasks of pictographic OBCs. It includes 20k meticulously collected OBC and real object images, forming over 15k multi-choice questions. We also conduct subjective annotations to investigate the consistency of the reference point between humans and LMMs in visual reasoning. Experiments indicate that general LMMs possess preliminary visual decipherment skills, and LMMs are not effectively using visual information, while most of the time they are limited by language priors. We hope that our dataset can facilitate the evaluation and optimization of visual attention in future OBC-oriented LMMs. The code and dataset will be available at https://github.com/OBI-Future/PictOBI-20k.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond Single Character: Evaluating MLLMs for Sentence-Level Oracle Bone Inscription Understanding

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    The paper introduces the S-OBI benchmark for sentence-level oracle bone inscription understanding and reports that current MLLMs remain dependent on character-level recognition due to propagating visual errors.

Reference graph

Works this paper leans on

31 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Since its discovery in 1899, its decipherment has been a primary objective for schol- ars

    INTRODUCTION Oracle bone character (OBC), as the oldest form of written Chinese, dating to the Shang Dynasty (1250–1050 BC), re- flects humanity’s early modes of production, social structures, and the evolution of civilization [ 1]. Since its discovery in 1899, its decipherment has been a primary objective for schol- ars. The existing methodologies for OB...

    work page

  2. [2]

    Genealogical comparison, 3) Contextual analysis, and 4) Object-referential method. From the perspectives of the source of evidence and the cer- tainty of conclusion, archaeological excavations undoubtedly represent the most direct and persuasive method of deciphering. However, its randomness and uncontrollability hinder wide implementation. Recently, gene...

    arXiv 2025

  3. [3]

    increase-first, saturate-later

    EXPERIMENTS 3.1. Evaluation Setup Baselines.We evaluate a total of 11 LMMs, with a mix of top proprietary models, large and small open-source models, including GPT-4o [8], Gemini 2.5 Pro [24], Claude 4 Sonnet [22], GLM-4.5V [23], Qwen2.5-VL-{3B, 7B, 32B, 72B} [5], and InternVL3-{8B, 38B, 78B} [19], based on API and their official implementations. To study...

    work page

  4. [4]

    Which real object corresponds to the given OBC image?

    THE PICTOBI-20K 2.1. Data Curation To construct a holistic benchmark specifically dedicated to the visual decipherment of pictographic OBCs, we mainly focus on the font appearances, categories, and visual align- ment. Specifically, we follow the taxonomy of pictographic oracle bone characters studied by Xigui Qiu [4], an authority in the field of OBI rese...

    work page 2024

  5. [5]

    Qwen2. 5-vl technical report,

    S. Baiet al., “Qwen2. 5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    PictOBI-20k consists of 15k multi-choice questions and 240 reference point maps with accurate human annotations, which cover 80 OBC categories

    CONCLUSION In this work, we construct PictOBI-20k, a large-scale OBC- object paired image dataset for evaluating LMMs in terms of visual decipherment capabilities on pictographic OBCs. PictOBI-20k consists of 15k multi-choice questions and 240 reference point maps with accurate human annotations, which cover 80 OBC categories. We conduct a thorough evalua...

    work page

  7. [7]

    Obi-bench: Can lmms aid in study of ancient script on oracle bones?

    Z. Chen, T. Chen, W. Zhang, and G. Zhai, “Obi-bench: Can lmms aid in study of ancient script on oracle bones?” inThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  8. [8]

    Mitigating Long-tail Distribution in Oracle Bone Inscriptions: Dataset, Model, and Benchmark

    J. Li, Z. Chen, R. Jiang, T. Chen, C. Wang, and G. Zhai, “Mitigating long-tail distribution in oracle bone inscrip- tions: Dataset, model, and benchmark,”arXiv preprint arXiv:2504.09555, 2025

    work page Pith review arXiv 2025

Show all 31 references
  1. [9]

    Obiformer: A fast attentive denoising framework for oracle bone inscriptions,

    J. Li, Z. Chen, T. Chen, Z. Liu, and C. Wang, “Obiformer: A fast attentive denoising framework for oracle bone inscriptions,”Displays, p. 103059, 2025

    2025

  2. [10]

    Chinese writing,

    X. Qiu, “Chinese writing,”The Commercial Press, 1988

    1988

  3. [11]

    Sundial-gan: A cascade generative adversarial networks framework for deciphering oracle bone inscriptions,

    X. Changet al., “Sundial-gan: A cascade generative adversarial networks framework for deciphering oracle bone inscriptions,” inACM MM, 2022, pp. 1195–1203

    2022

  4. [12]

    V-oracle: Making progressive reasoning in deciphering oracle bones for you and me,

    R. Qiao, Q. Tanet al., “V-oracle: Making progressive reasoning in deciphering oracle bones for you and me,” inProceedings of the 63rd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 20 124–20 150

    2025

  5. [13]

    Component-level segmentation for ora- cle bone inscription decipherment,

    Z. Huet al., “Component-level segmentation for ora- cle bone inscription decipherment,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 27, 2025, pp. 28 116–28 124

    2025

  6. [14]

    Hello gpt-4o,

    OpenAI, “Hello gpt-4o,” https://openai.com/index/ hello-gpt-4o, 2024, accessed: 2025-08-25

    2024

  7. [15]

    V-Oracle [6] applies principles of pic- tographic character formation and establishes a multi-stage oracle alignment tuning for improving the basic understand- ing of OBCs

    visual backbone and graph-based reasoning mechanisms for interpreting OBC. V-Oracle [6] applies principles of pic- tographic character formation and establishes a multi-stage oracle alignment tuning for improving the basic understand- ing of OBCs. However, despite the effectiv...

    1994

  8. [16]

    Self-supervised learning of orc-bert aug- mentator for recognizing few-shot oracle characters,

    W. Hanet al., “Self-supervised learning of orc-bert aug- mentator for recognizing few-shot oracle characters,” in Proceedings of the Asian Conference on Computer Vi- sion, 2020

    2020

  9. [17]

    Deciphering oracle bone language with diffusion models,

    H. Guanet al., “Deciphering oracle bone language with diffusion models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 554–15 567

    2024

  10. [18]

    Learning transferable visual mod- els from natural language supervision,

    A. Radfordet al., “Learning transferable visual mod- els from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  11. [19]

    Dynamic dataset augmentation for deep learning-based oracle bone inscriptions recognition,

    X. Yueet al., “Dynamic dataset augmentation for deep learning-based oracle bone inscriptions recognition,” ACM Journal on Computing and Cultural Heritage, vol. 15, no. 4, pp. 1–20, 2022

    2022

  12. [20]

    Oraclesage: Towards unified visual-linguistic understanding of oracle bone scripts through cross-modal knowledge fusion,

    H. Jianget al., “Oraclesage: Towards unified visual-linguistic understanding of oracle bone scripts through cross-modal knowledge fusion,”arXiv preprint arXiv:2411.17837, 2024

    2024 arXiv

  13. [21]

    Visual instruction tuning,

    H. Liuet al., “Visual instruction tuning,”Advances in neural information processing systems, vol. 36, pp. 34 892–34 916, 2023

    2023

  14. [22]

    An open dataset for oracle bone script recognition and decipherment,

    P. Wanget al., “An open dataset for oracle bone script recognition and decipherment,”arXiv preprint arXiv:2401.15365, 2024

    2024 arXiv

  15. [23]

    Unsupervised structure-texture sepa- ration network for oracle character recognition,

    M. Wanget al., “Unsupervised structure-texture sepa- ration network for oracle character recognition,”IEEE Transactions on Image Processing, vol. 31, pp. 3137– 3150, 2022

    2022

  16. [24]

    Obi dataset for ijdh and obi recognition application,

    Y . Fujikawa and L. Meng, “Obi dataset for ijdh and obi recognition application,” http://www.ihpc.se.ritsumei.ac. jp/OBIdataseIJDH.zip, 2020, accessed: 2025-07-21

    2020

  17. [25]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhuet al., “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,” arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  18. [26]

    Hidden in plain sight: Vlms overlook their visual representations,

    S. Fuet al., “Hidden in plain sight: Vlms overlook their visual representations,”arXiv preprint arXiv:2506.08008, 2025

    2025 arXiv

  19. [27]

    Dinov2: Learning robust visual fea- tures without supervision,

    M. Oquabet al., “Dinov2: Learning robust visual fea- tures without supervision,”Transactions on Machine Learning Research Journal, pp. 1–31, 2024

    2024

  20. [28]

    Introducing claude 4,

    Anthropic, “Introducing claude 4,” https://www. anthropic.com/news/claude-4, 2025, accessed: 2025-07- 05

    2025

  21. [29]

    Glm-4.5v and glm-4.1v- thinking: Towards versatile multimodal reasoning with scalable reinforcement learning,

    V . Team, W. Honget al., “Glm-4.5v and glm-4.1v- thinking: Towards versatile multimodal reasoning with scalable reinforcement learning,”arXiv preprint arXiv:2507.01006, 2025

    2025 arXiv

  22. [30]

    Gemini 2.5: Updates to our family of thinking mod- els,

    S. B. Mallick and L. Kilpatrick, “Gemini 2.5: Updates to our family of thinking mod- els,” https://developers.googleblog.com/en/ gemini-2-5-thinking-model-updates/, 2025, accessed: 2025-07-09

    2025

  23. [31]

    Just noticeable differ- ence for large multimodal models,

    Z. Chen, Y . Tian, Y . Sun, W. Sun, Z. Zhang, W. Lin, G. Zhai, and W. Zhang, “Just noticeable differ- ence for large multimodal models,”arXiv preprint arXiv:2507.00490, 2025

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.