Pith. sign in

REVIEW 3 major objections 5 minor 17 references

MORE is a 149-language document-parsing benchmark that shows models still fail on tables and rare scripts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:50 UTC pith:NUT76TBU

load-bearing objection Useful 149-language real-PDF document-parsing benchmark with standard metrics and clear baselines; annotation reliability for sparse long-tail structures is the main soft spot, not a collapse of the contribution. the 3 major comments →

arxiv 2607.02956 v1 pith:NUT76TBU submitted 2026-07-03 cs.CV cs.CL

MORE: A Multilingual Document Parsing Benchmark and Evaluation

Long Xu , Binghong Wu , Tinghao Yu , Hao Feng , Zhenyu Huang , Haoqing Jiang , Yunhao Wang , Shuo Huang
show 1 more author
Feng Zhang
This is my paper
classification cs.CV cs.CL
keywords multilingual document parsingvision-language modelsOCR benchmarklayout analysistable recognitionlong-tail languagesreading orderstructural elements
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern vision-language models claim they can parse documents in hundreds of languages, yet almost every public test set covers only English, Chinese, and a handful of others. MORE fills that gap with real pages drawn from the web, spanning 149 languages and six script families, and scoring not only plain text but also formulas, tables, code blocks, catalogs, and reading order. The authors build the ground truth with a multi-model pre-annotation step followed by human correction, then run leading specialized and general models on the set. The resulting scores show that character-level recognition is already strong for high-resource scripts, while table structure and long-tail scripts remain clear bottlenecks. The benchmark therefore supplies the first quantitative map of where multilingual document understanding actually works and where it still breaks.

Core claim

MORE is the most linguistically diverse document-parsing benchmark yet (149 languages). Its decoupled and layout-dependent metrics on authentic pages establish reliable baselines that expose remaining failures on structural elements (especially tables) and long-tail scripts, even for models that claim broad language coverage.

What carries the argument

A model-assisted, human-refined annotation pipeline that turns real-world PDF pages into gold-standard labels for six tasks (text, formula, table, code, catalog, reading order), scored by normalized edit distance, CDM, and TEDS and then averaged into both task-wise and layout-dependent overall scores.

Load-bearing premise

The claim rests on the assumption that the human-refined labels produced by the multi-model pipeline are accurate gold standards for all 149 languages and for the sparse structural elements.

What would settle it

Independent re-annotation of a stratified sample of rare-script pages and tables by native speakers who never saw the original model candidates; large systematic disagreements with the released ground truth would undermine the baselines.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers can now measure claimed multilingual OCR support on a common 149-language yardstick rather than relying on marketing statements.
  • Table parsing and complex-layout detection are identified as the primary remaining bottlenecks, so future model work can target them directly.
  • Long-tail scripts receive explicit numerical baselines, making progress on under-represented languages trackable.
  • The dual decoupled / layout-dependent scoring protocol can be reused by later benchmarks that want both recognition purity and end-to-end realism.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the structural-element counts are low for rare languages, absolute scores on tables and code for those languages will remain noisy until the dataset is expanded.
  • Models whose pre-training or fine-tuning data overlap the pre-annotation ensemble may inherit a quiet advantage that only independent annotation can reveal.
  • The same real-page collection method could be extended to multi-page documents and to historical scripts, turning MORE into a living test suite for global document intelligence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MORE, a document-parsing benchmark of 1,288 real-world PDF pages spanning 149 languages (six script families). Samples are obtained by web crawling, filtering, language classification, and stratified sampling (up to 10 pages per language after excluding Chinese/English). Annotation uses a model-assisted, human-refined pipeline covering six tasks: text, formula (LaTeX), table (HTML), code, catalog, and reading order. Metrics follow standard practice (NED for sequential elements, CDM for formulas, TEDS for tables); overall scores are arithmetic means of the six tasks, reported both task-wise/page-wise and under a layout-dependent (end-to-end) protocol. Several specialized and general VLMs are evaluated; HunyuanOCR leads the decoupled setting (92.42), while dots.ocr leads the layout-dependent setting (80.68). The authors claim MORE is the most linguistically comprehensive such benchmark and that it supplies reliable baselines exposing bottlenecks in structural parsing (especially tables) and long-tail scripts.

Significance. The linguistic scale (149 languages) and insistence on authentic, non-synthetic pages address a genuine evaluation gap: models now claim 100+ language support while existing public benchmarks cover far fewer languages and rarely include structural elements. The dual protocol (decoupled content recognition vs. layout-dependent end-to-end scoring) is methodologically useful and cleanly separates recognition from detection/order errors. Explicit COI disclosure and the planned open release of the dataset are strengths. If annotation quality for long-tail scripts and sparse structures can be substantiated, MORE would become a lasting community resource and a practical diagnostic tool for multilingual document intelligence.

major comments (3)
  1. [§3.2, Table 3] Section 3.2 and Table 3: The central claim that MORE supplies 'reliable' gold-standard baselines rests on a model-assisted, human-refined pipeline whose pre-annotation ensemble includes HunyuanOCR, PaddleOCR-VL, dots.ocr and Qwen-VL—the same families later ranked in Tables 4–12. No inter-annotator agreement, residual-error audit, or per-script expert qualification is reported. For rare scripts and for the sparse structural classes, residual label noise that systematically favors the pre-annotators cannot be ruled out; this is load-bearing for the ranking and bottleneck diagnoses.
  2. [Table 3, §5.3–5.4] Table 3 and §5.3–5.4: Structural element counts are extremely small (82 formulas, 94 tables, 73 code blocks, 104 catalogs across all 149 languages). Many languages contribute zero samples to the very tasks used to claim 'structural complexity' and table bottlenecks. Task-wise averages over such sparse sets are statistically fragile; language-level claims (e.g., Tables 7–10) and the assertion that tables remain the primary bottleneck for long-tail scripts therefore rest on thin evidence and need either denser sampling or explicit uncertainty quantification (confidence intervals / leave-one-out stability).
  3. [§4, Eq. (7)] §4 Overall Score (Eq. 7): The final score is the unweighted arithmetic mean of six tasks whose sample sizes differ by two orders of magnitude (8 221 text vs. 73 code). This equal weighting can let a handful of code/catalog/table pages dominate the ranking relative to the far larger text corpus. The paper should either justify the equal-task design with a sensitivity analysis or report sample-size-weighted and task-stratified aggregates so that readers can assess robustness.
minor comments (5)
  1. [Figure 6] Figure 6 label '(e). Catelogue' is misspelled; also 'Catelogue' appears inconsistently with 'Catalog' elsewhere.
  2. [Tables 13–14, Appendix B] Several languages (e.g., Amharic, Tibetan, Burmese, Sinhala) have zero paragraph annotations (Tables 13–14). Clarify how overall and text scores are defined when a language contributes only layout/order or empty content.
  3. [Figure 1] Figure 1 (winner-takes-all by script) is informative but the color legend and outer-ring script grouping are hard to parse at print scale; a supplementary tabular breakdown would help.
  4. [§5.4] The layout-dependent protocol is described as following OmniDocBench’s 'quick match'; a short formal definition or pseudocode in Appendix A would improve reproducibility.
  5. [§6 / Impact Statement] Impact Statement acknowledges sparsity of structural elements for rare languages; this limitation should also be stated more prominently in the main-text conclusion rather than only in the impact paragraph.

Circularity Check

1 steps flagged

No load-bearing circular derivation; mild annotation-pipeline involvement of evaluated models is disclosed COI risk, not tautological reduction of scores to inputs.

specific steps
  1. other [Sec. 3.2 Expert Annotation + Conflict of Interest Disclosure]
    "we implement a model-assisted, human-refined pipeline... leverages an ensemble of models to generate anonymous candidates (Li et al., 2025b; Cui et al., 2025; Team et al., 2025; Bai et al., 2025b;a)... Tables (HTML with complex spans) and Catalogs were generated via HunyuanOCR and PaddleOCR-VL... All authors are employed by Tencent, the company that leads the development of HunyuanOCR, one of the models evaluated in this paper."

    Pre-annotation for the very structural elements (tables, catalogs) on which HunyuanOCR later ranks first partially re-uses the same model family; human refinement is claimed but unquantified (no IAA). This is a mild validity risk of residual preference, not a definitional reduction of the reported scores to the inputs.

full rationale

MORE is an empirical benchmark paper, not a first-principles derivation. Its central claims (149-language coverage, structural-element support, real-world authenticity, and the resulting model rankings/baselines) rest on a constructed dataset plus direct evaluation metrics (NED, TEDS, CDM, arithmetic mean). There are no equations, fitted parameters renamed as predictions, uniqueness theorems, or ansatzes that reduce by construction to the paper’s own inputs. The sole soft spot is the model-assisted annotation pipeline (Sec. 3.2), which includes HunyuanOCR (authors’ employer) among the pre-annotators for tables/catalogs; humans then adopt exact matches or refine and discard ambiguities. This creates a possible residual label bias favoring the pre-annotators, but the paper does not claim the rankings are forced by that pipeline, discloses the COI explicitly, evaluates multiple independent models, and reports both decoupled and layout-dependent scores. Per the analyzer rules this is ordinary self-involvement, not circularity of the logical chain; the benchmark remains an independent artifact whose scores are not definitionally equivalent to its construction inputs. Score therefore remains near zero.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

As a benchmark paper the load-bearing commitments are methodological choices rather than physical axioms: stratified sampling limits, the model-assisted annotation protocol, and the decision to treat human-refined model outputs as gold. No free parameters are fitted to produce a scientific constant; the invented entity is the benchmark itself.

free parameters (2)
  • max_PDFs_per_language = 10
    Stratified sampling caps at 10 PDFs per language and one random page each; this hand-chosen limit directly determines language balance and total size (1 237–1 288 pages).
  • annotation_retention_filters
    Human experts discard ambiguous cases (e.g., 8 221/10 403 text, 73/151 code retained); the acceptance threshold is not quantified and affects final gold counts.
axioms (3)
  • domain assumption Human-refined outputs of an ensemble of VLMs (including models from the authors' organization) constitute reliable ground truth for all 149 languages and structural types.
    Stated in §3.2; no IAA or expert-qualification statistics are supplied for long-tail scripts.
  • domain assumption Normalized Edit Distance, TEDS, and CDM are appropriate and comparable metrics across scripts and layouts.
    Adopted from OmniDocBench and prior OCR literature (§4).
  • ad hoc to paper Excluding Chinese/English and unlabeled data, then sampling up to 10 pages per remaining language, yields an unbiased evaluation set for long-tail performance.
    §3.1 sampling design; web-crawl bias is acknowledged but not measured.
invented entities (1)
  • MORE benchmark no independent evidence
    purpose: Provide gold-standard multilingual document parsing evaluation covering 149 languages and six structural tasks.
    The dataset and its annotation protocol are the paper's primary contribution; independent evidence will exist once the GitHub release is public and used by third parties.

pith-pipeline@v1.1.0-grok45 · 47747 in / 2651 out tokens · 25343 ms · 2026-07-12T05:50:10.368551+00:00 · methodology

0 comments
read the original abstract

Multilingual documents encapsulate rich regional cultures, scientific discoveries, and historical records. Parsing this content into structured, machine-readable formats is critical for unlocking global knowledge. However, existing benchmarks predominantly focus on high-resource languages like English and Chinese, creating an evaluation blind spot concerning model performance on other languages. While recent Vision-Language Models (VLMs) claim support for hundreds of languages, the lack of ground truth makes it impossible to empirically verify these capabilities. To bridge this gap, we introduce MORE, a large-scale benchmark designed for multilingual document parsing evaluation. MORE distinguishes itself through three key dimensions: (1) Unprecedented Scale: It covers 149 languages, making it the most linguistically diverse benchmark to date; (2) Structural Complexity: Unlike previous works, it extends evaluation beyond plain text to include structural elements such as code blocks, tables, and catalogs; and (3) Data Authenticity: All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline. We evaluate state-of-the-art models using MORE, establishing new performance baselines for long-tail languages and validating the benchmark's effectiveness in diagnosing model capabilities in realistic, diverse scenarios. The MORE dataset will be available at https://github.com/zimoqingfeng/MORE.

Figures

Figures reproduced from arXiv: 2607.02956 by Binghong Wu, Feng Zhang, Hao Feng, Haoqing Jiang, Long Xu, Shuo Huang, Tinghao Yu, Yunhao Wang, Zhenyu Huang.

Figure 1
Figure 1. Figure 1: The Winner-Takes-All landscape of multilingual OCR. We evaluate state-of-the-art models on 149 languages spanning diverse script families (outer ring). The color of each bar denotes the model achieving the highest accuracy for that language, illus￾trating the competitive landscape across diverse linguistic regions. stands as the critical link transforming them into machine￾actionable knowledge for Large La… view at source ↗
Figure 2
Figure 2. Figure 2: Diverse document samples from the MORE benchmark. The dataset covers a wide spectrum of languages and scripts, including Lao (Sanskrit script), Bosnian (Latin script), Pashto (Arabic script), and Greek. These samples highlight the benchmark’s diversity in handling complex layouts and diverse content across 149 languages. established a strong baseline with support for over 10 lan￾guages. Building on this, Q… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the model-assisted, human-refined annotation pipeline. To guarantee data reliability, we employ a multi-stage approach: raw data is first processed by diverse models for pre-annotation, followed by rigorous human verification and validation to correct errors and resolve ambiguities [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Treemap visualizing MORE’s balanced language distri￾bution across six script families via stratified sampling. 3.1. PDF Collection and Stratified Sampling To ensure real-world diversity, our data collection process commenced with a robust web-crawling methodology in￾spired by CCpdf (Turski et al., 2023), harvesting an ex￾tensive pool of over 20 million PDFs from diverse web sources. To guarantee data quali… view at source ↗
Figure 5
Figure 5. Figure 5: Word clouds illustrating the most frequent tokens in the MORE dataset, categorized by six major language families. and Sanskrit together account for over 30%, followed by Arabic, Chinese, and the Other category (5.37%), which en￾compasses unique scripts like Greek, Hebrew, and Georgian. Collectively, these non-Latin scripts constitute nearly half of the dataset (46.31%), ensuring rigorous evaluation across… view at source ↗
Figure 7
Figure 7. Figure 7: Language confusion analysis on text. inherent document parsing capabilities; and Specialized VLMs (e.g., dots.ocr, PaddleOCR-VL, DeepSeek-OCR, HunyuanOCR, MinerU2.5), which have been specifically fine-tuned for document parsing tasks. 5.1. Overall Evaluation State-of-the-Art Performance: As presented in [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Model predictions on multilingual formula in MORE. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Model predictions on multilingual table in MORE. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Model predictions on multilingual code in MORE. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Model predictions on multilingual catalog in MORE. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 11 linked inside Pith

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    Qwen3- VL technical report.arXiv preprint arXiv:2511.21631, 2025a

    Bai, S., Cai, Y ., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. Qwen3- VL technical report.arXiv preprint arXiv:2511.21631, 2025a. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2.5- VL technical report.arXiv preprint arXiv:2502.13923, 2025b. Blecher, L., C...

    Pith/arXiv arXiv

  3. [3]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  4. [4]

    PaddleOCR- VL: Boosting multilingual document parsing via a 0.9B ultra-compact vision-language model.arXiv preprint arXiv:2510.14528,

    Cui, C., Sun, T., Liang, S., Gao, T., Zhang, Z., Liu, J., Wang, X., Zhou, C., Liu, H., Lin, M., et al. PaddleOCR- VL: Boosting multilingual document parsing via a 0.9B ultra-compact vision-language model.arXiv preprint arXiv:2510.14528,

    arXiv

  5. [5]

    Dolphin: Document im- age parsing via heterogeneous anchor prompting.arXiv preprint arXiv:2505.14059,

    Feng, H., Wei, S., Fei, X., Shi, W., Han, Y ., Liao, L., Lu, J., Wu, B., Liu, Q., Lin, C., et al. Dolphin: Document im- age parsing via heterogeneous anchor prompting.arXiv preprint arXiv:2505.14059,

    Pith/arXiv arXiv

  6. [6]

    Seed1.5-VL technical report.arXiv preprint arXiv:2505.07062,

    Guo, D., Wu, F., Zhu, F., Leng, F., Shi, G., Chen, H., Fan, H., Wang, J., Jiang, J., Wang, J., et al. Seed1.5-VL technical report.arXiv preprint arXiv:2505.07062,

    Pith/arXiv arXiv

  7. [7]

    FastText.zip: Compressing text classifi- cation models.arXiv preprint arXiv:1612.03651,

    Joulin, A., Grave, E., Bojanowski, P., Douze, M., J´egou, H., and Mikolov, T. FastText.zip: Compressing text classifi- cation models.arXiv preprint arXiv:1612.03651,

    Pith/arXiv arXiv

  8. [8]

    Beyond cropped regions: new benchmark and corresponding baseline for chinese scene text retrieval in diverse layouts.arXiv preprint arXiv:2506.04999, 2025a

    Li, G., Shen, H., and Zhou, Y . Beyond cropped regions: new benchmark and corresponding baseline for chinese scene text retrieval in diverse layouts.arXiv preprint arXiv:2506.04999, 2025a. Li, G., Lyu, P., Zhang, C., Shen, H., Wu, L., Wan, X., Zeng, G., Hu, H., Ma, C., and Zhou, Y . Towards real-world doc- ument parsing via realistic scene synthesis and d...

    Pith/arXiv arXiv

  9. [9]

    Kosmos- 2.5: A multimodal literate model.arXiv preprint arXiv:2309.11419,

    10 MORE: A Multilingual Document Parsing Benchmark and Evaluation Lv, T., Huang, Y ., Chen, J., Zhao, Y ., Jia, Y ., Cui, L., Ma, S., Chang, Y ., Huang, S., Wang, W., et al. Kosmos- 2.5: A multimodal literate model.arXiv preprint arXiv:2309.11419,

    Pith/arXiv arXiv

  10. [10]

    D- FINE: Redefine regression task in DETRs as fine-grained distribution refinement.arXiv preprint arXiv:2410.13842,

    Peng, Y ., Li, H., Wu, P., Zhang, Y ., Sun, X., and Wu, F. D- FINE: Redefine regression task in DETRs as fine-grained distribution refinement.arXiv preprint arXiv:2410.13842,

    Pith/arXiv arXiv

  11. [11]

    olmOCR: Unlocking trillions of tokens in PDFs with vi- sion language models.arXiv preprint arXiv:2502.18443,

    Poznanski, J., Rangapur, A., Borchardt, J., Dunkelberger, J., Huff, R., Lin, D., Wilhelm, C., Lo, K., and Soldaini, L. olmOCR: Unlocking trillions of tokens in PDFs with vi- sion language models.arXiv preprint arXiv:2502.18443,

    arXiv

  12. [12]

    MTVQA: Bench- marking multilingual text-centric visual question answer- ing

    Tang, J., Liu, Q., Ye, Y ., Lu, J., Wei, S., Wang, A.-L., Lin, C., Feng, H., Zhao, Z., Wang, Y ., et al. MTVQA: Bench- marking multilingual text-centric visual question answer- ing. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 7748–7763,

    2025

  13. [13]

    V ., Lyu, P., Wan, X., Li, G., Peng, S., Wang, W., Wu, L., Shen, H., Zhou, Y ., Tang, C., et al

    Team, H. V ., Lyu, P., Wan, X., Li, G., Peng, S., Wang, W., Wu, L., Shen, H., Zhou, Y ., Tang, C., et al. HunyuanOCR technical report.arXiv preprint arXiv:2511.19575,

    arXiv

  14. [14]

    LLaMA: Open and efficient founda- tion language models.arXiv preprint arXiv:2302.13971,

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and efficient founda- tion language models.arXiv preprint arXiv:2302.13971,

    Pith/arXiv arXiv

  15. [15]

    Vary: Scaling up the vision vocabulary for large vision-language model

    Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yang, J., Sun, J., Han, C., and Zhang, X. Vary: Scaling up the vision vocabulary for large vision-language model. In Proceedings of the European Conference on Computer Vision, pp. 408–424, 2024a. Wei, H., Liu, C., Chen, J., Wang, J., Kong, L., Xu, Y ., Ge, Z., Zhao, L., Sun, J., Peng, Y ., et al. General OCR ...

    Pith/arXiv arXiv

  16. [16]

    based on LaTeX. Following (Wang et al., 2025), we render LaTeX pairs (Pi, Gi) into images to perform spatial character matching, calculating the score as: CDM= 1 N NX i=1 2·T P i 2·T P i +F P i +F N i (6) where T Pi denotes the number of matched bounding box pairs, while F Pi and F Ni represent the count of unmatched bounding boxes in the prediction and g...

    arXiv 2025

  17. [17]

    http://www.w3.org/2000/svg

    = 1, 2 8 \cdot 1 0 ^ {6} \mathrm {M D} \text {区} / \mathrm {r o} \Omega .\]**Qwen3VL-2B**```html<svg xmlns="http://www.w3.org/2000/svg" width="1000" height="100" viewBox="0 0 1000 100"><text x="20" y="30" font-size="24" fill="black">Q<sub>бол</sub> =</text><text x="20" y="50" font-size="24" fill="black">12 950</text><text x="20" y="70" font-size="24" fill...

    2000