{"id":"8a6b1d6d-7d29-48b4-a38e-7fb842b114d5","arxiv_id":"2607.02956","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"MORE provides a 149-language, structure-aware document parsing benchmark from real PDFs and reports baselines showing specialized OCR models still fail on tables and rare scripts.","lead":"MORE is a new document-parsing benchmark spanning 149 languages with real-world pages and structural elements (tables, formulas, code, catalogs). It fills a measurement gap for long-tail scripts that current VLMs claim to support but cannot be checked against gold data.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Annotation reliability for long-tail scripts and sparse structures is the load-bearing soft spot under the 'reliable baselines' claim.","rationale":"The paper's central contribution—scale (149 languages), real-world authenticity, and structural tasks—is well-supported by the construction pipeline, stratified sampling, and extensive tables. The claim that these yield 'reliable performance baselines' that 'expose remaining bottlenecks' rests on the unquantified quality of the human-refined labels, especially where structural samples are sparse and pre-annotators include the top-ranked model. The reader correctly identified this as the weakest assumption; no stronger internal inconsistency appears. The disclosed COI and planned public release already justify CONDITIONAL rather than unconditional ACCEPT. The concrete re-annotation check would settle whether residual noise actually moves the rankings or merely adds noise. No other load-bearing flaw (e.g., metric definitions or sampling design) undermines the core argument as directly.","tokens_in":43636,"tokens_out":551,"duration_ms":5285,"concrete_test":"After public release, sample 50–100 text/table/formula items from the lowest-resource scripts (Sanskrit family + 'Other'), have independent native-script experts re-annotate blindly, and recompute NED/TEDS/CDM plus the overall ranking of HunyuanOCR vs. dots.ocr/PaddleOCR-VL. If IAA is low or the top-model margin shrinks >5–10 points on the re-labeled subset, the reliability premise fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim requires that MORE's ground truth is reliable enough to establish trustworthy baselines and diagnose bottlenecks (esp. tables and long-tail scripts). Section 3.2 describes a model-assisted, human-refined pipeline that aggregates candidates from an ensemble including HunyuanOCR (authors' employer, Conflict of Interest Disclosure) and other models, then has humans adopt exact matches or refine while discarding ambiguous cases. No inter-annotator agreement, no per-script expert qualifications, and no residual-error audit are reported. Structural counts are tiny (Table 3: 73 code, 82 formula, 94 table, 104 catalog total across 149 languages), so for many long-tail languages the 'gold' labels for the very elements used to claim structural bottlenecks rest on few samples whose pre-annotations came from the same family of models later ranked. If residual label noise systematically favors the pre-annotators, the ranking and bottleneck diagnoses in Tables 4–12 become less diagnostic than claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces MORE, a document-parsing benchmark of 1,288 real-world PDF pages spanning 149 languages (six script families). Samples are obtained by web crawling, filtering, language classification, and stratified sampling (up to 10 pages per language after excluding Chinese/English). Annotation uses a model-assisted, human-refined pipeline covering six tasks: text, formula (LaTeX), table (HTML), code, catalog, and reading order. Metrics follow standard practice (NED for sequential elements, CDM for formulas, TEDS for tables); overall scores are arithmetic means of the six tasks, reported both task-wise/page-wise and under a layout-dependent (end-to-end) protocol. Several specialized and general VLMs are evaluated; HunyuanOCR leads the decoupled setting (92.42), while dots.ocr leads the layout-dependent setting (80.68). The authors claim MORE is the most linguistically comprehensive such benchmark and that it supplies reliable baselines exposing bottlenecks in structural parsing (especially tables) and long-tail scripts.","tokens_in":43879,"tokens_out":1234,"duration_ms":20148,"significance":"The linguistic scale (149 languages) and insistence on authentic, non-synthetic pages address a genuine evaluation gap: models now claim 100+ language support while existing public benchmarks cover far fewer languages and rarely include structural elements. The dual protocol (decoupled content recognition vs. layout-dependent end-to-end scoring) is methodologically useful and cleanly separates recognition from detection/order errors. Explicit COI disclosure and the planned open release of the dataset are strengths. If annotation quality for long-tail scripts and sparse structures can be substantiated, MORE would become a lasting community resource and a practical diagnostic tool for multilingual document intelligence.","major_comments":[{"comment":"Section 3.2 and Table 3: The central claim that MORE supplies 'reliable' gold-standard baselines rests on a model-assisted, human-refined pipeline whose pre-annotation ensemble includes HunyuanOCR, PaddleOCR-VL, dots.ocr and Qwen-VL—the same families later ranked in Tables 4–12. No inter-annotator agreement, residual-error audit, or per-script expert qualification is reported. For rare scripts and for the sparse structural classes, residual label noise that systematically favors the pre-annotators cannot be ruled out; this is load-bearing for the ranking and bottleneck diagnoses.","section":"§3.2, Table 3"},{"comment":"Table 3 and §5.3–5.4: Structural element counts are extremely small (82 formulas, 94 tables, 73 code blocks, 104 catalogs across all 149 languages). Many languages contribute zero samples to the very tasks used to claim 'structural complexity' and table bottlenecks. Task-wise averages over such sparse sets are statistically fragile; language-level claims (e.g., Tables 7–10) and the assertion that tables remain the primary bottleneck for long-tail scripts therefore rest on thin evidence and need either denser sampling or explicit uncertainty quantification (confidence intervals / leave-one-out stability).","section":"Table 3, §5.3–5.4"},{"comment":"§4 Overall Score (Eq. 7): The final score is the unweighted arithmetic mean of six tasks whose sample sizes differ by two orders of magnitude (8 221 text vs. 73 code). This equal weighting can let a handful of code/catalog/table pages dominate the ranking relative to the far larger text corpus. The paper should either justify the equal-task design with a sensitivity analysis or report sample-size-weighted and task-stratified aggregates so that readers can assess robustness.","section":"§4, Eq. (7)"}],"minor_comments":[{"comment":"Figure 6 label '(e). Catelogue' is misspelled; also 'Catelogue' appears inconsistently with 'Catalog' elsewhere.","section":"Figure 6"},{"comment":"Several languages (e.g., Amharic, Tibetan, Burmese, Sinhala) have zero paragraph annotations (Tables 13–14). Clarify how overall and text scores are defined when a language contributes only layout/order or empty content.","section":"Tables 13–14, Appendix B"},{"comment":"Figure 1 (winner-takes-all by script) is informative but the color legend and outer-ring script grouping are hard to parse at print scale; a supplementary tabular breakdown would help.","section":"Figure 1"},{"comment":"The layout-dependent protocol is described as following OmniDocBench’s 'quick match'; a short formal definition or pseudocode in Appendix A would improve reproducibility.","section":"§5.4"},{"comment":"Impact Statement acknowledges sparsity of structural elements for rare languages; this limitation should also be stated more prominently in the main-text conclusion rather than only in the impact paragraph.","section":"§6 / Impact Statement"}],"recommendation":"major_revision","confidential_remarks":"All authors are employed by Tencent, developer of HunyuanOCR, which ranks first on the primary (decoupled) leaderboard. The COI is disclosed, but the pre-annotation ensemble also includes HunyuanOCR (and other evaluated models). An independent re-annotation audit on a stratified subset, or at least reported IAA, would substantially strengthen credibility. Scope-wise the paper is a solid systems/benchmark contribution for a CV/ML venue; the linguistic scale is real and valuable even if structural density remains limited."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The new thing here is a real-world PDF benchmark that actually covers 149 languages and six element types (text, formula, table, code, catalog, reading order), with both decoupled and layout-dependent scores. That scale and the structural extensions go past XDocParse (126 languages, no open images) and the English/Chinese-centric OmniDocBench line. They ship the numbers, the stratified sampling story, and multi-model tables that make the long-tail and table bottlenecks visible. Metrics are the usual NED/TEDS/CDM, clearly defined, and they report both task-wise and page-wise averages plus the end-to-end layout-dependent setting. That is solid engineering for the document-AI crowd.\n\nWhat they do well: authentic pages (no synthetic layouts), explicit COI disclosure on HunyuanOCR, and enough language/script breakdowns that you can see where models actually fall off (Sanskrit, tables, some Arabic/Other scripts). The winner-takes-all map and the script-family tables are useful diagnostics, not just leaderboard padding.\n\nSoft spots, in proportion: structural counts are tiny (73 code, 82 formula, 94 table, 104 catalog total). For many long-tail languages the “gold” for the very elements used to claim structural bottlenecks rests on a handful of samples. The annotation pipeline is model-assisted with an ensemble that includes the authors’ own model family; humans refine or discard, but there is no IAA, no residual-error audit, and no per-script expert qualification. That is a real but not fatal reliability gap for the strongest ranking claims. Sampling (≤10 PDFs per language, one page) is transparent; free parameters exist but are not hidden. Citation pattern is normal for the area.\n\nThis is for people who build or evaluate multilingual document parsers and VLMs. It deserves a serious referee. Once the GitHub dump is public, third parties can re-score and stress the labels. I would engage with it, cite the coverage numbers, and treat the sparse-structure rankings with the caution the sample sizes warrant.","headline":"Useful 149-language real-PDF document-parsing benchmark with standard metrics and clear baselines; annotation reliability for sparse long-tail structures is the main soft spot, not a collapse of the contribution.","tokens_in":44496,"tokens_out":539,"would_cite":true,"duration_ms":6851,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"MORE is a 149-language document-parsing benchmark that shows models still fail on tables and rare scripts.","keywords":["multilingual document parsing","vision-language models","OCR benchmark","layout analysis","table recognition","long-tail languages","reading order","structural elements"],"falsifier":"Independent re-annotation of a stratified sample of rare-script pages and tables by native speakers who never saw the original model candidates; large systematic disagreements with the released ground truth would undermine the baselines.","tokens_in":44512,"feed_emoji":"📄","tokens_out":780,"duration_ms":6870,"temperature":0.7,"pith_summary":"Modern vision-language models claim they can parse documents in hundreds of languages, yet almost every public test set covers only English, Chinese, and a handful of others. MORE fills that gap with real pages drawn from the web, spanning 149 languages and six script families, and scoring not only plain text but also formulas, tables, code blocks, catalogs, and reading order. The authors build the ground truth with a multi-model pre-annotation step followed by human correction, then run leading specialized and general models on the set. The resulting scores show that character-level recognition is already strong for high-resource scripts, while table structure and long-tail scripts remain clear bottlenecks. The benchmark therefore supplies the first quantitative map of where multilingual document understanding actually works and where it still breaks.","feed_headline":"149-language OCR test shows tables still break models","feed_subtitle":"Real pages and structural scores expose where multilingual document parsers still fail","key_machinery":"A model-assisted, human-refined annotation pipeline that turns real-world PDF pages into gold-standard labels for six tasks (text, formula, table, code, catalog, reading order), scored by normalized edit distance, CDM, and TEDS and then averaged into both task-wise and layout-dependent overall scores.","core_discovery":"MORE is the most linguistically diverse document-parsing benchmark yet (149 languages). Its decoupled and layout-dependent metrics on authentic pages establish reliable baselines that expose remaining failures on structural elements (especially tables) and long-tail scripts, even for models that claim broad language coverage.","pith_inferences":["Because the structural-element counts are low for rare languages, absolute scores on tables and code for those languages will remain noisy until the dataset is expanded.","Models whose pre-training or fine-tuning data overlap the pre-annotation ensemble may inherit a quiet advantage that only independent annotation can reveal.","The same real-page collection method could be extended to multi-page documents and to historical scripts, turning MORE into a living test suite for global document intelligence."],"forward_implications":["Researchers can now measure claimed multilingual OCR support on a common 149-language yardstick rather than relying on marketing statements.","Table parsing and complex-layout detection are identified as the primary remaining bottlenecks, so future model work can target them directly.","Long-tail scripts receive explicit numerical baselines, making progress on under-represented languages trackable.","The dual decoupled / layout-dependent scoring protocol can be reused by later benchmarks that want both recognition purity and end-to-end realism."],"fun_headline_variants":["149-language test shows tables still break doc parsers","MORE benchmark: 149 tongues expose table parsing fails","149-lang doc parsing test finds tables still break models","Real pages in 149 languages reveal table layout failures","MORE: 149-language baselines flag table and script gaps"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"The claim rests on the assumption that the human-refined labels produced by the multi-model pipeline are accurate gold standards for all 149 languages and for the sparse structural elements.","fun_headline_variants_meta":{"raw":{"variants":["149-language test shows tables still break doc parsers","MORE benchmark: 149 tongues expose table parsing fails","149-lang doc parsing test finds tables still break models","Real pages in 149 languages reveal table layout failures","MORE: 149-language baselines flag table and script gaps"]},"model":"grok-4.5","effort":"low","cost_usd":0.002954,"raw_usage":{"total_tokens":1057,"prompt_tokens":751,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":29540000,"prompt_tokens_details":{"text_tokens":751,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":244,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":751,"tokens_out":62,"duration_ms":2766,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T05:50:10.368551+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Independent re-annotation of a stratified sample of rare-script pages and tables by native speakers who never saw the original model candidates; large systematic disagreements with the released ground truth would undermine the baselines.","supporting_citations":[],"review_version":1}