{"id":"cf7490ec-c3f2-447b-aca2-e90de2b036e2","arxiv_id":"2411.17835","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Arabic-specific fine-tunes of Nougat, trained on a synthetic Hindawi-derived dataset, report high in-distribution BLEU and structure accuracy for Arabic book page OCR.","lead":"This paper fine-tunes Meta's Nougat vision-language model on 13.7k synthetic Arabic book pages, producing three models that output structured Markdown from Arabic book images. It matters because it adds open Arabic OCR resources, including a 1.1 billion token corpus, to a domain dominated by Latin-script tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is tested only on 160 pages from the same synthetic arabic-img2md pipeline used for training, and the headline Structure Accuracy metric has no published definition; an independent real-book test is required before the claim can stand.","rationale":"The paper's core value would be a model that converts unseen Arabic book pages to Markdown with CER 6.6% and MSA 98.8%. The evidence for this is a synthetic test set drawn from the same generator used to make training data. That design cannot establish the deployment claim, because the test set does not measure domain shift; it measures near-distribution fit. This is the same weak spot the Pith reader identified, and I agree with CONDITIONAL. I do not accuse the authors of fabrication; in fact, the release of models and dataset is a real strength and makes a decisive independent test possible. The undefined 'Structure Accuracy' metric and the discrepancy between Table 1 and Section 5 (98.84% vs. 94.7%; 6.62% vs. 6.1%) mean the exact numbers in the abstract should be treated with caution until the metric and test procedure are pinned down. If the external test confirms comparable CER/WER on real scans, the conditional can be lifted. If not, the SOTA claim should be REJECTed. Thus UNCHANGED relative to the reader's CONDITIONAL verdict.","tokens_in":6651,"tokens_out":3475,"duration_ms":33799,"concrete_test":"Take 100 real Arabic book page scans from a source independent of Hindawi's HTML pipeline (e.g., scanned printed books from the Internet Archive or the Arabic subset of a public OCR benchmark), have annotators produce Markdown ground truth, run arabic-large-nougat, and compare CER/WER and a publicly defined MSA. If CER is more than about twice the reported 6.62% or MSA falls below 90%, the central SOTA claim fails. Also publish the Structure Accuracy computation code so that the reported 98.84% is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 evaluates on '160 random, unseen Arabic book pages from arabic-img2md,' and Section 3.3 describes arabic-img2md as generated from Hindawi HTML converted to PDFs with Markdown extracted from the same HTML. Test pages therefore share the exact rendering pipeline, font stack, and document templates used to create the training pairs. A model can memorize layout patterns and markup regularities without generalizing to scanned Arabic books; Section 6 itself admits that the dataset 'may not generalize well to other domains.' The headline metric 'Structure Accuracy' is called custom but no formula is provided, so the 98.84% value cannot be independently reproduced. In addition, Section 5 reports 94.7% MSA and 6.1% CER for arabic-large-nougat, contradicting Table 1's 98.84% and 6.62%; even the paper's own numbers are not stable. These issues mean the abstract's SOTA claim is currently unsupported, though the released artifacts make external evaluation feasible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Arabic-Nougat, a suite of three fine-tuned variants of Meta's Nougat model (arabic-small-nougat, arabic-base-nougat, arabic-large-nougat) for Arabic OCR and Markdown extraction. The models are trained on a synthetic dataset, arabic-img2md, of 13.7k Arabic book page / Markdown pairs, using the Aranizer-PBE-86k tokenizer, bfloat16 precision, and Flash Attention 2. The paper reports BLEU, CER, WER, and a custom Markdown Structure Accuracy (MSA) on a test set of 160 Arabic book pages, claiming that arabic-large-nougat achieves the highest MSA and lowest CER, and releases a 1.1-billion-token Arabic corpus (arabic-books) extracted with the best model.","tokens_in":6879,"tokens_out":3663,"duration_ms":35747,"significance":"If the reported results held on real Arabic documents, this would be a useful and timely contribution: Arabic book OCR with structured Markdown output is underserved, and the open-sourced models, tokenizer, and corpus would be concrete assets. The use of Aranizer-PBE-86k to extend effective context length and the bfloat16/Flash Attention 2 training choices are sensible engineering contributions. However, the central 'state-of-the-art' claim is not yet supported. The evaluation set is drawn from the same synthetic generation pipeline that produced the training data, the primary MSA metric is not defined, the baseline set contains only non-Arabic Meta Nougat models, and the paper's own Section 5 numbers contradict Table 1. These are load-bearing issues that require additional experiments and a corrected, externally validated evaluation before the headline claim can be accepted.","major_comments":[{"comment":"The test set of 160 'unseen' pages is sampled from arabic-img2md, the same synthetic pipeline (scraping Hindawi HTML, converting to PDFs, extracting Markdown) that generated the training data. Reported BLEU, CER, WER, and Structure Accuracy are therefore same-distribution estimates, not evidence of generalization to scanned Arabic books or other domains. Section 6 itself acknowledges that the dataset 'may not generalize well to other domains.' The central claim requires an independent evaluation on real, scanned Arabic book pages, ideally with genre/layout diversity, and a comparison of error patterns beyond aggregate scores.","section":"§4.1 / §3.3 / §6"},{"comment":"The paper contains an internal numerical contradiction. Table 1 reports arabic-large-nougat with Structure Accuracy 0.9884 and CER 0.0662, while Section 5 states that arabic-large-nougat achieves the highest Markdown Structure Accuracy (94.7%) and lowest CER (6.1%). These numbers are not consistent. The authors must reconcile the two sets of values and state which corresponds to the released model checkpoints.","section":"§4.2 / Table 1 / §5"},{"comment":"Markdown Structure Accuracy (MSA) is described only as 'a custom metric' that evaluates structural elements such as headers and lists. No formula, matching algorithm, or implementation is given. Since MSA is the headline metric supporting the state-of-the-art claim, the paper must specify exactly how structure tokens are extracted, how predicted and reference Markdown are aligned, and how partial credit is assigned, and should release the evaluation script.","section":"§4.3"},{"comment":"The 'state-of-the-art' claim is not supported by the chosen baselines. Meta's Nougat Small and Nougat Base were trained on English scientific documents and are not Arabic-capable OCR systems; their near-zero BLEU and CER values above 1.0 only show that a non-Arabic model fails on Arabic text. A SOTA claim requires comparison with state-of-the-art Arabic OCR or document-parsing systems, for example fine-tuned TrOCR, PaddleOCR, Tesseract with Arabic language data, or recent multimodal document-understanding models trained or adapted for Arabic.","section":"§4.2 / Abstract"}],"minor_comments":[{"comment":"The effective context length is presented as both '8192 tokens' (Section 3.5) and '32k tokens' (Introduction and Section 3.5). The relationship between raw decoder length and the tokenizer-compression-based 'equivalent' length should be stated explicitly to avoid confusion.","section":"§3.5 / Introduction"},{"comment":"Token Efficiency Ratio (TER) is defined as 'the ratio of tokens produced by the tokenizer to ground truth tokens,' but the paper does not state whether lower or higher values are better. As reported, TER 1.05 for arabic-large-nougat and 1.25 for arabic-small-nougat appears to be lower-is-better; please clarify.","section":"§4.4"},{"comment":"The manuscript mixes decimal fractions (e.g., 0.0662) and percentages (e.g., 6.1%) for the same metric types. Please choose one convention and apply it consistently in the table and text.","section":"§4.2 / Table 1, Section 5"},{"comment":"References [24] and [25] both list arXiv:2409.03420; the second citation appears to be a different paper and the arXiv identifier should be corrected.","section":"References"},{"comment":"The text inside Figure 1 appears garbled ('قد م# BOS ة ق\\n\\n'), which makes the figure hard to interpret. A clean example of an input page and its tokenized output would be more informative.","section":"Figure 1"},{"comment":"BCE-Arabic-v1 is mentioned without a reference. Please provide a citation or remove the mention if it is not used elsewhere.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's main value is likely as an open-source release (models, dataset, tokenizer) rather than as a rigorously established SOTA result. The current evaluation is internally consistent only in the sense that all data comes from the same synthetic pipeline; the abstract overstates the findings. Given that the authors have released artifacts, an external evaluation is feasible and should be required before publication. I would also encourage the editor to verify that the arabic-books corpus is accompanied by quality controls, since it is generated by the authors' own model without apparent human validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a workmanlike fine-tuning of Nougat for Arabic, and the three model checkpoints, the arabic-img2md training set, and the 1.1B-token arabic-books corpus are all genuinely useful releases. The actual SOTA claim, though, is not supported by the evaluation in the paper.\n\nWhat's new: the architecture and loss are borrowed entirely from Nougat/Donut, and the Aranizer tokenizer is external, so the new bits are the fine-tuned weights, the synthetic data pipeline (Hindawi HTML to PDF to Markdown), and the large corpus. Those are real, they're open-sourced, and they make independent testing feasible. Credit where it's due.\n\nSoft spots: the headline comparison only pits the models against Meta's Nougat checkpoints that were never trained on Arabic, which is a strawman. The 160-page test set is sampled from the same synthetic pipeline that generated the training pairs, so the numbers measure in-distribution fit, not generalization. The paper's own Section 6 admits the dataset 'may not generalize well to other domains.' Structure Accuracy is called a custom metric, but no formula is given, so the 98.84% is not independently reproducible. Worse, Section 5 reports 94.7% MSA and 6.1% CER for arabic-large-nougat, while Table 1 says 98.84% and 6.62%; the paper contradicts itself. That needs to be fixed regardless of anything else.\n\nThe reader's conditional verdict is fair. I'd add that the internal inconsistency is the most immediately damning issue, because it undermines confidence in the reported numbers even before external benchmarking.\n\nWho this is for: people working on Arabic document digitization who want a ready-made baseline and a large corpus. The artifacts justify a serious referee, but the paper needs major revision: define Structure Accuracy, retest on a genuinely external set of real scanned Arabic books, and reconcile the numbers. I'd engage with the artifacts, not the abstract.","headline":"Useful open-sourced Arabic OCR resource, but the SOTA claim rests on a same-distribution test set and an undefined metric; the artifacts, not the numbers, are the contribution.","tokens_in":7361,"tokens_out":2092,"would_cite":true,"duration_ms":20625,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Arabic-Nougat claims that fine-tuning the Nougat document-parsing transformer with an Arabic tokenizer and a synthetic paired dataset converts Arabic book pages to Markdown with 98.84% structure accuracy and 6.62% character error on the…","keywords":["Arabic OCR","Markdown extraction","document parsing","vision transformer","encoder-decoder","Aranizer tokenizer","synthetic dataset","Arabic books"],"falsifier":"Run arabic-large-nougat on a set of scanned Arabic books that were not produced by the synthetic pipeline, covering old print, varied fonts, tables, and marginal noise, and compare CER and structure accuracy against the reported 6.62% and 98.84%. If character error rises severalfold or structure accuracy falls well below 98%, the central claim of state-of-the-art transfer is not supported.","tokens_in":6454,"feed_emoji":"📖","tokens_out":8645,"duration_ms":76922,"temperature":0.7,"pith_summary":"This paper claims that a family of OCR models fine-tuned from the Nougat document-parsing architecture can turn images of Arabic book pages into structured Markdown text with high formatting accuracy. On a held-out set of 160 synthetic Arabic book pages, the largest model reports 98.84% Markdown Structure Accuracy and a 6.62% Character Error Rate, far outperforming the original Nougat models, which score below 1% BLEU on Arabic. The paper's case rests on three pieces working together: an Arabic-specific tokenizer (Aranizer-PBE-86k) that compresses Arabic text, a synthetic dataset (arabic-img2md) of 13.7k page-Markdown pairs scraped from publisher HTML, and reduced-precision training with Flash Attention 2 to allow longer sequences. A sympathetic reader would care because reliable Arabic document-to-Markdown conversion is a missing building block for digitizing Arabic literature and for generating training data.","feed_headline":"Arabic OCR model hits 98.8% Markdown structure accuracy","feed_subtitle":"Fine-tuning with an Arabic tokenizer cuts character error to about 6.6 percent on Arabic book pages.","key_machinery":"The apparatus that carries the argument is the Nougat-style encoder-decoder: a Donut vision encoder downsamples an 896×672 page image to a 588-token visual sequence, and an MBART decoder autoregressively emits Markdown. Onto this, the paper adds the Aranizer-PBE-86k tokenizer, whose 86k vocabulary packs roughly four base tokens into one Arabic token and extends the practical decoder context to 32k tokens, plus a synthetic pipeline that scrapes publisher HTML, renders pages to PDFs, and extracts Markdown to create 13.7k aligned image-text pairs.","core_discovery":"The central claim, on the paper's own terms, is that fine-tuning a visual document-parsing transformer on Arabic-specific data moves it from near-total failure on Arabic (BLEU 0.0037 and 0.0094 for the base models) to a usable OCR engine: the best model, arabic-large-nougat, reaches 0.6771 BLEU, 0.0662 CER, 0.1916 WER, and 0.9884 structure accuracy on the reported test set. The authors attribute the gain less to the vision backbone than to the Arabic tokenizer and the synthetic paired corpus, and they release the corpus, the trained models, and a 1.1-billion-token Arabic book corpus extracted by the best model as evidence that the approach scales.","pith_inferences":["The reported test numbers come from the same synthetic generation pipeline as the training data, so a real-book benchmark with scanned older print, marginal noise, and varied typesetting is the natural next test; if transfer is weak, the 98.8% figure is a pipeline-similarity score rather than a deployment number.","The 1.1-billion-token corpus was extracted by the model itself, so OCR errors and hallucinated Markdown artifacts may be baked into downstream NLP training; filtering or re-scoring this corpus could change how useful it is.","The tokenizer's compression suggests the same fine-tuning recipe might lift other Arabic document tasks, such as layout-aware translation, table extraction, or handwritten text recognition, but nothing in the paper verifies that transfer yet."],"forward_implications":["With arabic-large-nougat, a page of Arabic prose can be converted to Markdown at roughly 1-in-15 character error and better than 98% structure accuracy, which is enough to make bulk digitization of standard modern Arabic books feasible.","The 13.7k-pair arabic-img2md dataset and the 1.1-billion-token arabic-books corpus are open resources that let other teams fine-tune or build on the approach without collecting new paired data.","Because the Aranizer tokenizer compresses Arabic roughly 4:1 relative to the base Nougat tokenizer, sequence-length limits that constrained Latin-script models are less binding for Arabic, and the model can process longer pages.","For practical use, the paper recommends arabic-base-nougat for general pages and arabic-large-nougat for lengthy or complex layouts, with a repetition penalty above 1 to control the repetition the large models exhibit."],"supporting_citations":[{"why":"Defines the Nougat architecture that Arabic-Nougat fine-tunes and supplies the original models used as baselines.","marker":"[1]"},{"why":"Provides the Donut vision encoder that converts page images into 588 visual tokens.","marker":"[5]"},{"why":"Defines the MBART decoder that generates Markdown text autoregressively.","marker":"[2]"},{"why":"Supplies the Aranizer-PBE-86k Arabic tokenizer that drives the token-efficiency gains.","marker":"[12]"},{"why":"Supplies the arabic-img2md dataset of 13.7k Arabic page-Markdown pairs used for training and evaluation.","marker":"[13]"},{"why":"Is the publisher website whose HTML was scraped to generate the synthetic pairs.","marker":"[7]"},{"why":"Enables the memory-efficient Flash Attention 2 training that makes the 32k-token contexts practical.","marker":"[3]"}],"fun_headline_variants":["Arabic OCR: from 0.4% to 67.7% BLEU with fine-tuning","Nougat fine-tuned for Arabic hits 98.8% structure accuracy","Arabic tokenizer boosts OCR to 67.7% BLEU, 6.6% CER","Open-sourced Arabic OCR: 1.1B token corpus, 98.8% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy numbers assume that a held-out set of pages generated by the same synthetic HTML-to-PDF-to-Markdown pipeline as the training data is representative of real Arabic book pages.","fun_headline_variants_meta":{"raw":{"variants":["Arabic OCR: from 0.4% to 67.7% BLEU with fine-tuning","Nougat fine-tuned for Arabic hits 98.8% structure accuracy","Arabic tokenizer boosts OCR to 67.7% BLEU, 6.6% CER","Open-sourced Arabic OCR: 1.1B token corpus, 98.8% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000798,"raw_usage":{"total_tokens":3497,"prompt_tokens":921,"completion_tokens":2576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2475}},"tokens_in":537,"tokens_out":2576,"duration_ms":18701,"temperature":1.0,"reasoning_tokens":2475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:29:58.475187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run arabic-large-nougat on a set of scanned Arabic books that were not produced by the synthetic pipeline, covering old print, varied fonts, tables, and marginal noise, and compare CER and structure accuracy against the reported 6.62% and 98.84%. If character error rises severalfold or structure accuracy falls well below 98%, the central claim of state-of-the-art transfer is not supported.","supporting_citations":[{"cited_title":"Donut: Document Understand- ing Transformer without OCR,","cited_arxiv_id":null,"evidence_quote":"Provides the Donut vision encoder that converts page images into 588 visual tokens."},{"cited_title":"riotu-lab/Aranizer-PBE-86k · Hug- ging Face,","cited_arxiv_id":null,"evidence_quote":"Supplies the Aranizer-PBE-86k Arabic tokenizer that drives the token-efficiency gains."},{"cited_title":"MohamedRashad/arabic- img2md · Hugging Face,","cited_arxiv_id":null,"evidence_quote":"Supplies the arabic-img2md dataset of 13.7k Arabic page-Markdown pairs used for training and evaluation."},{"cited_title":"https://www","cited_arxiv_id":null,"evidence_quote":"Is the publisher website whose HTML was scraped to generate the synthetic pairs."}],"review_version":1}