Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a 2B vision-language model on 50,000 synthetic diacritized Arabic pages yields an open-source state-of-the-art OCR system, with QARI v0.2 at 0.160 WER, 0.061 CER, and 0.737 BLEU.

desk verdict Useful released models and an honestly reported SARD comparison, but the headline SOTA claim rests on a private test set the model was selected on—the abstract numbers should not be taken at face value. read the letter →

arxiv 2506.02295 v1 pith:NA3TE5TB submitted 2025-06-02 cs.CV cs.AI

Ahmed Wasfy , Omer Nacar , Abdelakreem Elkhateb , Mahmoud Reda , Omar Elshehy , Adel Ammar , Wadii Boulila This is my paper
classification cs.CVcs.AI
keywords ArabicOCRtashkeeldiacriticsvision-languagemodelsyntheticdatadocumentunderstandinghandwritingrecognitionmultimodalfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to show that a general-purpose vision-language model can be turned into a high-fidelity Arabic OCR engine by fine-tuning it on synthetic pages that carry full diacritics (tashkeel). The authors report that their middle model, QARI v0.2, reaches a 0.160 word error rate, 0.061 character error rate, and 0.737 BLEU score on a 200-page benchmark of vocalized traditional Arabic print, beating every open baseline and an API-based commercial OCR they tested. That result matters because diacritical marks change pronunciation and meaning in Arabic, and existing OCR systems handle them poorly; a reliable open model would ease digitization of classical and religious texts and downstream retrieval and translation. The paper also documents a layout-focused third model, QARI v0.3, that qualitatively recovers document structure and handles low-resolution and handwritten input, and a quantization study showing 8-bit precision preserves accuracy while 4-bit collapses it.

What carries the argument

The load-bearing mechanism is a synthetic data curriculum: a pipeline that renders Arabic sentences from news and classical corpora through a set of fonts at varied sizes, with and without full tashkeel, under clean, moderate, and heavy degradation, and pairs every image with its exact transcription. The v0.2 training set, 50,000 such images with diacritics and a broader font set, is the specific intervention that separates the state-of-the-art QARI v0.2 from the weaker no-diacritics v0.1. The argument therefore rests on this data distribution being rich enough that a 2B vision-language model, fine-tuned for one epoch with low-rank adapters, transfers from synthetic renderings to scanned printed pages.

What would settle it

Take the released QARI v0.2 weights and transcribe a held-out collection of vocalized Arabic books whose fonts and source texts were not used in Dataset v0.2; if character and word error rates rise toward 0.2 and 0.4 or higher while the API OCR the paper compares against stays near its reported 0.01-0.05 CER range, the claimed state of the art is an in-distribution artifact.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that the bottleneck for Arabic OCR is less the model architecture than the training data: a 2-billion-parameter vision-language backbone, after a single epoch of parameter-efficient fine-tuning on 50,000 synthetic diacritized page images, reads vocalized classical Arabic print more accurately than the specialized and larger systems the authors compared. QARI v0.2 scores 0.061 CER, 0.160 WER, and 0.737 BLEU on the paper's 200-page test set, ahead of all baselines including the API-based commercial OCR on all three metrics there. On an external five-font benchmark, the same API OCR records lower character and word errors on every font, while QARI v0.2 wins BLEU on several fonts, so the paper's state-of-the-art claim is tied to its diacritic-heavy benchmark. The v0.3 model, trained on a smaller layout-focused set, shows qualitative ability to reproduce HTML-like structure and to transcribe low-resolution and handwritten samples, but its average text metrics are worse than v0.2.

Load-bearing premise

The headline error rates assume the 200 test pages are independent of the synthetic training distribution and representative of general vocalized Arabic print; the paper does not report whether those pages overlap the classical corpus or fonts used to build Dataset v0.2, and no separate validation set is used before selecting v0.2 as final.

Editorial extensions

If this is right

  • Fully vocalized Arabic print becomes a practical OCR target for open tools, reducing the need for manual retyping of classical and religious texts.
  • The released synthetic pipeline and models give others a reproducible recipe for diacritic-heavy scripts beyond Arabic, such as Urdu and Persian.
  • Because 8-bit quantization retains most of the accuracy while 4-bit does not, a deployment path exists on a single GPU at 8-bit precision rather than requiring full-precision infrastructure.
  • The v0.3 results suggest layout structure can be recovered as markup from images, though the current evidence for that capability is qualitative.
  • The paper's benchmark and released evaluation setup provide a common yardstick for measuring diacritic-aware Arabic OCR, which was previously lacking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The state-of-the-art label should be read as specific to the paper's diacritic-heavy benchmark: on the external five-font set, the API OCR the authors compare against has lower CER and WER on every font, so the general Arabic OCR crown is not established.
  • Because the paper does not report a separate validation split or state whether the test pages overlap the classical corpus used to synthesize Dataset v0.2, the headline numbers could overstate generalization; a strict train/test separation with held-out fonts and source texts would settle this.
  • The 4-bit collapse suggests post-hoc quantization is not viable for this task, and quantization-aware fine-tuning may be needed before the model can run at reduced memory.
  • The qualitative HTML-reconstruction results point toward a measurable task: with a tree-edit-distance metric over predicted versus ground-truth markup, the layout capability could be turned into a quantitative claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents Qari-OCR, a family of vision-language models obtained by fine-tuning Qwen2-VL-2B-Instruct on three progressively more complex synthetic Arabic datasets (v0.1, v0.2, v0.3). The central claim is that QARI v0.2 establishes a new open-source state of the art on diacritically rich Arabic print, with WER 0.160, CER 0.061, and BLEU 0.737 on a private 200-page test set. The authors also report SARD benchmark results, a quantization analysis, and qualitative examples for layout and handwriting. The paper states that models and datasets are released.

Significance. If the headline numbers were supported by an independent, public benchmark, Qari-OCR would be a practically useful contribution to Arabic OCR, particularly for vocalized text, and the paper would provide a reproducible recipe for adapting small multimodal LLMs to a low-resource script. The paper also deserves credit for comparing several baselines, including Tesseract, EasyOCR, Mistral OCR, AIN, and Qwen variants, and for attempting a public SARD evaluation. However, the current evidence does not establish the claimed state of the art: the private test set is not shown to be independent of the synthetic training distribution, the model was selected on that same set, and the public SARD results show Mistral OCR substantially outperforming QARI v0.2 on most fonts. The central claim therefore needs substantial re-evaluation before it can be accepted.

major comments (5)
  1. [§4.1, Table 3] The headline SOTA claim is not supported because the private 200-page test set is not shown to be independent of the training data. Section 3.1 states that Dataset v0.2 is generated from a modern news corpus and a classical Islamic corpus; Section 4.1 describes the test set as 'traditional Arabic print' with diacritics and dense layouts but never states whether any test pages, fonts, or rendering settings overlap with the synthetic training pipeline. In addition, QARI v0.2 was selected as the final model after comparing three versions on this same 200-page set, and no separate validation split is described. The reported CER of 0.061 could therefore reflect selection on the test set and near-duplicate rendering from the same source corpus. The authors must either release the test set and demonstrate non-overlap, or re-evaluate on an independently collected public benchmark.
  2. [§4.3, Table 4] The public SARD results directly undermine the abstract's claim of 'superior handling of tashkeel, diverse fonts' and the 'open-source state-of-the-art' framing. On SARD, QARI v0.2 has CER 0.200 on Amiri, 0.230 on Arial, 0.193-0.216 on Calibri, Sakkal Majalla, and Scheherazade, whereas Mistral OCR achieves CER 0.011-0.051 on the same five fonts. A model that is 3-4 times worse than a general API on clean printed fonts cannot be described as superior on diverse typography without a clear explanation of why the private 200-page set is more representative of the claimed use case. Please provide a direct comparison that acknowledges and contextualizes this discrepancy.
  3. [§3.2, Table 5] The quantization analysis is inconsistent with the main results and with the training description. Section 3.2 and Figure 1 describe training with 4-bit QLoRA, but Table 5 reports that 4-bit QARI v0.2 collapses to CER 3.452, WER 4.516, and BLEU 0.001, while the 8-bit version gives CER 0.091, WER 0.255, and BLEU 0.583. Neither set of numbers matches Table 3's CER 0.061, WER 0.160, and BLEU 0.737. The paper must specify exactly which precision (training and inference) produced the Table 3 numbers, and explain the discrepancy with Table 5; otherwise the headline results are not reproducible.
  4. [§4.2] The evaluation protocol is under-specified. The paper does not define how CER and WER are computed for Arabic text (e.g., whether diacritics, tatweel, and punctuation are normalized before alignment), nor how BLEU is computed over full pages (tokenization, smoothing, and whether it is sentence-level or corpus-level). These choices can change CER and BLEU by large margins for vocalized Arabic, and the claimed 0.061 CER cannot be verified without this information. Please provide the exact metric implementation and release the evaluation script.
  5. [§4.1, §6] Qalam, a specialized Arabic OCR and handwriting model cited in Section 2, is not included in any quantitative comparison. Given that Qalam is a purpose-built Arabic OCR system and the paper claims a new open-source state of the art, the absence of a Qalam baseline is a significant omission. The authors should either add this comparison or justify its absence explicitly.
minor comments (6)
  1. [Table 3] QARI v0.1 reports CER 1.915 and WER 2.025, which exceed 1.0; if insertions are counted in the Levenshtein normalization, this should be stated explicitly so the reader understands the metric's range.
  2. [Table 4] There is a typo in the text: 'Sakkak' should be 'Sakkal', and 'BLUE' on page 8 should be 'BLEU'.
  3. [Figure 6] The labels in Figure 6 are inconsistent with the model names in the rest of the paper ('QARI 3' appears alongside 'QARI v0.1' and 'QARI v0.2'); please use consistent version names.
  4. [§4.3.1] The sentence 'employing 8-bit quantization during fine-tuning or inference' is ambiguous; please state separately whether the reported models were fine-tuned in 8-bit then inferred in 8-bit, or fine-tuned in 4-bit and inferred in 8-bit.
  5. [§1, §7] The paper says all models and datasets are publicly released, but no Hugging Face link or repository reference appears in the manuscript; please add the exact URLs so the release claim is verifiable.
  6. [Section 6] The limitations section is honest about several weaknesses, but it does not mention the benchmark-independence issue or the lack of an external validation set; these are the most consequential limitations for the paper's central claim and should be addressed in the text.

Circularity Check

1 steps flagged · score 2.0 of 10

No equation-level circularity; the headline SOTA number is, however, selected on the same private 200-page test set used to label QARI v0.2 the leading model, so the benchmark is partly self-referential.

  1. fitted input called prediction [Abstract; Section 4.1 ('Experimental Setup'); Section 4.3 ('Results', Table 3); Section 5 ('Discussion')]
    "Abstract: 'Our leading model, QARI v0.2, establishes a new open-source state-of-the-art with a Word Error Rate (WER) of 0.160, Character Error Rate (CER) of 0.061, and BLEU score of 0.737...' Section 4.3: 'As shown in Table 3, QARI v0.2 significantly outperforms all other open source models evaluated, establishing a new benchmark on our test set with a CER of 0.061, a WER of 0.160, and a BLEU score of 0.737.'"

    The 'leading model' label is assigned after inspecting the same test-set scores that define the claimed state of the art: Section 3.2 says 'We conducted three matched fine-tuning runs, each on a different synthetic dataset version,' and no separate validation set is described before v0.2 is declared the winner in Table 3. The headline result is therefore 'the version best on this benchmark scores best on this benchmark,' i.e., model choice and evaluation are selected on the same private 200-page set. The paper also never states whether those test pages are disjoint from the classical Islamic corpus used to build Dataset v0.2 (Section 3.1), so the 0.061 CER may reflect near-training-distribution recognition rather than an external prediction.

full rationale

The derivation chain contains no equation-level circularity: OCR outputs are measured against ground truth with standard CER/WER/BLEU metrics, and no model parameter is predicted from the metric it evaluates. There are no load-bearing self-citations or imported uniqueness theorems. The only self-referential element is the benchmark itself: v0.2 is called the leading model because it won on the authors' own 200-page test set, and the same numbers are then presented as open-source state of the art. Independence of the test set from the synthetic training distribution is not documented, and the external SARD results (Table 4) show Mistral OCR outperforming QARI v0.2 on CER and WER for all five fonts, so the abstract's SOTA claim rests entirely on the private, selection-contaminated benchmark. This is a real evaluation weakness but not a derivation that reduces to its inputs; hence the low score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by training choices and transfer assumptions. The headline numbers depend on the authors' private test set and the choice of v0.2 after evaluation, which are not external benchmarks. No new physical or conceptual entities are introduced.

free parameters (5)
  • LoRA rank = 16
    Adapter rank chosen by hand; affects capacity and final CER/WER.
  • Learning rate = 2e-4
    Standard AdamW value, not tuned; affects convergence and final accuracy.
  • Number of fine-tuning epochs = 1
    Single epoch; more epochs may change results.
  • Dataset v0.2 size = 50,000 synthetic images
    Scale of synthetic training data; larger or smaller corpora would shift metrics.
  • Final model selection (v0.2 vs v0.3) = v0.2
    Version chosen after seeing test-set numbers; this selection uses the test set as a model-selection criterion.
assumptions (5)
  • domain assumption WeasyPrint and pdf2image render synthetic HTML pages into images that resemble real scanned Arabic documents.
    The synthetic-to-real transfer is assumed, not validated against a public corpus beyond the private 200-page test.
  • domain assumption Qwen2-VL-2B-Instruct provides a sufficient visual-language backbone for Arabic OCR after lightweight fine-tuning.
    The paper does not compare against full fine-tuning or other backbones.
  • domain assumption The 200-page test set ground truth is accurate and the pages are representative of general diacritically rich Arabic print.
    Test set construction is described only as 'traditional Arabic print'; no annotation procedure or inter-rater checks are given.
  • domain assumption BLEU is a meaningful fidelity metric for OCR output.
    BLEU is borrowed from machine translation and is not standard for OCR; it rewards n-gram overlap and can mask diacritic errors.
  • domain assumption 4-bit QLORA training followed by 8-bit inference preserves the quality of full fine-tuning.
    The paper does not show a full-precision baseline; Table 5 shows large sensitivity to quantization, so the assumption is fragile.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation." pith.science (2026). https://pith.science/paper/NA3TE5TB

@misc{pith2026250602295,
  author       = {Pith},
  title        = {Pith review of: QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NA3TE5TB}},
  note         = {Machine review of arXiv:2506.02295}
}
read the original abstract

The inherent complexities of Arabic script; its cursive nature, diacritical marks (tashkeel), and varied typography, pose persistent challenges for Optical Character Recognition (OCR). We present Qari-OCR, a series of vision-language models derived from Qwen2-VL-2B-Instruct, progressively optimized for Arabic through iterative fine-tuning on specialized synthetic datasets. Our leading model, QARI v0.2, establishes a new open-source state-of-the-art with a Word Error Rate (WER) of 0.160, Character Error Rate (CER) of 0.061, and BLEU score of 0.737 on diacritically-rich texts. Qari-OCR demonstrates superior handling of tashkeel, diverse fonts, and document layouts, alongside impressive performance on low-resolution images. Further explorations (QARI v0.3) showcase strong potential for structural document understanding and handwritten text. This work delivers a marked improvement in Arabic OCR accuracy and efficiency, with all models and datasets released to foster further research.

Figures

Figures reproduced from arXiv: 2506.02295 by the authors.

Figure 1
Figure 1. Qari-OCR Dataset Generation and Model Training Pipeline [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Qualitative example demonstrating Qari-OCR’s handling of various Arabic script complexities. The input [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Example of Qari-OCR (v0.3) accurately transcribing Arabic text from a low-resolution and tightly cropped [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qari-OCR v0.3 successfully transcribing handwritten Arabic text, maintaining sentence struc￾ture, punctuation, and recognizing itemized formatting. (  @) and crucial letter-distinguishing dots. The corresponding output from our Qari-OCR model (right panel of [PITH_FU…
Figure 5
Figure 5. Figure 5: Qualitative comparison of QARI v0.2 and QARI v0.3 outputs against Input and Ground Truth for various [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison of estimated resource consump [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Expert Routing for Multi-Domain Low-Resource OCR: A Manchu Case Study

    cs.CV 2026-07 conditional novelty 4.0 of 10

    A style classifier routing Manchu pages to checkpoints from a fine-tuning history matches the per-domain best CERs (0.30/1.57/4.83%), but the specialists were selected on the same frozen test sets.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    I Saleh Al-Sheikh, MASNIZAH Mohd, and L Warlina. 2020. A review of arabic text recognition dataset. Asia-Pacific J. Inf. Technol. Multimedia, 9(1):69--81

    work page 2020

  2. [2]

    Naseem Alrobah and Saleh Albahli. 2022. Arabic handwritten recognition using deep learning: A survey. Arabian Journal for Science and Engineering, 47(8):9943--9963

    work page 2022

  3. [3]

    Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of arabic multimodal large language models and benchmarks. arXiv preprint arXiv:2403.01031

    work page Pith review arXiv 2024

  4. [4]

    Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih, and Muhammad Abdul-Mageed. 2024. Qalam: A multimodal llm for arabic optical character and handwriting recognition. arXiv preprint arXiv:2407.13559

    arXiv 2024

  5. [5]

    Alex Graves and J \"u rgen Schmidhuber. 2008. Offline handwriting recognition with multidimensional recurrent neural networks. Advances in neural information processing systems, 21

    work page 2008

  6. [6]

    Ahmed Heakl, Sara Ghaboura, Omkar Thawkar, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, and Salman Khan. 2025. Ain: The arabic inclusive large multimodal model. arXiv preprint arXiv:2502.00094

    work page Pith review arXiv 2025

  7. [7]

    Mahdi Nsaif Jasim. 2020. Arabic optical characters recognition by neural network based arabic unicode

    work page 2020

  8. [8]

    Dietrich Klakow and Jochen Peters. 2002. Testing the correlation of word error rate and perplexity. Speech Communication, 38(1-2):19--28

    work page 2002

Show all 20 references
  1. [9]

    Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2023. Trocr: Transformer-based optical character recognition with pre-trained models. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pa...

    2023

  2. [10]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  3. [11]

    Mistral AI Team . 2025. Mistral ocr: Introducing the world’s best document understanding api. https://mistral.ai/news/mistral-ocr. Research, March 6, 2025

    2025

  4. [12]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  5. [13]

    Binod Kumar Pattanayak, Anil Kumar Biswal, Suprava Ranjan Laha, Saumendra Pattnaik, Bibhuti Bhusan Dash, and Sudhansu Shekhar Patra. 2023. A novel technique for handwritten text recognition using easy ocr. In 2023 International Conference on Self Sustainable Artificial Intelli...

    2023

  6. [14]

    Joan Puigcerver. 2017. Are multidimensional recurrent layers really necessary for handwritten text recognition? In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 67--72. IEEE

    2017

  7. [15]

    Ray Smith. 2007. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629--633. IEEE

    2007

  8. [16]

    UNESCO . 2024. https://www.unesco.org/en/world-arabic-language-day#: :text=The UNESCO Official Website. The Arabic language is a pillar of the cultural diversity of humanity. It is one of the most widely spoken languages in the world, used daily by more than 400 million people

    2024

  9. [17]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  10. [18]

    Mohamed Yousef, Khaled F Hussain, and Usama S Mohammed. 2020. Accurate, data-efficient, unconstrained text recognition with convolutional neural networks. Pattern Recognition, 108:107482

    2020

  11. [19]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  12. [20]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.