Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Zero-shot OCR Accuracy of Low-Resourced Languages: A Comparative Analysis on Sinhala and Tamil

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Zero-shot OCR tests find Surya leads Sinhala and Document AI leads Tamil on clean synthetic images.

desk verdict A useful, honest benchmark with a new synthetic Tamil OCR dataset; the Sinhala result (Surya) is solid, but the Tamil 'Document AI wins' claim is a near-tie without error bars. read the letter →

arxiv 2507.18264 v2 pith:BWETAFIH submitted 2025-07-24 cs.CL

Nevidu Jayatilleke , Nisansa de Silva This is my paper · ORCID
classification cs.CL
keywords OCRzero-shotlearningSinhalaTamillow-resourcelanguagessyntheticdatasetbenchmarkingcharactererrorrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish which OCR engines, used with no fine-tuning, currently read Sinhala and Tamil printed text best, and to provide a public benchmark for measuring that. On synthetic clean-text images, Surya is the best zero-shot engine for Sinhala on every metric, with a word error rate of 2.61%. Document AI is the best for Tamil, with a character error rate of 0.78%. The study also introduces a new synthetic Tamil OCR dataset of 7,000 images built from subtitle text in six fonts. If correct, the results give practitioners a current performance ranking and a reusable dataset for a pair of scripts that remain open OCR problems.

What carries the argument

The engine of the comparison is a controlled synthetic benchmark: 6,969 Sinhala images from an existing five-font dataset and 7,000 newly generated Tamil images in six fonts, all black text on white, rendered from text filtered to at least 40 characters. Each engine is run zero-shot, meaning without any fine-tuning on the target languages, and scored with five metrics: CER and WER via Levenshtein distance, BLEU, ANLS (a thresholded Levenshtein similarity), and METEOR. The Tamil dataset pipeline, which draws text from a large parallel subtitle corpus, segments it by line, filters by length, randomly samples to match the Sinhala sample size, and renders the images, is what makes the cross-language comparison and the claimed fairness of the benchmark possible.

What would settle it

Run the same six engines zero-shot on a ground-truth corpus of real Sinhala and Tamil documents, such as scanned newspapers, book pages, and phone-captured print, and compare the five metrics; the central claim would fail if Surya is not the Sinhala leader on every metric, if Document AI is not the Tamil leader, or if the leading margins collapse.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a zero-shot performance ranking with a clear split by language: Surya outperforms the other tested engines on Sinhala across all five metrics (CER 0.76%, WER 2.61%, BLEU 0.9396, ANLS 0.9920, METEOR 0.9723), while Document AI outperforms the others on Tamil (CER 0.78%, WER 11.98%, BLEU 0.5803, ANLS 0.9923, METEOR 0.8762). The paper further reports that zero-shot Tesseract 5.5.0 beats the Sinhala-tuned Subasa OCR system on every metric, and that Tamil word-level performance lags character-level performance across engines, pointing to spacing and word formation as the weak spot. As a companion contribution, it publishes a synthetic Tamil dataset of 7,000 black-on-white images derived from subtitle text in six fonts, sized to match the existing Sinhala dataset.

Load-bearing premise

The rankings rest on the assumption that clean synthetic images of black text on white, generated from subtitle text, predict how the engines would perform on real scanned or photographed documents; the paper itself notes these images do not reflect real-world capture conditions.

Editorial extensions

If this is right

  • For clean synthetic Sinhala documents, Surya is the most accurate zero-shot choice on every character- and word-level measure tested.
  • For clean synthetic Tamil documents, Document AI is the strongest zero-shot option, though its very low character error rate does not translate into a proportionally low word error rate.
  • Zero-shot Tesseract 5.5.0 outperforms the Sinhala-tuned Subasa OCR system, indicating that current vanilla releases can close much of the gap with per-language fine-tuning for this script.
  • Tamil OCR's main bottleneck for the best engines appears to be word formation and spacing rather than character recognition.
  • The released synthetic Tamil dataset provides a public baseline that future Tamil OCR studies can use for direct comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • On degraded or camera-captured images the ranking is likely to shift: the Tamil gap between Document AI and Cloud Vision is tiny (CER 0.78% versus 0.79%), so either commercial engine could lead on real scans.
  • The character-error analysis suggests Sinhala diacritics, not base letters, dominate the remaining mistakes; a synthetic dataset that over-samples diacritic-heavy words would be a cheap way to stress-test engines.
  • Because the Tamil benchmark text comes from subtitles, it under-represents book and form layouts, so treating the reported scores as upper bounds on real-world accuracy is safer than treating them as typical.
  • Benchmarking fine-tuned systems from prior work on the same synthetic images would quantify the remaining headroom and show whether zero-shot commercial engines are already near the practical ceiling.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript reports a zero-shot benchmark of six OCR engines (Google Cloud Vision API, Document AI, Surya, Tesseract 5.5.0, Subasa OCR, and EasyOCR) on synthetic printed-text datasets for two low-resource South Asian languages: Sinhala (6,969 images from an existing Hugging Face dataset) and Tamil (7,000 newly created images sampled from OPUS OpenSubtitles and rendered in six Google Fonts). The systems are evaluated with CER, WER, BLEU, ANLS, and METEOR, and the authors also perform a character-level error analysis for the best engine per language. The headline results are that Surya is best for Sinhala across all metrics (WER 2.61%) and that Document AI is best for Tamil across all metrics (CER 0.78%). The paper additionally releases the synthetic Tamil OCR dataset on Hugging Face.

Significance. If the claims are supported, the paper provides a useful current snapshot of zero-shot OCR capabilities for Sinhala and Tamil on clean synthetic images, and the new public Tamil dataset is a concrete asset for future benchmarking. The comparison between vanilla Tesseract 5.5.0 and the fine-tuned Subasa OCR is also informative. However, the Tamil headline result rests on extremely small margins over Cloud Vision (differences of order 10^-3 to 10^-4 in Table 1) with no uncertainty quantification, so the central comparative claim for Tamil is not established by the reported numbers. The Sinhala result is much better supported because Surya's margin is large. The authors are transparent about the synthetic-image limitation, which appropriately scopes the contribution to clean rendered text rather than realistic scanned or camera-captured documents.

major comments (3)
  1. [Section 4, Table 1; Abstract; Conclusion] The claim that 'Document AI excelled across all metrics for Tamil' is not supported by the reported numbers. In Table 1, Document AI's advantage over Cloud Vision on Tamil is CER 0.0078 vs 0.0079, WER 0.1198 vs 0.1204, BLEU 0.5803 vs 0.5790, ANLS 0.9923 vs 0.9922, and METEOR 0.8762 vs 0.8751. These differences are one to four orders of magnitude smaller than the precision of the metrics, and the paper reports no confidence intervals, no significance tests, and no repeated runs. Because the two commercial APIs are black boxes that may be nondeterministic or version-dependent, the ranking could invert on a different random sample or a repeated call. Please add per-record bootstrap confidence intervals or significance tests, or explicitly rephrase the Tamil conclusion as a numerical near-tie rather than a decisive win.
  2. [Section 3.2] The construction of the Tamil sample is not reproducible. The authors state that they randomly selected 7,000 records from 222,658 filtered OPUS lines, but they do not report the random seed, the sampling procedure, or the selection code. This matters directly for the Tamil conclusion because the leading margin between Document AI and Cloud Vision is tiny. Please release the exact sample (or the seed and a script that reproduces it) together with the dataset.
  3. [Limitations; Abstract; Conclusion] The Limitations section concedes that the black-on-white synthetic images 'do not accurately reflect real-world conditions when capturing printed text,' yet the abstract and conclusion state language-level conclusions about 'zero-shot OCR accuracy' without this scope restriction. Since the ranking, particularly the near-tie between Document AI and Cloud Vision on Tamil, could change on realistic documents, the headline findings should be explicitly qualified as applying to clean synthetic printed text, or the paper should include at least one realistic test condition to probe whether the rankings persist.
minor comments (5)
  1. [Figure 3 caption] The caption spells the font name as 'Kavinar'; the correct name used in the text is 'Kavivanar'.
  2. [Section 3.2] The text-to-image function description does not specify the font size, rendering resolution, or DPI used to generate the images; these parameters are needed for reproduction and could affect all metrics.
  3. [Section 3.2] The post-processing differs between the two languages: Sinhala references and outputs were stripped of non-Sinhala characters, while Tamil text was filtered before rendering. This asymmetry should be justified or shown not to affect the cross-language WER comparison, since the paper draws a cross-language conclusion from those WER values.
  4. [Section 4] The phrase 'Tamil contains 247' is imprecise: 247 is the standard Unicode character count for the Tamil script including vowel signs and consonant clusters, not a count of basic letters, and the paper's own footnote acknowledges this. Please state the figure with that context.
  5. [General reproducibility] The paper does not mention releasing the evaluation scripts or configuration for computing CER, WER, BLEU, ANLS, and METEOR; releasing these would strengthen reproducibility, since tokenization choices materially affect BLEU and WER.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the ranking is an empirical benchmark with standard metrics and no fitted parameter renamed as a prediction.

full rationale

This is an empirical benchmarking study rather than a derivation chain. The central conclusions, that Surya performs best for Sinhala and Document AI performs best for Tamil, are obtained by running six OCR engines on fixed image-text pairs and comparing the outputs to the reference strings using standard string-similarity metrics (CER, WER, BLEU, ANLS, METEOR). None of these metrics is defined in terms of the engines' outputs in a way that forces a particular ranking, and no model parameters are fitted to the test data before evaluation. The newly introduced Tamil synthetic dataset is constructed from OPUS/OpenSubtitles text through line segmentation, length-based filtering, random selection of 7,000 records, and Pillow-based rendering in six fonts; the reference strings are the same strings used to render the images, which is normal ground-truth construction for an OCR benchmark and does not make the OCR outputs equal to the inputs by construction. The paper's self-citations, such as Ranathunga and de Silva (2022) for low-resource language categorization and de Silva (2025) for data scarcity, are background claims and are not load-bearing for the measured rankings. The Limitations section explicitly concedes that the clean synthetic black-on-white images do not accurately reflect real-world capture conditions; this is an external-validity caveat, not evidence of circularity. Concerns about missing confidence intervals, reproducibility of the random sample, and the very small Tamil margin between Document AI and Cloud Vision are robustness or correctness issues rather than circularity. The comparison is self-contained as an empirical benchmark, so no circular step is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are postulated. The main upstream commitments are domain assumptions about the validity of synthetic data and the applicability of MT metrics to OCR. The free parameters are dataset construction choices, not fitted model parameters.

free parameters (3)
  • Tamil text length filter = > 40 characters
    Chosen in Section 3.2 to include only records long enough for word-level evaluation; this changes the difficulty and composition of the Tamil dataset.
  • Tamil sample size = 7,000 records
    Selected to equalize with the 6,969-record Sinhala dataset; the random selection has no stated seed, making the exact sample non-reproducible.
  • Error-analysis thresholds = 5,000 for Sinhala; 1,600 for Tamil
    Arbitrary cutoffs used in Section 4 to list the most error-prone characters; descriptive and not central to the main ranking.
assumptions (3)
  • domain assumption Clean synthetic black-on-white images are representative of practical zero-shot OCR performance.
    The entire benchmark rests on synthetic datasets. The Limitations section concedes the images 'do not accurately reflect real-world conditions,' yet the headline rankings are drawn from them.
  • domain assumption BLEU and METEOR, designed for machine translation evaluation, are valid quality metrics for OCR output.
    Section 3.4 applies these MT metrics to OCR without validation or prior adoption evidence, which affects how the reported scores should be interpreted.
  • domain assumption Equalizing sample sizes makes Sinhala and Tamil results comparable.
    Section 3.2 states that matching sample sizes prevents skewed scores, but the two languages differ in script complexity, fonts, text source, and alphabet size, so cross-language comparisons such as 'best WER for Tamil is higher than Sinhala' are not clean comparisons of engine quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-shot OCR Accuracy of Low-Resourced Languages: A Comparative Analysis on Sinhala and Tamil." pith.science (2026). https://pith.science/paper/BWETAFIH

@misc{pith2026250718264,
  author       = {Pith},
  title        = {Pith review of: Zero-shot OCR Accuracy of Low-Resourced Languages: A Comparative Analysis on Sinhala and Tamil},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWETAFIH}},
  note         = {Machine review of arXiv:2507.18264}
}
read the original abstract

Solving the problem of Optical Character Recognition (OCR) on printed text for Latin and its derivative scripts can now be considered settled due to the volumes of research done on English and other High-Resourced Languages (HRL). However, for Low-Resourced Languages (LRL) that use unique scripts, it remains an open problem. This study presents a comparative analysis of the zero-shot performance of six distinct OCR engines on two LRLs: Sinhala and Tamil. The selected engines include both commercial and open-source systems, aiming to evaluate the strengths of each category. The Cloud Vision API, Surya, Document AI, and Tesseract were evaluated for both Sinhala and Tamil, while Subasa OCR and EasyOCR were examined for only one language due to their limitations. The performance of these systems was rigorously analysed using five measurement techniques to assess accuracy at both the character and word levels. According to the findings, Surya delivered the best performance for Sinhala across all metrics, with a WER of 2.61%. Conversely, Document AI excelled across all metrics for Tamil, highlighted by a very low CER of 0.78%. In addition to the above analysis, we also introduce a novel synthetic Tamil OCR benchmarking dataset.

Figures

Figures reproduced from arXiv: 2507.18264 by the authors.

Figure 1
Figure 1. An example of the use of rounded script in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Tamil synthetic OCR dataset [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Three examples of Tamil sentences from our dataset in the fonts Hind Madurai, Anek Tamil, and Kavinar, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Results for Sinhala and Tamil at 11.98% with that of Sinhala at an impressive 2.61% as depicted in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 10 canonical work pages

  1. [9]

    In 2020 sec- ond international conference on inventive research in computing applications (ICIRCA), pages 357–362

    Text extrac- tion using OCR: A Systematic Review. In 2020 sec- ond international conference on inventive research in computing applications (ICIRCA), pages 357–362. IEEE. Meharuniza Nazeem, Anitha R, Navaneeth S, and Ra- jeev R. R

    work page 2020

  2. [11]

    arXiv preprint arXiv:1906.02358v25

    Survey on Publicly Available Sinhala Natural Language Processing Tools and Re- search. arXiv preprint arXiv:1906.02358v25. Ray Smith

    arXiv 1906

  3. [13]

    arXiv preprint arXiv:2409.01704

    General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model. arXiv preprint arXiv:2409.01704. Yudhanjaya Wijeratne, Nisansa de Silva, and Yashothara Shanmugarajah

    arXiv

  4. [2007]

    In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633

    An overview of the Tesseract OCR en- gine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633. IEEE. J¨org Tiedemann

    work page 2007

  5. [2015]

    In 2015 Fifteenth International Conference on Advances in ICT for Emerging Regions (ICTer), pages 130–134

    Developing a commercial grade Tamil OCR for recognizing font and size independent text. In 2015 Fifteenth International Conference on Advances in ICT for Emerging Regions (ICTer), pages 130–134. IEEE. YV ANT Maduranga and Shantha Jayalal

    work page 2015

  6. [2017]

    arXiv preprint arXiv:1701.06538

    Out- rageously large neural networks: The sparsely- gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. Nisansa de Silva

    arXiv

  7. [2019]

    In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1–6

    Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1–6. IEEE. KGND Karunarathne, KV Liyanage, DAS Ruwanmini, GKA Dias, and ST Nandasara

    work page 2019

  8. [2020]

    In 2020 20th International Conference on Advances in ICT for Emerging Regions (ICTer), pages 298–299

    Deep Learning Based Sinhala Optical Character Recognition (OCR). In 2020 20th International Conference on Advances in ICT for Emerging Regions (ICTer), pages 298–299. IEEE. Ninad Awalgaonkar, Prashant Bartakke, and Ravindra Chaugule

    work page 2020

Show all 13 references
  1. [2021]

    In 2021 international symposium of Asian control association on intelli- gent robotics and industrial automation (IRIA), pages 394–399

    Automatic License Plate Recog- nition System Using SSD. In 2021 international symposium of Asian control association on intelli- gent robotics and industrial automation (IRIA), pages 394–399. IEEE. Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee

    2021

  2. [2022]

    In 2022 2nd International Conference on Advanced Research in Computing (ICARC), pages 120–124

    Multi- Style Printed Sinhala Character Recognition and Dig- italization Using Artificial Neural Network. In 2022 2nd International Conference on Advanced Research in Computing (ICARC), pages 120–124. IEEE. Rishabh Mittal and Anchal Garg

    2022

  3. [2023]

    In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing , pages 4895– 4901, Singapore

    GQA: Training generalized multi-query trans- former models from multi-head checkpoints. In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing , pages 4895– 4901, Singapore. Association for Computational Lin- guistics. Isuri Anuradha, Cha...

    2023

  4. [2024]

    In Proceedings of the 4th Workshop on Natural Lan- guage Processing for Indigenous Languages of the Americas (AmericasNLP 2024), pages 88–102, Mex- ico City, Mexico

    A concise survey of OCR for low-resource languages. In Proceedings of the 4th Workshop on Natural Lan- guage Processing for Indigenous Languages of the Americas (AmericasNLP 2024), pages 88–102, Mex- ico City, Mexico. Association for Computational Lin- guistics. Joshua Ainslie...

    2024

  5. [2025]

    In Proceedings of the 1st Workshop on Language Models for Under- served Communities (LM4UC 2025), pages 86–103, Albuquerque, New Mexico

    Nayana OCR: A scalable framework for document OCR in low-resource languages. In Proceedings of the 1st Workshop on Language Models for Under- served Communities (LM4UC 2025), pages 86–103, Albuquerque, New Mexico. Association for Compu- tational Linguistics. Pierre Lison and J...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.