Pith. sign in

REVIEW 4 major objections 5 minor 15 references

Three LLMs acting as closed-loop architecture designers discover compact OCR networks that exceed 93 percent mean test accuracy on Arabic, Persian, and English handwriting without human tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An LLM-driven closed-loop neural architecture search is applied to Arabic, Persian, and English handwriting, claiming mean test accuracies above 93% and 41–44 ms inference.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Well-executed LLM-driven architecture search for handwriting, but feeding test accuracy into the search loop means the headline accuracies are not valid generalization estimates. the 4 major comments →

arxiv 2607.15509 v1 pith:MR4YLZZE submitted 2026-07-16 cs.CV cs.AIcs.LG

LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4

Mobina Kashaniyan , Amirhossein Ghassemi , Nasser Mozayani This is my paper
classification cs.CV cs.AIcs.LG
keywords large language modelsneural architecture searchhandwritten text recognitionmultilingual OCRAutoMLclosed-loop feedbackArabic handwritingPersian handwriting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that large language models can serve as autonomous neural architecture designers for handwritten OCR, rather than as mere recognizers. In the proposed pipeline, each LLM proposes a model specification in JSON, trains the model, reads back measured performance, and refines the next proposal. Across 270 trials spanning Arabic, Persian, and English, the authors report mean test accuracies above 93 percent, a best accuracy of 98.1 percent, and per-sample inference latency of roughly 41–44 ms. If the results stand, they indicate that LLM-driven AutoML can replace manual architecture design and hyperparameter tuning for cross-lingual handwriting recognition.

Core claim

The paper's central claim is that a closed feedback loop — prompt an LLM with dataset metadata and prior performance, have it return a complete architecture and training schedule in JSON, train the resulting model, and feed the measured accuracies back into the next prompt — reliably converges to accurate, compact models for handwritten Arabic, Persian, and English. Each of the three tested LLMs independently ran thirty trials per script, producing mean test accuracies from 93.1 to 95.9 percent depending on script and designer, and a best overall trial of 98.1 percent on Arabic. The authors further argue that inference latency stays near 41–44 ms even when model size varies several-fold, ind

What carries the argument

The central mechanism is the closed performance-feedback loop: a structured prompt containing dataset details (class count, image dimensions) plus a JSON summary of the previous trial's training, validation, and test accuracies is sent to an LLM, which responds with a new architecture specification in JSON. A deterministic parser converts that specification into a trainable deep-learning model with convolutional, pooling, normalization, dropout, and optional transformer blocks, and then training and evaluation produce the metrics for the next loop iteration. Repeated over thirty trials per language per LLM, this loop is what allows the search to converge without any human designer in the loo

Load-bearing premise

The reported test accuracies are treated as independent estimates of generalization, but the paper states in Section III-E that test accuracy is part of the JSON feedback passed into each subsequent LLM prompt; if the search is fitting to the test set through that feedback, the reported numbers are not clean generalization estimates.

What would settle it

Re-run the pipeline on the same three datasets while withholding the test set from the feedback channel, giving the LLM only training and validation accuracies; if the mean held-out test accuracy drops materially below the reported 93–98 percent range, the published numbers were inflated by test-set feedback.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The same pipeline could be pointed at a new script by changing only the dataset metadata and class count, making script adaptation a prompt-engineering problem rather than an architecture-design problem.
  • Since mean accuracies above 93 percent are reached with compact models (under about 1.4 million parameters for GPT-5), accuracy and deployment efficiency do not appear to trade off in this setup.
  • The reported latency stability across model sizes suggests that real-time constraints can be satisfied even when an LLM proposes larger architectures.
  • The closed-loop design removes the need for expert architecture selection and manual hyperparameter tuning, which could shorten the OCR model development cycle.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A fair accuracy estimate would require withholding the test set from the feedback channel; the reported numbers are likely optimistic if the search is fitting to the test set through that feedback.
  • The systematic differences among the three LLM designers — compact models from GPT-5, larger models from Claude Sonnet 4 — point to prompt priors and model-specific biases as a variable worth isolating in future NAS comparisons.
  • The constant-latency finding, if replicated, would imply that parameter count is a poor proxy for deployment cost, and that latency budgets should be measured rather than inferred from model size.
  • The method should transfer to sequence-level handwritten line or paragraph recognition, but transfer would need validation because character-level feedback may not capture sequential error patterns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a fully automated closed-loop AutoML pipeline in which LLMs (GPT-5, GPT-4o, Claude Sonnet 4) propose, train, and iteratively refine Keras neural architectures for handwritten OCR in Arabic, Persian, and English. For each trial the LLM receives a JSON prompt containing dataset metadata and the previous trial's training, validation, and test accuracies, and responds with a full architecture/hyperparameter specification. The authors run 30 trials per script per LLM (270 total) and report mean test accuracies, best-trial accuracies, parameter counts, and inference latencies. They claim the pipeline consistently discovers accurate and efficient models without manual architecture design, domain-specific preprocessing, or hyperparameter tuning, and that LLMs can act as effective AutoML agents.

Significance. If the reported results were valid, the paper would demonstrate an interesting application of LLMs as NAS agents for cross-lingual OCR. The scale of experimentation — 270 trials across three scripts and three LLMs — is a strength, as is the documentation of parameter counts and latencies. However, the central evaluation is methodologically compromised: test accuracy is included in the feedback loop that guides subsequent architecture proposals, so the reported test accuracies are not independent estimates of generalization. In addition, the paper lacks any non-LLM baseline, provides no code or data despite a reproducibility claim, and draws conclusions from small accuracy differences without significance testing. Because the headline claim rests on these flawed estimates, the paper cannot be accepted in its current form.

major comments (4)
  1. [§III-E and §III-B, Table I] The feedback summary passed to the LLM explicitly includes test accuracy, and Section III-B confirms that performance metrics from the previous iteration, including test accuracy, are included in subsequent prompts. This makes the test set part of the optimization signal. Consequently, the mean and best test accuracies in Table I are selection-optimized rather than unbiased estimates. The 'Best Acc' column is particularly affected, since it is the maximum over 30 trials guided by test feedback. The authors must either use a held-out test set that is never exposed to the LLM, or retrain the search using only training/validation feedback and then evaluate the final selected architecture once on a separate test split.
  2. [§IV and Table II] The evaluation contains no comparison to non-LLM baselines such as a fixed CNN, random architecture sampling, or a conventional NAS method (e.g., GPT-NAS [3] or GENIUS [4]). Table II compares only to Cerescu & Bumbu [14], which is not a NAS baseline and uses a different task and metrics. Without a baseline, the high accuracies may simply reflect the relative ease of the chosen datasets, and the claim that LLMs are effective AutoML agents is unsupported. The paper needs at least one simple baseline (e.g., a standard CNN trained with the same data pipeline) and ideally a random-search control.
  3. [§IV-B] The paper reports 'clear differences' among the three LLMs based on mean accuracies whose standard deviations overlap substantially. For example, English mean accuracies are 0.937±0.008 for GPT-5 and 0.936±0.004 for GPT-4o; this 0.001 difference is far smaller than the reported variability. No confidence intervals, effect sizes, or significance tests are provided for any pairwise comparison. Yet the paper concludes that GPT-5 offers the 'best accuracy-to-size profile' and that GPT-4o is 'balanced.' These conclusions are not supported by the reported statistics. The authors should either add appropriate significance testing or downgrade the comparative claims to descriptive observations.
  4. [§I, §III-D, §IV] The abstract and contribution list claim 'Transparency and Reproducibility,' and Section III-D states that all trial metrics and JSON specifications are logged in CSV files. However, no code, data, or logs are provided or linked in the manuscript. Without these artifacts, the 270 experiments cannot be audited, the 'deterministic parsing' cannot be verified, and the paper's central results cannot be reproduced by readers. For an empirical AutoML paper, the absence of a reproducibility package is a significant omission, especially given the explicit reproducibility claim.
minor comments (5)
  1. [Abstract/Title] The abstract contains a typo: 'Claude Sonnate 4' should be 'Claude Sonnet 4.'
  2. [§I Contributions vs Abstract] The contribution list states that 'Each language benefits from tailored preprocessing and encoding strategies,' while the abstract claims the pipeline requires 'no manual intervention, domain specific preprocessing.' These statements are contradictory and should be reconciled.
  3. [§III-A] The data split section says 10% of the data is dedicated to validation, but the test-set proportion is not specified. As the test set plays a central role in the feedback loop, its construction and size should be described precisely.
  4. [Table II and Ref. [14]] The table refers to 'Cerescu & Bumbu (2024),' but the reference list entry [14] is authored solely by 'T. Bumbu.' The author attribution is inconsistent and should be corrected.
  5. [§IV (Figure references)] The text 'as seen in Figures 9, 6, and 3' lists figures out of numeric order, which is confusing; reorder to 3, 6, and 9.

Circularity Check

1 steps flagged

Test-set accuracy is fed back into the LLM's prompt, so the reported Table I test accuracies are not independent generalization estimates.

specific steps
  1. fitted input called prediction [Section III-B, III-E and Table I]
    "At the beginning of each trial, we prepare a structured prompt containing essential dataset details ... and, when available, performance metrics from the previous iteration (training, validation, and test accuracies). ... Following each training trial, we compile a structured summary containing key performance metrics training accuracy, validation accuracy, and test accuracy into a concise JSON object. This summary is passed as feedback into the subsequent LLM prompt."

    The test accuracy is part of the optimization signal: the LLM adjusts subsequent architectures based on test accuracy from previous trials. The paper then reports mean/best test accuracies (Table I: 0.921-0.959 mean, up to 0.981 best) as evidence of the pipeline's generalization. Because the test set is used to guide the search, these numbers are maxima over a search that is fitted to the same test set; they are not held-out estimates. The evaluation metric is thus also the objective, making the claim of 'strong generalization' self-referential rather than an independent validation.

full rationale

The paper does not derive an equation, but its central empirical claim is circular in the evaluation sense. Section III-E explicitly states that the feedback summary passed to the LLM includes test accuracy, and Section III-B confirms test accuracy is included in subsequent prompts. Therefore the LLM's architecture proposals are conditioned on test-set performance, and the reported Table I test accuracies are biased by construction. No separate final-test protocol, code, or held-out data is provided to show the results survive an unbiased evaluation. The concept might be salvageable by using only training/validation feedback, but as written the evidence does not support the generalization claim. No self-citation issues or imported uniqueness theorems are present.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claims rest on a flawed evaluation assumption (test feedback does not bias results), unverified dataset splits, stochastic LLM outputs, and an unsupported multiple-trial selection procedure. No free parameters or invented entities are introduced; the main burden is the self-referential search loop.

axioms (4)
  • domain assumption Test accuracy can be fed back into the LLM prompt without invalidating the reported generalization estimates.
    Section III-E states test accuracy is part of the feedback; Section IV reports these test accuracies as the paper's headline numbers. This premise is false: using test results to guide search makes the test set part of the optimization loop.
  • domain assumption EMNIST, SADRI, and AHCD splits are representative and balanced; a 10% validation split is used, but the test split is unspecified.
    Section III-A describes data loading and validation split but does not specify test set size or class distribution; reported accuracies depend on this unstated split.
  • domain assumption The LLMs (GPT-5, GPT-4o, Claude Sonnet 4) return parseable, reproducible architecture JSONs.
    Section III-B/C depends on deterministic JSON parsing; LLM output is stochastic and version-dependent, yet the paper treats each trial as independent and reproducible.
  • domain assumption Thirty trials per script per LLM are sufficient for stable comparisons without multiple-testing correction.
    Section IV reports 'best trial' metrics, but selecting a max over 30 trials inflates expected best accuracy; no correction is applied.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4." pith.science (2026). https://pith.science/paper/MR4YLZZE

@misc{pith2026260715509,
  author       = {Pith},
  title        = {Pith review of: LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MR4YLZZE}},
  note         = {Machine review of arXiv:2607.15509}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present a fully automated closed-loop AutoML framework that uses GPT-5, GPT-4o, and Claude Sonnet 4 as autonomous neural architecture designers for cross-lingual handwritten optical character recognition. Each large language model independently generates, trains, evaluates, and iteratively refines neural network architectures using performance feedback from previous trials. The framework is evaluated on Arabic, Persian, and English handwriting datasets through 270 independent experiments. It consistently discovers accurate and computationally efficient models without manual architecture design, domain-specific preprocessing, or hyperparameter tuning. The generated models achieve mean test accuracies above 93 percent, a best accuracy of 98.1 percent, and inference latency between 41 and 44 milliseconds. The results demonstrate that large language models can function as effective AutoML agents for neural architecture search, enabling scalable, script-adaptive, and reproducible handwriting recognition across languages.

Figures

Figures reproduced from arXiv: 2607.15509 by Amirhossein Ghassemi, Mobina Kashaniyan, Nasser Mozayani.

Figure 1
Figure 1. Figure 1: The fully automated LLM-driven cross-lingual OCR [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Claude Sonnet 4 parameter count and per sample inference time across thirty trials for Arabic, English, and Persian. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Claude Sonnet 4 training and validation accuracy across thirty generated models for Arabic, English, and Persian. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Claude Sonnet 4 distribution of test accuracy across [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: GPT 4o parameter count and per sample inference time across thirty trials for Arabic, English, and Persian. Bars show [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: GPT 4o training and validation accuracy across thirty trials for Arabic, English, and Persian. Validation accuracy tracks [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: GPT 4o distribution of test accuracy across thirty trials [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: GPT 5 parameter count and per sample inference time across thirty trials for Arabic, English, and Persian. Bars indicate [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: GPT 5 training and validation accuracy across thirty trials for Arabic, English, and Persian. Validation closely follows [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: GPT 5 distribution of test accuracy across thirty trials [PITH_FULL_IMAGE:figures/full_fig_p006_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 5 linked inside Pith

  1. [1]

    Gagan, EM

    B. Gagan, EM. Billah Nagoudi, F. Alwajih, & M. Abdul-Mageed. Qalam: A multimodal LLM for Arabic optical character and handwriting recognition. arXiv preprint arXiv:2407.13559, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Motamedisedeh,F

    O. Motamedisedeh,F. Zagia, , & A. Alaei. A new optimization approach to improve an ensemble learning model: application to Persian/Arabic handwritten character recognition. In International conference on doc- ument analysis and recognition (pp. 180-194). Cham: Springer Nature Switzerland, 2023

    2023

  3. [3]

    C. Yu, X. Liu, Y . Wang, Y . Liu , W. Feng ,X. Deng , C. Tang , J. Lv . GPT-NAS: Evolutionary neural architecture search with the generative pre-trained model. arXiv preprint arXiv:2305.05351. 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Zheng, X

    M. Zheng, X. Su, S. You, F. Wang, C. Qian, C. Xu, S. Albanie . Can gpt- 4 perform neural architecture search?. arXiv preprint arXiv:2304.10970. 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Sharma, M

    V . Sharma, M. Nazim, & S. Prakash Sharma, A Systematic Literature Review on Handwritten Text Recognition: Approaches, Challenges, and Future Directions. Challenges, and Future Directions, 2025

    2025

  6. [6]

    ”Gpt-4o system card.” arXiv preprint arXiv:2410.21276 (2024)

    Aaron Hurst, et al. ”Gpt-4o system card.” arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  7. [7]

    Cohen, S

    G. Cohen, S. Afshar, J. Tapson, & A. Van Schaik, EMNIST: Extending MNIST to handwritten letters. In 2017 international joint conference on neural networks (IJCNN) - pp. 2921-2926-. IEEE, 2017

    2017

  8. [8]

    Sadri, M

    J. Sadri, M. R. Yeganehzad, and J. Saghi, ”A novel comprehensive database for offline Persian handwriting recognition,” Pattern Recog- nition, vol. 60, pp. 378-393, 2016

    2016

  9. [9]

    Arabic handwritten characters recognition using convolutional neural network,

    A. El-Sawy, M. Loey, and H. EL-Bakry, “Arabic handwritten characters recognition using convolutional neural network,” WSEAS Transactions on Computer Research, vol. 5, pp. 11–19, 2017

    2017

  10. [10]

    Bonyani, S

    M. Bonyani, S. Jahangard, & M. Daneshmand, Persian handwritten digit, character and word recognition using deep learning. International Journal on document analysis and recognition (IJDAR), 24(1), 133-143, 2021

    2021

  11. [11]

    Khosrobeigi, H

    Z. Khosrobeigi, H. Veisi, E. Hoseinzade, & H. Shabanian, Persian optical character recognition using deep bidirectional long short-term memory. Applied Sciences, 12(22), 11760, 2022

    2022

  12. [12]

    Jafarzadeh, Offline Persian handwriting recognition with CNN and RNN-CTC

    Safarzadeh, V .Mohammadi & P. Jafarzadeh, Offline Persian handwriting recognition with CNN and RNN-CTC. In 2020 25th international computer conference, computer society of Iran (CSICC) (pp. 1-10). IEEE, 2020

    2020

  13. [13]

    Garrido-Munoz, A

    C. Garrido-Munoz, A. Rios-Vila, & J. Calvo-Zaragoza, Handwritten Text Recognition: A Survey. arXiv preprint arXiv:2502.08417, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    T. Bumbu. Comparison of Large Language Models and Traditional Neural Networks in Optical Character Recognition for Old Alphabets. In International Conference dedicated to the 60th anniversary of the foundation of Vladimir Andrunachievici Institute of Mathematics and Computer Science, MSU pp. 259-265, 2024

    2024

  15. [15]

    Introducing GPT-5

    OpenAI. Introducing GPT-5. 2025. Available at: https://openai.com/ index/introducing-gpt-5/. Accessed: 2025-08-12

    2025

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.