{"id":"fe0ae362-80db-4d00-9775-9c1ce28907a6","arxiv_id":"2411.13350","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A standard CNN-BiLSTM+CTC model is applied to Ge'ez handwriting, reporting SOTA CER/NED on HHD-Ethiopic, but the claimed two-stage approach is absent from the method.","lead":"This paper reports a CNN-LSTM system for recognizing handwritten Ge'ez script and claims the lowest error rates on the HHD-Ethiopic benchmark. But the abstract describes a two-stage pipeline that the method section never implements, and no code or training details are provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper claims a two-stage CNN-then-LSTM recognition pipeline, but the Methods section describes only a conventional single-stage CRNN trained with CTC; the SOTA claim rests on a method that is not actually specified.","rationale":"The reader correctly identifies the baseline comparison protocol as a serious weakness. However, the most load-bearing issue is more basic: the paper's central contribution is a two-stage recognition process, but the method section describes a standard single-stage CRNN. This is an internal inconsistency, not a disagreement with external consensus. If the evaluated model is the single-stage CRNN, then the claimed novelty ('dual-stage recognition') is not what produced the reported results, and the SOTA claim loses its basis regardless of how carefully the baselines were handled. The reader's concern about baseline comparability is valid and remains relevant, but it is secondary to the architecture mismatch. The concrete test — reproducing the described single-stage model on the public dataset — would settle whether the reported numbers can be attributed to the described method. Since this concern reinforces the reader's REJECT verdict rather than changing it, the verdict is unchanged.","tokens_in":5883,"tokens_out":3284,"duration_ms":36064,"concrete_test":"Implement the single-stage CRNN exactly as specified in Section III.B (four residual blocks, channels 64 to 512, two BiLSTM layers with 512 units, CTC loss) and train/evaluate it on the public HHD-Ethiopic dataset using the same IID/OOD splits. If this model alone reproduces the reported OOD CER of 26.95 and NED of 26.50, then the results do not require and do not provide evidence for the claimed two-stage pipeline; if it does not reproduce them, the reported numbers are unexplained by the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a 'dual-stage recognition approach' — first train a CNN to recognize individual characters, then use it as a feature extractor for an LSTM word recognizer — achieves state-of-the-art results on HHD-Ethiopic. The Methods section does not describe this two-stage pipeline. Section III.A and III.B specify a single end-to-end CRNN: four residual CNN blocks, two stacked BiLSTM layers, and a CTC loss, with no character-classification pretraining step. Section III.A even says the model 'recognize[s] Amharic text' rather than Ge'ez. The only trace of a character-level model is a passing sentence in Section IV about 'loss and classification accuracy of the character recognition model,' but no architecture, loss, or training schedule for that stage is given. Thus the reader cannot tell what was actually evaluated, and the claimed SOTA numbers in the abstract and Table I cannot be attributed to the stated contribution. This is load-bearing because the novelty and the performance claim are both defined by the dual-stage design; if the evaluated system is actually the single-stage CRNN of Section III, the paper's central claim is unsupported even if the numeric comparison to baselines is valid.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a CNN-LSTM system for handwritten Ge'ez word recognition, claiming a two-stage process in which a CNN is first trained to recognize individual characters and then used as a feature extractor for an LSTM-based word recognizer. The authors report Character Error Rate (CER) and Normalized Edit Distance (NED) on the HHD-Ethiopic dataset under both in-distribution (IID) and out-of-distribution (OOD) test sets, and claim state-of-the-art results, including outperforming human performance. Section IV presents comparisons with SVTR, ASTER, ABINet, CRNN, Plain-CTC, Attn-CTC, and two HPopt variants.","tokens_in":6162,"tokens_out":4791,"duration_ms":51720,"significance":"If the results are verified, a strong handwritten-text recognition system for the low-resource Ge'ez script would be a meaningful contribution, with clear applications to historical document digitization and cultural-heritage preservation. The paper uses a publicly available benchmark, standard evaluation metrics, and includes human-performance comparison. However, the current manuscript does not provide enough evidence to verify the central claim: the architecture described in the Methods section is a conventional single-stage CRNN, no code or training details are supplied, and the baseline numbers appear to be imported from prior work without a documented evaluation protocol. These issues are load-bearing because the stated novelty and the state-of-the-art claim are both defined by the two-stage design.","major_comments":[{"comment":"The abstract and the contribution bullet in Section I claim a 'two-stage recognition process': first a CNN is trained to recognize individual characters, then it acts as a feature extractor for an LSTM-based word recognizer. However, Section III.A and Section III.B describe only a single end-to-end CRNN: four residual CNN blocks, two stacked bidirectional LSTM layers, and CTC training. No step of separately training a character classifier is described, and the only trace of such a model is the passing mention in Section IV of the 'loss and classification accuracy of the character recognition model' associated with Figure 4. The reader therefore cannot determine what system produced the numbers in Table I, and the claimed state-of-the-art results cannot be attributed to the stated contribution. The authors should either provide a complete description of the two-stage training procedure or revise the abstract and contribution list to describe the actual CRNN system.","section":"Abstract and Section I, versus Section III"},{"comment":"The baseline comparison in Table I is not sufficiently specified. The caption says only 'trained on HHD [8]', and the row values for Human-performance, SVTR, ASTER, ABINet, CRNN, Plain-CTC, Attn-CTC, HPopt-Plain-CTC, and HPopt-Attn-CTC appear to be taken directly from reference [8] or other cited works. The manuscript does not describe whether the same preprocessing, same IID/OOD splits, same training data, same decoding method, or same random seeds were used for all methods. Without this information, the reported margins (for example, OOD CER 26.95 versus 28.65 for the best prior method) may simply reflect differences in evaluation protocol rather than algorithmic improvements. The authors should re-run the baselines under a shared protocol or provide a detailed description of how each baseline number was obtained, together with error bars or confidence intervals.","section":"Section IV, Table I"},{"comment":"There is a direct numerical inconsistency in the reported results. The text in Section IV states that 'our model obtains 15.48 for CER using the IID test set, and 27.96 for the OOD test set', but Table I lists the OOD CER for 'Ours' as 26.95, and the abstract and contribution bullet also report a CER of 26.95 and an NED of 26.50. The authors should reconcile these values and ensure that all reported numbers correspond to the same test split and metric definition. In addition, Eq. (2) defines NED as a value between 0 and 1, but the values in Table I are reported as percentages (e.g., 26.50), so the scaling convention should be clarified.","section":"Section IV and Table I"},{"comment":"The experimental setup is not reproducible from the information given. Section III.B mentions only that training and evaluation were performed on a Linux system with an NVIDIA RTX 3090 GPU; it does not report the optimizer, learning rate, batch size, number of epochs, input image size, data augmentation, or random seeds. No code or trained model is provided in the submission. Because the central claim is a new state of the art on a public benchmark, the authors should provide the code and/or checkpoint, or at a minimum a complete training and evaluation protocol, including how many independent runs were performed and how variability was measured.","section":"Section III.B and IV"}],"minor_comments":[{"comment":"The section says the model is designed to 'recognize Amharic text in images', while the title and abstract concern Ge'ez. Although Amharic and Ge'ez share the Ethiopic script, the terminology should be consistent and clearly distinguish the target language/script.","section":"Section III.A"},{"comment":"There is a typo: 'A Finally, a fully connected layer' should read 'Finally, a fully connected layer'. Also, the phrase 'maps the attended features' refers to attention, but no attention mechanism is described anywhere in the architecture.","section":"Section III.B"},{"comment":"Figure 4 is said to show 'the loss and classification accuracy of the character recognition model', but the caption and text do not explain what this character recognition model is, how it was trained, or how it relates to the word-level CRNN described in Section III. The figure also lacks axis labels and a clear caption.","section":"Section IV, Figure 4"},{"comment":"The sentence describing the HHD-Ethiopic dataset contains the fragment 'with in the testing distribution', which appears to be incomplete. It also says the dataset comprises 79,684 samples including two test sets; it would be clearer to state the number of training samples separately from the IID and OOD test sets.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a serious mismatch between the claimed two-stage contribution and the actually described single-stage CRNN, and the baseline comparison is not documented. These issues are fixable in principle, but only if the authors can provide a complete description of the evaluated system, release code or detailed training/evaluation protocols, and reconcile the inconsistent reported numbers. If the authors cannot clarify the baseline provenance or provide reproducibility artifacts, I would recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is a new set of numbers on the HHD-Ethiopic benchmark: CER 15.48 IID, 26.95 OOD, NED 26.50 OOD, which beat the HPopt-Attn-CTC baseline from Belay et al. If those numbers hold up, that is a real step for a low-resource script. The authors also do something right: they evaluate on the same public splits, report IID/OOD separately, and include the human-performance row from [8]. That is the right way to frame a niche OCR result.\n\nThe soft spot is not minor. The abstract and contributions claim a two-stage pipeline: first train a CNN on isolated characters, then use it as a frozen feature extractor for an LSTM word recognizer. The Methods section, III-A and III-B, describes a single end-to-end CRNN with four residual CNN blocks, two BiLSTMs, and CTC loss. No character-level pretraining step is described. There is one sentence in Section IV about the 'character recognition model' loss/accuracy and a figure, but no architecture or training details. So the reader cannot tell what was actually evaluated. The copy-paste slip in III-A ('recognize Amharic text') adds to the impression that the write-up was rushed.\n\nAlso missing: code, training epochs, learning rate, optimizer, preprocessing, and any error bars or repeated runs. Table I lists baseline numbers but the caption just says 'trained on HHD [8]' with no protocol. The numbers may be perfectly comparable, but the paper gives no reason to trust that they were run under identical conditions. That is a comparability concern, though I don't see evidence of fabricated numbers: this is a plausibly correct CRNN on a public dataset.\n\nSo the empirical result is new and could matter to the Ethiopic OCR community, but the central claim as stated is unsupported by the method. The paper should not be accepted in this form. It deserves a serious referee, not a desk reject, because the benchmark question is real and the issues are fixable in revision. If the two-stage claim is false and it's just a CRNN, the authors need to say so and reposition the contribution as a strong CRNN baseline. If the two-stage system exists, they need to specify it fully. Either way, the letter should be a revise-with-major-changes, not a clean reject.","headline":"New numbers on a public benchmark that beat prior Ge'ez OCR baselines, but the abstract's two-stage method doesn't match the described CRNN, so the SOTA claim isn't supported as written.","tokens_in":6678,"tokens_out":3162,"would_cite":false,"duration_ms":32164,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CNN–LSTM recognizer trained on the HHD-Ethiopic dataset claims state-of-the-art handwritten Ge'ez recognition, with OOD CER 26.95 and NED 26.50, below all eight compared systems and below human performance.","keywords":["Ge'ez script","handwritten text recognition","CNN-LSTM","connectionist temporal classification","Ethiopic OCR","HHD-Ethiopic dataset","character error rate","low-resource script recognition"],"falsifier":"Reproduce the experiment under one protocol: download the released model and code, retrain each of the eight baselines on the same HHD-Ethiopic IID/OOD splits, and compute CER/NED on the OOD set. Any baseline scoring below 26.95 CER or 26.50 NED, or a rerun of the proposed model producing a higher value than 26.95, would disprove the state-of-the-art claim.","tokens_in":5725,"feed_emoji":"📜","tokens_out":8069,"duration_ms":81543,"temperature":0.7,"pith_summary":"This paper argues that a hybrid CNN–LSTM recognizer can set a new best result for handwritten Ge'ez word recognition on the HHD-Ethiopic dataset, beating eight published OCR systems and the reported human-level baseline. The proposed pipeline is dual-stage: a CNN is trained to recognize individual Ge'ez characters and then acts as a feature extractor for an LSTM word-level recognizer, trained with Connectionist Temporal Classification so no character boundary labels are needed. On the in-distribution test set the paper reports CER 15.48 and NED 15.00, and on the harder out-of-distribution 18th-century manuscript test set it reports CER 26.95 and NED 26.50. The authors present this as a step toward digitizing historical Ethiopic manuscripts and as evidence that low-resource ancient scripts can be served by a relatively standard CNN–LSTM+CTC design.","feed_headline":"Ge'ez handwriting model beats human readers and 8 OCR systems","feed_subtitle":"On the HHD-Ethiopic benchmark it reaches 26.95 CER on historical manuscripts, the best reported.","key_machinery":"The load-bearing mechanism is a CNN–LSTM sequence recognizer trained with Connectionist Temporal Classification on the 182-character Ethiopic syllabary. Four residual convolutional blocks, with channels growing from 64 to 512, extract hierarchical visual features and reduce spatial dimensions through max pooling, with dropout and batch normalization for regularization; the resulting feature map feeds two stacked bidirectional LSTM layers of 512 units per direction, and a fully connected layer outputs character probabilities per time step plus a blank CTC token. The abstract's 'dual-stage' framing adds a separate CNN character-recognition stage whose weights then serve as the feature extractor for the LSTM word recognizer. The evaluation machinery is the HHD-Ethiopic dataset's IID split (6,375 samples) and OOD split (15,935 historical samples), scored by Character Error Rate and Normalized Edit Distance.","core_discovery":"The paper claims that its CNN–LSTM architecture reaches state-of-the-art performance for handwritten Ge'ez recognition on the HHD-Ethiopic dataset, surpassing eight compared methods and the human-level baseline. The decisive reported numbers are CER 26.95 and NED 26.50 on the out-of-distribution test set, against 28.65 and 27.37 for the best prior baseline and 33.20 and 33.70 for human readers. The design intent is that the convolutional blocks extract visual features from characters and the stacked bidirectional LSTMs model the sequential dependencies of words, with CTC allowing learning from unsegmented image–text pairs. The authors present the result as an advance for digitizing historical Ethiopic manuscripts and for low-resource script recognition.","pith_inferences":["The paper does not ablate the two stages, so I would not infer from it alone that the separate character-level CNN pretraining is what drives the gain; an end-to-end CTC model without that stage could perform equally well.","The reported margin over the strongest prior baseline (OOD CER 28.65 to 26.95) is about 1.7 points; whether that translates into practically useful transcription depends on the error distribution across common and rare characters, which the paper does not report.","Because the baselines' training details are not described, the fair inference is that the architecture works well on this dataset; the comparative ranking's strength will be settled only by reproducing all methods in one protocol.","A testable extension the authors do not run: apply the same dual-stage pipeline to other low-resource historical scripts with syllabic writing systems and small corpora, using their own CNN-first-then-LSTM schedule."],"forward_implications":["Handwritten Ge'ez can be transcribed without character-level segmentation, since CTC training aligns unsegmented image sequences to word or line transcripts.","The benchmark gap between IID (CER 15.48) and OOD (CER 26.95) gives a quantified measure of how far historical-manuscript style diverges from modern handwriting.","If the results reproduce, the HHD-Ethiopic human-performance baseline is no longer the upper bound for this task, and future systems should be compared against this CNN–LSTM result.","The architecture is not script-specific beyond the output character set, so the same dual-stage recipe can be lifted directly to other Ethiopic-script languages such as Amharic."],"supporting_citations":[{"why":"Supplies the HHD-Ethiopic dataset, the IID/OOD test splits, the human-performance baseline, and the HPopt baseline results that the proposed model is claimed to beat.","marker":"[8]"},{"why":"Provides the SVTR baseline that appears in Table I and is outperformed by the claimed results.","marker":"[11]"},{"why":"Provides the CRNN baseline in Table I, defining the convolutional recurrent sequence-recognition baseline family.","marker":"[12]"},{"why":"Provides the ASTER baseline in Table I, one of the eight compared systems.","marker":"[13]"},{"why":"Provides the ABINet baseline in Table I, one of the eight compared systems.","marker":"[14]"},{"why":"Provides the Plain-CTC baseline and the Amharic OCR sequence-modeling approach that the proposed method builds on.","marker":"[9]"},{"why":"Provides the Attn-CTC baseline in Table I, one of the eight compared systems.","marker":"[15]"}],"fun_headline_variants":["Ge'ez AI outreads humans and 8 OCR systems on handwritten text","CNN-LSTM beats human and 8 OCR baselines for Ge'ez script","Deep learning secures top score on Ge'ez handwriting benchmark","Ge'ez handwriting: AI surpasses human readers and 8 OCR tools","New Ge'ez OCR model outperforms humans and eight prior systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's superiority claim stands on the assumption that every baseline in Table I was trained and evaluated on the same HHD-Ethiopic splits with the same preprocessing and metric calculation as the proposed model; if the baselines came from different protocols, the CER/NED gaps would not be comparable.","fun_headline_variants_meta":{"raw":{"variants":["Ge'ez AI outreads humans and 8 OCR systems on handwritten text","CNN-LSTM beats human and 8 OCR baselines for Ge'ez script","Deep learning secures top score on Ge'ez handwriting benchmark","Ge'ez handwriting: AI surpasses human readers and 8 OCR tools","New Ge'ez OCR model outperforms humans and eight prior systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":3097,"prompt_tokens":869,"completion_tokens":2228,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":2131}},"tokens_in":485,"tokens_out":2228,"duration_ms":15917,"temperature":1.0,"reasoning_tokens":2131,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:30:41.756224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the experiment under one protocol: download the released model and code, retrain each of the eight baselines on the same HHD-Ethiopic IID/OOD splits, and compute CER/NED on the OOD set. Any baseline scoring below 26.95 CER or 26.50 NED, or a rerun of the proposed model producing a higher value than 26.95, would disprove the state-of-the-art claim.","supporting_citations":[{"cited_title":"Hhd-ethiopic a historical handwritten dataset for ethiopic ocr with baseline models and human-level performance,","cited_arxiv_id":null,"evidence_quote":"Supplies the HHD-Ethiopic dataset, the IID/OOD test splits, the human-performance baseline, and the HPopt baseline results that the proposed model is claimed to beat."},{"cited_title":"Aster: An attentional scene text recognizer with flexible rectification,","cited_arxiv_id":null,"evidence_quote":"Provides the ASTER baseline in Table I, one of the eight compared systems."},{"cited_title":"Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the ABINet baseline in Table I, one of the eight compared systems."},{"cited_title":"Amharic ocr: an end-to-end learning,","cited_arxiv_id":null,"evidence_quote":"Provides the Plain-CTC baseline and the Amharic OCR sequence-modeling approach that the proposed method builds on."},{"cited_title":"A blended attention-ctc network architecture for amharic text-image recognition.,","cited_arxiv_id":null,"evidence_quote":"Provides the Attn-CTC baseline in Table I, one of the eight compared systems."}],"review_version":1}