{"id":"3405c735-1c05-400f-ba71-6d8d65adbe7a","arxiv_id":"2505.04678","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Five pretrained CNN models are trained on a cuneiform dataset and evaluated on Hammurabi's Law 1, with reported internal accuracies above 98% and translations made by a lookup table.","lead":"This paper benchmarks five standard deep learning models for recognizing cuneiform symbols and reports near-perfect test accuracy, then applies the two best models to a scanned image of Hammurabi's Law 1. A generalist might read it to see whether off-the-shelf image classifiers are ready to help decipher ancient scripts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant new concern beyond the reader's: the reported 98.31% Hammurabi accuracy lacks any demonstrated character-level split, so the central claim of generalization is unsupported by the paper as written.","rationale":"The reader's REJECT verdict is well supported. My stress-test pass confirmed the central claim and isolated the same load-bearing weakness: the paper does not demonstrate character-disjoint splits, and without that, the 98.31% and 88.87% accuracies on Hammurabi Law 1 could simply reflect the models memorizing augmented variants of the same source characters that also appear in training. No internal inconsistency is needed to see the gap: the paper states the split only as percentages and counts of images, never as a split of the 235 characters. The dataset description also leaves a minor arithmetic ambiguity (235 characters x 10 preprocessed representations = 2,350, and augmentation by 5 more per character would give 14,100 only if the '5 more' are added to 55 per-character variants, which is not stated), but this is secondary to the leakage concern. Table II's near-perfect metrics on the training-domain test set are consistent with memorization when augmented near-duplicates appear in both training and test. The Hammurabi evaluation is a small, manually labeled set of 35 characters, so the 98.31% result is a point estimate with no uncertainty, further weakening the generalization claim. The translation portion is explicitly database lookup, so it does not validate recognition accuracy. The central claim therefore stands or falls on the unseen-character generalization assumption, and the paper provides no evidence for it. My concern is fully aligned with the reader's weakest_assumption, and the concrete test I propose would settle it by forcing a character-disjoint split. For these reasons I recommend no change to the reader's verdict.","tokens_in":6254,"tokens_out":1865,"duration_ms":16169,"concrete_test":"Obtain the dataset and training script (or the released notebook if the authors provide it) and re-run the exact Table I split with a modified splitter that guarantees all augmented variants of each of the 235 source characters stay within the same partition. Then re-train EfficientNetV2M and VGG16 with identical hyperparameters and report test accuracy on the character-disjoint split. If the Hammurabi Law 1 accuracy drops materially (e.g., from 98.31% toward chance or below 80%), the reported accuracy is an artifact of within-character leakage and the central claim of automated cuneiform-reading generalization fails. Alternatively, if the authors produce a per-character split manifest and the 98.31% figure survives, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EfficientNetV2M and VGG16 trained on the 14,100-image cuneiform dataset transfer to a scanned image of Hammurabi Law 1 at 98.31% and 88.87% accuracy, and that the recognized symbols support precise English translation. The load-bearing assumption is that the train/test/validation split in Table I is character-disjoint, so test accuracy measures generalization to unseen cuneiform signs rather than memorization of near-duplicate augmented crops of the same 235 source characters. The paper never states that all augmented variants of a given source character are confined to one split; Section II describes loading 235 distinct cuneiforms, preprocessing them into 10 varied representations, then augmenting with 5 more per character (14,100 = 235 x 60 is also not explicitly reconciled, and the reported 40/24/36 percentages do not match the 5,076/3,384/5,640 counts). The reader's weakest_assumption correctly targets this character-level leakage. Additionally, the 98.31% result is computed on only 35 segmented characters with manually chosen ground truth labels, no per-symbol confusion analysis, and no error bars; Table II reports only aggregate metrics on the training-domain test set. The translation component is explicitly a lookup from a predetermined database after recognition, so the linguistic claim adds no independent support for the perception claim. Because the dataset and code are not released, the character-level split cannot be verified externally; the paper itself contains no sentence stating that splits respect character identity. Thus the generalization claim is the least secure part of the argument, and the reader's rejection is appropriate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated pipeline for recognizing cuneiform symbols from images. Five deep learning models (VGG16, EfficientNetV2M, MobileNet, InceptionResNetV2, and a custom 2D CNN) are trained on a dataset claimed to contain 14,100 cuneiform character images derived from 235 source characters. The two best models are then applied to a scanned image of Hammurabi's Law 1, with reported accuracies of 98.31% (EfficientNetV2M) and 88.87% (VGG16) over 35 segmented characters. The recognized symbols are mapped through a lookup table to Akkadian and English translations, and a qualitative linguistic comparison between Akkadian and Arabic is presented.","tokens_in":6566,"tokens_out":4076,"duration_ms":38925,"significance":"If the central claim is supported, the paper would demonstrate that standard image classification architectures can recognize cuneiform signs from scans and support automated translation, which is valuable for digital archaeology and computational linguistics. The systematic comparison of five models and the use of an external evaluation image are constructive steps. However, the significance is conditional: the near-perfect training-domain accuracies and the Hammurabi evaluation results are not credible as evidence of generalization unless the dataset split and evaluation methodology are clarified. The paper does not ship code or data, so the results cannot currently be reproduced or independently verified.","major_comments":[{"comment":"The dataset description is internally inconsistent. The text states that 235 distinct cuneiforms were loaded, preprocessing yielded 10 varied representations per character, and augmentation added five more symbols for each character. This implies 235 × (10 + 5) = 3,525 images, not 14,100. Table I reports 5,640 test, 3,384 validation, and 5,076 training images, which sum to 14,100 and match the stated 40/24/36 percentages, but the generation process for reaching 14,100 from 235 source characters is not explained. Please clarify the exact number of images produced at each preprocessing and augmentation step.","section":"Section II"},{"comment":"The paper does not establish that the train/validation/test split is character-disjoint. The text describes splitting the dataset by percentage, but the evaluation goal is to measure generalization to unseen cuneiform signs. If augmented variants of the same source character appear in both training and test sets, the near-perfect accuracies in Table II (0.9857–0.9999) can be inflated by memorization of near-duplicate images. Please state explicitly whether all images derived from a single source character are confined to exactly one split, and if not, re-run the experiments with a character-level split. Releasing the dataset and code would also allow verification of this point.","section":"Section II and Table I"},{"comment":"The Hammurabi Law 1 evaluation relies on only 35 segmented characters with manually chosen ground truth labels, yet the paper reports only aggregate accuracy percentages. With 35 samples, a single misclassification changes accuracy by 2.86 percentage points, so the difference between the reported 98.31% and 88.87% corresponds to about three to four characters. Please provide per-character prediction results, a confusion matrix, and a description of how the ground truth labels were established and by whom. Without this information, the reported accuracies are not statistically meaningful.","section":"Section V-C"},{"comment":"The English translation is produced by mapping recognized symbols to a predetermined database, so the translation output does not provide independent evidence for the accuracy of the visual recognition system. The linguistic comparison in Section IV and Table III is qualitative and does not constitute a quantitative validation of the recognition or translation. Please either present a separate evaluation of the translation component or temper the claims to reflect that the translations are deterministic lookups contingent on the recognition results.","section":"Section V-D"}],"minor_comments":[{"comment":"References [7] and [14] are identical; please deduplicate and cite the earlier workshop paper consistently.","section":"References"},{"comment":"The methodology states that the study examined CNNs, RNNs, and hybrid models, but no RNN or hybrid architecture is implemented or evaluated in the paper. Please correct this description.","section":"Section III-A"},{"comment":"The text refers to correctly identified characters being highlighted in green, but the figure is not reproduced in color in the manuscript image. Please ensure the figure is legible and the highlighting is described in the caption.","section":"Figure 4"},{"comment":"The phrase 'unsupervised testing' is misleading; the evaluation uses ground truth labels and thus is supervised. Please rephrase.","section":"Section V-C"},{"comment":"The reported metrics include no uncertainty estimates. If multiple runs are performed, report means and standard deviations; otherwise, state that a single run was performed.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an extension of the authors' prior workshop paper (reference [7]/[14]), but the relationship is not disclosed explicitly beyond self-citation. The lack of code and data release, combined with the unresolved dataset inconsistency and the missing character-level split description, means that the central claim of generalization is not currently supported. These issues are fixable in principle, but the revision must address them concretely rather than cosmetically."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Straight to the point: this is a benchmark of five standard CNN backbones on a small cuneiform dataset, with an extra test on a scan of Hammurabi's Law 1. The reported numbers look too good to be true, and the paper gives you enough reason not to trust them.\n\nThe dataset section says 14,100 images, but it also says 235 characters with 10 variants plus 5 augmentations each, which comes to 3,525. That is a hard internal inconsistency. The train/validation/test split is 36/24/40, with more held-out than training data, and the paper never says the split was done by character rather than by image. If augmented crops of the same source sign land in both training and testing, the 0.9999 accuracy is mostly memorization. The Hammurabi evaluation uses 35 manually segmented characters with manually chosen labels, no error bars, no confusion matrix, and the \"translation\" is a database lookup after recognition. So the headline claim of generalized cuneiform reading is not supported by the paper as written.\n\nCredit where due: the paper is clearly written, the architectures are chosen sensibly, and testing on a real inscription is the right idea. The authors also engage with recent work on cuneiform detection and transliteration. If the split issue were fixed and the data released, this could be a modestly useful engineering contribution for digital archaeology.\n\nThe soft spots are not minor. The inconsistent dataset count and the likely leak undermine the empirical contribution. Without code or data, there is no way to verify. The linguistic comparison is decorative, and the self-citation is fine but does not rescue the methodology.\n\nWho is this for? Maybe a reader wanting a quick overview of what pre-trained CNNs do on cuneiform. But as a research claim it fails. I would desk reject it with an invitation to resubmit after fixing the dataset description, performing a character-disjoint split, and releasing the data and code.\n\nMy recommendation: do not send to referees. A desk reject with a specific note about the leakage concern is the right outcome.","headline":"Routine CNN benchmark on cuneiform whose headline results are undermined by a dataset count mismatch and a likely character-level leakage; desk reject rather than referee.","tokens_in":7121,"tokens_out":3139,"would_cite":false,"duration_ms":30519,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims EfficientNetV2M reads cuneiform from a scan at 98.31% accuracy and translates it into English.","keywords":["cuneiform","deep learning","symbol recognition","Akkadian","Hammurabi's Law","transfer learning","optical character recognition","computational linguistics"],"falsifier":"Re-train EfficientNetV2M under identical settings but split the 235 source characters into disjoint training and test sets; if accuracy on held-out characters or on Hammurabi's Law 1 drops far below 98.31%, the reported result depended on seeing augmented copies of the same signs during training.","tokens_in":6076,"feed_emoji":"🏺","tokens_out":9079,"duration_ms":85240,"temperature":0.7,"pith_summary":"This paper tries to establish that a fully automated deep-learning pipeline can recognize cuneiform symbols from a scanned tablet and convert them into English. It trains five convolutional architectures on a dataset of 14,100 augmented images representing 235 cuneiform characters, then applies the two best models to a scan of Hammurabi's Law 1. The paper reports 98.31% accuracy for EfficientNetV2M and 88.87% for VGG16 on the law's 35 segmented characters. If this holds, archaeologists and historians could machine-read well-preserved cuneiform texts far faster than manual transcription.","feed_headline":"Deep learning reads cuneiform at 98 percent accuracy","feed_subtitle":"EfficientNetV2M and VGG16 turn a scan of Hammurabi's Law 1 into Akkadian and English text.","key_machinery":"The central mechanism is a recognition-and-lookup cascade: contour detection splits a tablet scan into individual signs; a pretrained convolutional network fine-tuned on cuneiform classes each sign as one of 235 symbols; and a fixed cuneiform-to-Akkadian-to-English dictionary turns the predicted class into a translated word. The classifier does the heavy lifting, since the paper's accuracy figures describe it, while the dictionary does the linguistic work. Transfer learning is the enabling assumption: starting from networks pretrained on ordinary images lets the relatively small cuneiform dataset train deep models without starting from scratch.","core_discovery":"On its own terms, the paper reports that standard convolutional image classifiers, transfer-learned from pretrained weights, can identify cuneiform signs on a scanned page of Hammurabi's Law 1. Trained on 14,100 augmented images representing 235 distinct cuneiform characters, EfficientNetV2M achieved 98.31% accuracy on the law's 35 segmented characters and VGG16 achieved 88.87%, while the other three architectures scored above 98.5% on the random test split. The recognized characters are passed through a lookup table that gives their Akkadian readings and English meanings, and the paper shows the reconstructed first words of the law. It also draws lexical parallels between Akkadian and Arabic as supporting linguistic context.","pith_inferences":["A stricter evaluation the paper does not report would hold out entire source characters rather than individual augmented images; until that is run, the 98.31% figure should be read as an upper bound on generalization.","Testing on artificially eroded or cracked tablets would show whether the pipeline survives real archaeological damage, since the paper evaluates one clean scan.","The Akkadian-Arabic parallels in Table III could be strengthened by a systematic sound-correspondence analysis; a handful of cognate-like pairs is suggestive but not demonstrative.","Replacing the fixed lookup table with a context-sensitive language model could resolve signs that the per-character classifier finds ambiguous."],"forward_implications":["A scanned cuneiform text can be converted into machine-readable sign labels without manual transcription, as long as its signs belong to the 235-character training inventory.","The per-sign lookup design means translation quality is bounded by per-sign recognition: at 98.31% accuracy on Law 1, nearly every sign is correct and the reader can spot the few failures.","The same training and evaluation procedure could be applied to other Akkadian legal texts, and eventually to other cuneiform languages, by expanding the sign inventory.","Because five architectures trained under identical conditions all exceed 98.5% accuracy on the random test split, the reported effect is not specific to a single model."],"supporting_citations":[{"why":"Supplies the 235 cuneiform characters used to build the 14,100-image dataset and the scanned Hammurabi's Law 1 image used for evaluation.","marker":"[10]"},{"why":"The authors' earlier VGG16 transfer-learning study provides the recognition approach this paper extends with four additional architectures.","marker":"[7]"},{"why":"Demonstrates CNN-based detection of cuneiform signs from 3D renderings and photographs, the detection baseline the method adopts.","marker":"[4]"},{"why":"Shows CNNs recognize cuneiform symbols in fragmented and partial inscriptions, supporting the goal of processing damaged artifacts.","marker":"[5]"},{"why":"Establishes automated transliteration from parallel lines with sparse data, the predecessor step for turning recognized signs into readable text.","marker":"[6]"},{"why":"Provides a machine-learning classification baseline using unigram features on a balanced cuneiform dataset, against which the deep models are implicitly compared.","marker":"[9]"}],"fun_headline_variants":["AI decodes cuneiform on Hammurabi's Law with 98% accuracy","Deep learning translates cuneiform to English at 98% accuracy","EfficientNetV2M reads cuneiform signs at 98.31% accuracy","Deep learning reads cuneiform from Hammurabi's Law","Cuneiform signs translated automatically with deep learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the assumption that the training, validation, and test images are pictures of different cuneiform characters, because the paper only reports how many images went into each set, not how the 235 source characters were divided; if the same character appears in both training and testing, the model can memorize rather than recognize.","fun_headline_variants_meta":{"raw":{"variants":["AI decodes cuneiform on Hammurabi's Law with 98% accuracy","Deep learning translates cuneiform to English at 98% accuracy","EfficientNetV2M reads cuneiform signs at 98.31% accuracy","Deep learning reads cuneiform from Hammurabi's Law","Cuneiform signs translated automatically with deep learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000691,"raw_usage":{"total_tokens":3080,"prompt_tokens":851,"completion_tokens":2229,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":2134}},"tokens_in":467,"tokens_out":2229,"duration_ms":15356,"temperature":1.0,"reasoning_tokens":2134,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:30:34.544863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train EfficientNetV2M under identical settings but split the 235 source characters into disjoint training and test sets; if accuracy on held-out characters or on Hammurabi's Law 1 drops far below 98.31%, the reported result depended on seeing augmented copies of the same signs during training.","supporting_citations":[{"cited_title":"Marcus, A manual of Akkadian","cited_arxiv_id":null,"evidence_quote":"Supplies the 235 cuneiform characters used to build the 14,100-image dataset and the scanned Hammurabi's Law 1 image used for evaluation."},{"cited_title":"Fragmented-cuneiform-based convolu- tional neural network for cuneiform character recognition,","cited_arxiv_id":null,"evidence_quote":"Shows CNNs recognize cuneiform symbols in fragmented and partial inscriptions, supporting the goal of processing damaged artifacts."},{"cited_title":"Automating Transliteration of Cuneiform from Parallel Lines with Sparse Data,","cited_arxiv_id":null,"evidence_quote":"Establishes automated transliteration from parallel lines with sparse data, the predecessor step for turning recognized signs into readable text."},{"cited_title":"Classifying cuneiform symbols using machine learning algorithms with unigram features on a balanced dataset,","cited_arxiv_id":null,"evidence_quote":"Provides a machine-learning classification baseline using unigram features on a balanced cuneiform dataset, against which the deep models are implicitly compared."}],"review_version":1}