{"id":"1d8f9861-7a38-44f9-ad7e-81f4a2286bd1","arxiv_id":"2507.13959","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A ResNet50 classifier reaches 87.1% top-1 accuracy on Old Babylonian cuneiform signs, with lighting visualization and training-set diversity as the main drivers of performance.","lead":"This paper trains a machine learning model to classify handwritten cuneiform signs from Old Babylonian clay tablets, reaching 87.1% top-1 accuracy on well-attested signs. It shows that lighting angle and visualization type matter, and that models generalize better to unseen cities when trained on multiple proveniences.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 87.1% top-1 may be inflated by the per-crop 80/20 split: crops from the same tablet side appear in both train and test, so the model can exploit tablet-specific surface, lighting, handwriting, and neighboring-sign context.","rationale":"I read the paper as a careful, honest benchmark study with repeated runs, a public code repository, and genuinely informative experiments on visualization types and provenience transfer. There is no internal inconsistency in the reported pipeline and no reason to doubt the authors' good faith. The load-bearing weakness is not the choice of labels per se but the evaluation unit: the 80/20 split is performed on cropped sign instances (Section 3.4), while the paper's scientific claim is about classifying Old Babylonian signs, presumably on tablets not used in training. Because the crops are extracted from full tablet-side images and intentionally contain parts of neighboring signs (Section 3.3), the train and test sets share tablet-level background, lighting, surface, and scribal context. This creates a plausible shortcut that a ResNet50 can exploit, and it is not captured by the reported standard deviation. The reader's weakest_assumption emphasized annotator-label ambiguity, which is also real but more clearly acknowledged and harder to isolate; the reader's rationale did mention the crop-level split issue, which is why my agreement is partial rather than full. A tablet-level split is a concrete, low-cost check that directly tests whether the headline 87.1% number reflects sign recognition or tablet recognition. If the check shows a large drop, the paper should be revised or the claim qualified; if not, the current CONDITIONAL verdict stands. I therefore recommend no change to the reader's verdict.","tokens_in":17339,"tokens_out":5002,"duration_ms":69793,"concrete_test":"Use the authors' internal annotations to construct a tablet-side-level split: assign entire tablet sides (or entire tablets, if finer granularity is not feasible) to training or test, preserve the ≥20-instance class filter, and re-run the same SketchB protocol with the same seeds and training schedule. Compare the \"Fine-tune Nippur\" row and the base all-provenience row to Table 1. If top-1 accuracy drops by more than roughly 2–3 points, outside the reported ±0.3 standard deviation, crop-level leakage is real and the headline should be qualified; if accuracy is unchanged, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 states: \"For every sign category, 80% of the data examples are used as training data and 20% as test data.\" The unit is the cropped sign, not the tablet or tablet side. Because Section 3.3 crops are made from full tablet-side images and deliberately include partial neighboring signs, two crops assigned to different splits can still come from the same tablet side with identical lighting, surface texture, scribal hand, and often overlapping context. A ResNet50 can exploit such tablet-specific cues, and the reported standard deviations only cover random re-splitting of crops, not the choice of which tablets are held out. This matters most for the headline \"Fine-tune Nippur\" row (87.1 ± 0.3 top-1): fine-tuning on the same crop-level 80% split may tune the model to tablet-level appearance rather than to sign categories. The OOD Marad result (93.4% of in-distribution performance) is less affected because Marad tablets are fully withheld, but it does not validate the in-distribution numbers. The paper's own qualitative analysis, e.g., the ANSZE/GIRI3 confusion in Figure 1c and Section 4.3.2, shows that contextual cues inside the crop matter for difficult signs, so the leakage path is not merely hypothetical. The claim as currently supported is closer to \"classification of crops from tablet sides partially seen in training\" than to \"classification of unseen tablets.\"","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains ResNet50 models to classify individual cuneiform signs cropped from 2D+ visualizations of Old Babylonian tablets from three proveniences (Nippur, Dūr-Abiešuḫ, Sippar), evaluates a held-out Marad set for out-of-distribution performance, compares twelve visualization types, studies the effect of fine-tuning, and uses TSNE plots for qualitative palaeographic analysis. The headline results are a top-1 accuracy of 87.1% ± 0.3 and top-5 accuracy of 96.5% ± 0.1 on the Nippur test set after fine-tuning, with base-model accuracies of 77.2–85.6% top-1 across the three in-distribution proveniences and an OOD performance of 93.4% of the in-distribution performance when training on all three.","tokens_in":17728,"tokens_out":3966,"duration_ms":45727,"significance":"If the evaluation is trustworthy, this is a useful first benchmark for automated Old Babylonian sign classification. The paper's strengths include repeated runs with reported standard deviations, a systematic comparison of lighting angles and depth visualizations, an external OOD test set (Marad) completely withheld from training, and a publicly available code repository and suggestion tool. The results also speak directly to data acquisition standards for cuneiform tablets, which is a practical contribution to digital Assyriology.","major_comments":[{"comment":"The 80/20 split is performed per sign category at the level of cropped signs, not at the level of tablets or tablet sides. Because crops are square bounding boxes that deliberately include partial neighboring signs (§3.3), crops from the same tablet side can appear in both training and test partitions, sharing lighting, surface texture, scribal hand, and often overlapping context. A ResNet50 can exploit such tablet-specific cues, so the reported accuracies, especially the fine-tuned Nippur row in Table 1 (87.1% ± 0.3), may be optimistic. The standard deviations only reflect random re-splitting of crops, not the choice of which tablets or tablet sides are held out. Please report results with a tablet-level or tablet-side-level split (for example, leave-one-tablet-out or grouped split), or at minimum quantify the fraction of test crops that share a tablet side with training crops.","section":"§3.4"},{"comment":"The class labels are contextual readings assigned by human annotators using the Nuolenna sign list. As the paper acknowledges, many signs have multiple readings and visually similar signs from different classes can overlap, as the ANSZE/GIRI3 confusion in §4.3.2 shows. This means the reported accuracy conflates visual classification with the annotators' context-based reading decisions. The paper should report inter-annotator agreement on a subset of signs, or at least explicitly discuss how label noise and class overlap affect the interpretation of the accuracy numbers. Without this, the absolute accuracies are difficult to interpret as purely palaeographic classification performance.","section":"§3.2.2, §3.1.1"},{"comment":"The fine-tuning procedure is not fully specified. The text states that the base model is 'fine-tuned using either all data or only the data of one specific provenience,' but it does not explicitly state that the held-out 20% test crops are excluded from the fine-tuning set. If the test partition is included in fine-tuning, the results in Table 1 would be invalid; if not, this should be stated clearly. Please clarify the exact data split used for each fine-tuning experiment, including whether the test set is the same 20% crop-level partition described in §3.4.","section":"§4.3.3, Table 1"}],"minor_comments":[{"comment":"In the related work section, 'Cobanaglu' should be 'Cobanoglu' to match reference [7].","section":"§2"},{"comment":"The phrase 'do to their combined state' should be 'due to their combined state.'","section":"§4.3.1"},{"comment":"Reference [8] contains a typo in the title: 'Old Babylonian Peiod' should be 'Old Babylonian Period.'","section":"References"},{"comment":"The data availability statement says 'The data and scripts published with this paper can be found on our Zenodo/Github page (link),' but the link is a placeholder. Please provide the actual DOI or URL for the dataset and scripts.","section":"§3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid applied ML contribution for a digital humanities audience, but the crop-level split is a serious threat to the validity of the headline accuracy claims. This is fixable by re-running the experiments with a tablet-level split, which is within the scope of a revision. I would also encourage the editor to ensure the data availability placeholder is resolved before publication, since the paper's reproducibility partly depends on access to the annotated dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things to know. First, this is the first stand-alone Old Babylonian sign classification benchmark, and it is a genuinely useful empirical study of how imaging choices affect model performance. Second, the headline 87.1% top-1 accuracy is a crop-level number: the 80/20 split is per sign instance, not per tablet, so crops from the same tablet side appear in both train and test. That almost certainly inflates the in-distribution results. Third, the out-of-distribution result (Marad, fully held out) is the most trustworthy number in the paper, and it is still good: training on three proveniences brings OOD to 93.4% of in-distribution performance.\n\nNow credit where it is earned. The systematic comparison of 12 visualizations is new and actionable: they show lighting from top/bottom beats left/right, normal maps and sketches beat photorealistic renders, and they give concrete suggestions for acquisition. The provenience transfer experiments are thoughtful, and the TSNE analyses are used carefully, not oversold. The paper is honest about limitations, including the contextual-reading label issue and the small number of rare signs. Code is public, and data is promised though not yet linked.\n\nThe soft spots are in the evaluation split and the missing data link. The stress-test concern about per-crop splitting holds up: because crops include neighboring signs and share tablet-specific surface, lighting, and handwriting, the model can memorize tablet-level cues. The reported standard deviations only cover random re-splitting of crops, not held-out tablets. The authors should either provide a tablet-level split or clearly label the metric as “crop classification within partially seen tablets.” The missing dataset link/DOI is minor but should be fixed before publication. The labeling assumption—that contextual readings are the ground truth—is inherent to the task and acknowledged; it doesn't undermine the benchmark but should be kept in mind when interpreting accuracy.\n\nThis deserves peer review. The issues are addressable, and the benchmark fills a real gap. A referee should ask for a tablet-level split or a clear statement of what the numbers mean, and for the dataset link. I'd cite it if I worked in this area, and I'd bring it to a reading group as an example of a domain-specific benchmark done carefully enough.","headline":"First Old Babylonian cuneiform sign classification benchmark with genuinely useful imaging and transfer experiments, but the headline accuracy is crop-level and likely inflated by a per-crop split that leaks tablet-level cues.","tokens_in":18148,"tokens_out":2118,"would_cite":true,"duration_ms":23399,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A standard image classifier trained on depth renderings of Old Babylonian clay tablets identifies cuneiform signs with 87.1% top-1 accuracy after fine-tuning, and training across three proveniences lifts out-of-distribution accuracy to…","keywords":["cuneiform sign classification","Old Babylonian","ResNet50","2D+ visualization","handwritten text recognition","transfer learning","cultural heritage","palaeography"],"falsifier":"Take a random set of crops from the test data, have two trained Assyriologists independently assign sign classes without seeing the corpus labels, and compare each human's agreement with the model's accuracy against the same reference labels. If human-human agreement is no higher than human-model agreement, the $87.1\\%$ figure is partly an artifact of ambiguous labels rather than of visual recognition.","tokens_in":17130,"feed_emoji":"🏺","tokens_out":5728,"duration_ms":64574,"temperature":0.7,"pith_summary":"This paper establishes the first automatic classification benchmark for handwritten Old Babylonian cuneiform signs, using a standard ResNet50 network on crops from 2D+ visualizations of tablets from Nippur, Dūr-Abiešuḫ, and Sippar. It reports $87.1\\% \\pm 0.3$ top-1 accuracy after fine-tuning on the target provenience, with top-5 accuracy of $96.5\\%$. The central claim is that sign shape varies enough across cities and imaging conditions to dominate the difficulty, and that training on several proveniences nearly closes the gap to unseen sites. The authors argue the practical path forward is acquiring depth-based visualizations and more geographically varied data rather than more of the same.","feed_headline":"A neural net reads Old Babylonian signs at 87 percent accuracy","feed_subtitle":"The first benchmark for automatic sign reading on Old Babylonian tablets shows lighting and provenance, not model size, decide accuracy.","key_machinery":"The load-bearing object is a ResNet50 image classifier trained on square crops of individual signs cut from 2D+ visualizations, where the crucial input type is the SketchB visualization: a rotation-invariant sketch derived from the surface normal map. The argument is carried by controlled training-set ablations over visualizations (twelve render types), proveniences (single versus combined), and a fine-tuning stage that removes augmentations and lowers the learning rate to match the test distribution. A secondary mechanism is the TSNE projection of the 2048-dimensional penultimate-layer features, used to show that clusters correspond to script variants documented in sign lists.","core_discovery":"The authors claim that a standard convolutional network can classify Old Babylonian cuneiform signs cut from dome-captured visualizations, provided the rendering exposes depth: non-photorealistic sketches and normal maps beat any single lighting angle by 4–5 percentage points. The model reaches $87.1\\% \\pm 0.3$ top-1 on Nippur after fine-tuning, and the base model ranges $77.2\\%$ to $85.6\\%$ across Dūr-Abiešuḫ, Nippur, and Sippar. Training on all three proveniences raises accuracy on the held-out Marad site to $93.4\\%$ of in-distribution performance, whereas single-provenience training drops to $51.4\\%$–$79.0\\%$ of in-distribution levels. Qualitative TSNE analysis shows that the model's feature space mirrors palaeographic variants that Assyriologists recognize, including the known ANSZE/GIRI3 overlap. The paper concludes that this is a viable assistive tool for reading tablets, with top-5 performance strong enough for a suggestion tool, while top-1 remains limited without context.","pith_inferences":["The authors do not test legacy flatbed scans or photographs; a direct extension would be to measure how much accuracy drops on such images and then apply domain adaptation from SketchB renderings to legacy data.","The embedding space that TSNE visualizes could be turned into a quantitative palaeographic instrument, for example dating undated tablets by nearest-neighbor positions among dated exemplars—something the paper only gestures at.","The out-of-distribution claim rests on a single unseen site, Marad; a stronger stress test would hold out an entire region or a chronologically distinct corpus, which could lower the $93.4\\%$ figure.","Because labels are contextual readings, the accuracy ceiling may be set by inter-annotator agreement; measuring that agreement would place the reported scores in perspective."],"forward_implications":["A top-5 accuracy of $96.5\\%$ means a human-in-the-loop tool can offer five likely signs per crop, which is already useful for reading unfamiliar proveniences.","Including multiple proveniences in training is the single most effective step for generalizing to unseen sites, raising out-of-distribution accuracy to $93.4\\%$ of in-distribution performance.","Fine-tuning the base model on the target provenience's own data improves results for every city tested, making specialization a cheap final step after broad training.","Acquisition standards should prioritize depth-revealing visualizations (normal maps, sketches) and variety of provenience over raw image quantity.","Signs on the curved left and right edges of tablets are the hardest to classify under directional light, so even illumination or depth-based renderings are preferable for those zones."],"supporting_citations":[{"why":"Supplies the ResNet50 architecture used for all classification experiments in the paper.","marker":"[13]"},{"why":"Supplies the 2D+ visualization methodology and the naming convention for the twelve render types, including SketchA and SketchB.","marker":"[11]"},{"why":"Supplies the White Light Portable Light Dome acquisition system that produced the photometric data behind the visualizations.","marker":"[32]"},{"why":"Prior work on illumination-based augmentation and lighting transfer that motivates the lighting-angle experiments.","marker":"[23]"},{"why":"Supplies the Nuolenna Unicode-based sign list used to define the 206 sign classes.","marker":"[16]"},{"why":"Labat's palaeographic sign list used to validate that the model's clusters match recognized script variants.","marker":"[17]"},{"why":"Supplies the t-SNE algorithm used to visualize the 2048-dimensional feature representations of signs.","marker":"[30]"},{"why":"Supplies the Cuneur annotation software in which sign polygons and contextual readings were created.","marker":"[14]"}],"fun_headline_variants":["Cuneiform AI: 87% accuracy on Old Babylonian signs","Neural net deciphers ancient cuneiform with depth cues","First benchmark: AI reads Old Babylonian tablets at 87%","Deep learning reads cuneiform signs better with depth","AI classifies Old Babylonian cuneiform signs at 87%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies assume that each cropped sign has exactly one correct class, namely the contextual reading assigned by a human annotator in the Cuneur software; if those readings are inconsistent or the Unicode-based sign list splits or merges signs differently from actual palaeography, every accuracy number inherits that bias.","fun_headline_variants_meta":{"raw":{"variants":["Cuneiform AI: 87% accuracy on Old Babylonian signs","Neural net deciphers ancient cuneiform with depth cues","First benchmark: AI reads Old Babylonian tablets at 87%","Deep learning reads cuneiform signs better with depth","AI classifies Old Babylonian cuneiform signs at 87%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1394,"prompt_tokens":1006,"completion_tokens":388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":622,"tokens_out":388,"duration_ms":4318,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:11:05.430192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random set of crops from the test data, have two trained Assyriologists independently assign sign classes without seeing the corpus labels, and compare each human's agreement with the model's accuracy against the same reference labels. If human-human agreement is no higher than human-model agreement, the $87.1\\%$ figure is partly an artifact of ambiguous labels rather than of visual recognition.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet50 architecture used for all classification experiments in the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 2D+ visualization methodology and the naming convention for the twelve render types, including SketchA and SketchB."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the White Light Portable Light Dome acquisition system that produced the photometric data behind the visualizations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on illumination-based augmentation and lighting transfer that motivates the lighting-angle experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Labat's palaeographic sign list used to validate that the model's clusters match recognized script variants."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the t-SNE algorithm used to visualize the 2048-dimensional feature representations of signs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Cuneur annotation software in which sign polygons and contextual readings were created."}],"review_version":1}