{"id":"cbcd62af-cc5a-4c95-bcbd-27c64e070f53","arxiv_id":"2608.13343","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AmalthAI is a self-hostable, no-code computer vision platform for cultural heritage that was used by archaeologists to train and validate classification and segmentation models on clay textile imprints.","lead":"AmalthAI is an open-source, self-hostable computer vision platform that lets cultural heritage experts train, evaluate, and interpret classification, segmentation, and detection models without programming. A case study on clay textile imprints shows domain experts can build and validate models, with added Grad-CAM and vision-language explanations for archaeology.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing concern is covariate leakage: high accuracy may reflect clay texture, lighting, or camera differences correlated with textile class rather than imprint features; the paper's own Sec. 4.3 caveat leaves the archaeological conclusion unsupported without leakage-controlled tests.","rationale":"The reader's weakest assumption aligns with the point I find most load-bearing. The platform architecture, self-hosting, and model repository are plausible and code is linked, which supports the systems contribution; but the paper's scientific payoff depends on the case study, and the case study's inference is underdetermined. The only evidence that imprints encode material and technique is classification accuracy on the same controlled dataset; the dataset was created by impressing different textiles into clay and imaging them, and Fig. 5 advertises variation in clay texture, lighting, and camera settings. If those covariates are imbalanced across classes, a shortcut classifier could achieve the reported accuracy without reading the imprint. Repeated random splits in Tab. 2 cannot remove a covariate that is constant across the split. The authors' own Sec. 4.3 caveat concedes the point. The usability claim is also self-reported by the archaeologist co-authors, so the 'independently' part of the abstract is not yet evidenced; however, the dataset-artifact issue is the more fundamental threat because it would invalidate the archaeological conclusion even if usability were proven. I therefore agree with the CONDITIONAL verdict; the conditions should include a leakage-controlled evaluation (masked-ROI or cross-batch training) and release of the dataset and per-image metadata.","tokens_in":15085,"tokens_out":5598,"duration_ms":59705,"concrete_test":"Take the trained segmentation model and apply it to all classification images; use the highest-confidence mask to crop or zero out everything outside the imprint region, then retrain and test the best material and technique classifiers on the masked images with the same split protocol as Tab. 2. If masked accuracy falls toward chance while full-image accuracy stays high, the reported results were driven by imprint-external covariates. As a second check, partition the dataset by acquisition session or clay batch and test cross-batch; a large drop would confirm session-level leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central archaeological claim requires that classification accuracy is driven by textile structure preserved in the imprint. The experiments do not rule out an alternative: the material and technique classes may be confounded with acquisition covariates. Fig. 5 explicitly states the dataset spans 'different clay textures, lighting conditions and camera settings'; the paper reports no per-image metadata, no balance check across classes for these covariates, and no cross-session or cross-clay-batch generalization. Classification was trained on full images, not on the segmented imprint ROI, so the model can use background clay texture, illumination direction, or camera settings as shortcuts. The repeated random splits in Tab. 2 do not address this, because a covariate that is stable within each class survives random splitting. Section 4.3 concedes the results 'do not by itself exclude reliance on dataset specific cues.' Since the hypothesis test's success condition is that imprints themselves carry label-relevant information, this unfalsified alternative leaves the headline archaeological finding unsecured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AmalthAI, an open-source, self-hostable web-based machine learning platform aimed at cultural heritage (CH) researchers without machine learning expertise. The platform supports image classification, semantic segmentation, and object detection, with dataset management, training via Kubeflow/Katib, hyperparameter search, Grad-CAM explanations, and a vision-language assistant. The authors validate the platform on an experimental archaeology case study: textile imprints on clay. They report segmentation mIoU around 90%, material classification accuracy of 77.53%, and production-technique accuracy of 85.57%, with repeated random splits yielding consistent results. They argue these results support the hypothesis that clay imprints preserve visual features informative of the original textile's raw material and production technique, while acknowledging that the results do not by themselves exclude reliance on dataset-specific cues.","tokens_in":15248,"tokens_out":4704,"duration_ms":48326,"significance":"If the platform works as described, it addresses a real accessibility gap: CH researchers often cannot configure, train, and evaluate CV models, and many restricted heritage datasets cannot be uploaded to commercial cloud platforms. The paper's engineering is detailed and credible: the architecture is modular, the code is released, inference is run in ephemeral containers matching the training environment, storage is content-addressed and supports on-premise deployment, and the multi-user/OAuth design is sensible. The repeated-split reporting with means and standard deviations is stronger than a single-split evaluation. However, the central archaeological finding is not yet rigorously established: the case-study dataset spans different clay textures, lighting conditions, and camera settings, and the classification models were trained on full images, leaving open the possibility that accuracy reflects acquisition covariates rather than textile-structure information. The usability evidence also rests on self-evaluation by co-author experts. These issues are fixable, but they currently limit the strength of the paper's headline claims.","major_comments":[{"comment":"The headline archaeological conclusion is not secured against covariate leakage. The dataset is described in Fig. 5 as spanning different clay textures, lighting conditions and camera settings, yet the paper reports no per-image metadata, no balance checks across these covariates, and classification is trained on full images rather than on the segmented imprint region. Because a covariate that is stable within a material or technique class survives repeated random splits, the reported 77.53% material and 85.57% technique accuracies could reflect background or acquisition cues rather than textile-structure information in the imprints. The authors' own caveat in Sec. 4.3 ('does not by itself exclude reliance on dataset specific cues') is to the point. I request leakage-controlled evidence: classification restricted to the segmented imprint ROI, cross-clay or cross-lighting generalization experiments, or explicit demonstration that covariates are balanced across classes.","section":"Sec. 4.3, Table 3, Fig. 5"},{"comment":"The usability claim that CH experts with minimal ML knowledge can independently produce and validate findings is supported only by a self-evaluation by co-author experts. The experts who ran the experiments are also authors of the platform and of the prior framework [3]; no independent participant pool, task-completion metrics, time-on-task, error rates, or comparison against a baseline workflow is reported. This is not fatal for a systems paper, but the phrasing 'enabling non-ML CH experts to independently produce and validate archaeologically meaningful findings' in the Abstract overstates the evidence. Please either soften the claim or add a structured usability assessment.","section":"Sec. 4.2"},{"comment":"The hypothesis success condition requires that misclassifications be attributable to structural similarities between textile classes, but the only evidence is qualitative expert opinion. Table 3 shows nettle and splicing have lower accuracies, and the text states these are 'according to CH experts' difficult cases, without a formal confusion-pattern analysis, a human-baseline comparison, or pre-registered criteria. This leaves the link between model errors and domain-valid similarity anecdotal and compounds the confound identified in the previous comment.","section":"Sec. 4.1, Sec. 4.3"}],"minor_comments":[{"comment":"Please clarify why EfficientNetB0 material accuracy is 76.32±1.21 in Table 2 but 77.53% in Table 3; if Table 3 uses the best split or a different test set, state this explicitly and provide variance for the reported inference results.","section":"Tables 2 and 3"},{"comment":"The sentence 'with only a few samples outside the defined classes' is vague; report exact counts or explain how such samples were treated in the experiments.","section":"Sec. 4.2"},{"comment":"The VLM system prompt forbids describing materials as 'clay', but the examples in Figures 8 and 10 describe 'clay imprint' and 'clay or similar material'; the prompt-output inconsistency should be resolved or explicitly discussed.","section":"Sec. 3.4 and Figures 7–10"},{"comment":"The case-study dataset is not released, so the quantitative results cannot be reproduced independently; given the data-sovereignty motivation this is understandable, but the paper should state explicitly which parts of the case study can be reproduced from the public code and which require the private dataset.","section":"General"},{"comment":"The phrase 'demonstrating its practical effectiveness and usability' is asserted from a single case study; consider 'demonstrated in a case study' to avoid overclaiming generalizations about usability.","section":"Sec. 1, contribution 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is the platform, and the archaeological case study is presented as its validation. The covariate-leakage issue directly affects the case-study conclusion, but it is addressable: adding ROI-only classification, cross-condition generalization, or covariate-balance analysis would substantially strengthen the paper. A structured usability study with non-co-author participants would also be needed to justify the 'independently produce and validate' framing. I would not reject the paper; the platform itself appears well-engineered and relevant to the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: AmalthAI is a real, self-hosted, no-code CV platform for cultural heritage labs, and the paper is upfront that the experiments are workflow validation, not benchmark claims. The platform is the contribution, and it looks solid: dockerized, multi-user with federated identity, three tasks, Katib hyperparameter search, and a Grad-CAM plus VLM explanation loop for expert review. The code is on GitHub, the architecture is described in enough detail to reproduce, and the authors explicitly state that the numerical results are not a benchmark claim.\n\nThe new thing here is the assembly. Prior work from the same group was a single-user binary segmentation framework; this extends to multiclass classification, detection, multi-user identity, and the Grad-CAM+VLM review loop. Being self-hostable matters for rights-encumbered data, and the storage abstraction with local fallback is a real design point.\n\nThe case study is a good-faith attempt to show non-ML archaeologists can run the whole pipeline. The reported segmentation mIoU around 90% and classification around 77-85% are plausible. The class-level misclassification patterns (nettle, splicing) aligning with what archaeologists find visually is the kind of evidence that makes the hypothesis credible.\n\nNow the soft spots, in proportion. The load-bearing archaeological claim—that imprints carry label-relevant information—is not secured against covariate leakage. The dataset spans different clay textures, lighting, and camera settings, and the models are trained on full images, not the segmented imprint ROI. A class-correlated acquisition covariate survives random splits. The authors admit in Sec. 4.3 that the results do not exclude dataset-specific cues. That caveat is honest, but it leaves the archaeological conclusion unsupported as stated. The fix is standard: release the dataset or per-image metadata, train on the segmented ROI, and report cross-session or cross-clay-batch generalization. The usability evidence is also from the co-author expert users, which is weak; an independent user study would help but is minor for a platform paper.\n\nThe citation pattern is fine. Related work covers Roboflow, Geti, Ultralytics, MLOps tools, and CH applications; self-citation to [3] is appropriate and disclosed. No red flags there.\n\nWho is this for? Anyone building ML tooling for cultural heritage or other domain-expert communities will find the architecture and the honest workflow discussion useful. The archaeological hypothesis test is not yet a settled result, but the paper is candid about that.\n\nMy recommendation: this deserves a serious referee, with the expectation of revision. The platform is publishable on engineering merit, and the case study becomes defensible once leakage controls are reported or the claim is downgraded accordingly.","headline":"AmalthAI is a genuine open-source platform contribution; the archaeological hypothesis test is honestly framed but not yet leakage-controlled.","tokens_in":15903,"tokens_out":1739,"would_cite":true,"duration_ms":17737,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Textile imprints on clay preserve visual features that trained classifiers can read, and a no-code platform lets archaeologists do the reading.","keywords":["cultural heritage","computer vision","machine learning platform","MLOps","archaeology","textile imprints on clay","model interpretability","data sovereignty"],"falsifier":"Take the same experimental materials and images, then retrain and test under a protocol where clay preparation, lighting, and camera settings are systematically crossed with material and technique labels; if accuracy drops to near chance when the trained model is evaluated on imprint classes imaged under unseen clay and lighting conditions, or when those covariates are shuffled, the hypothesis that imprints preserve recoverable textile information would be refuted.","tokens_in":14891,"feed_emoji":"🏺","tokens_out":7581,"duration_ms":69053,"temperature":0.7,"pith_summary":"The paper tries to establish two linked claims: that textile imprints pressed into clay retain visual features distinguishing raw material (flax, nettle, lime bast, wool) and production technique (drilling, spinning, splicing), and that an archaeologist without machine-learning training can recover that information using a self-hosted, no-code platform. The evidence comes from controlled experimental imprints: a classifier reached 77.53% material accuracy and 85.57% technique accuracy, and segmentation isolated the imprint region at roughly 90% mean intersection-over-union. The authors present these numbers as workflow-validation evidence for the platform rather than as a benchmark claim. If the claims hold, cultural heritage researchers could test archaeological hypotheses about degraded or indirect textile evidence on their own infrastructure, without sending state-owned or restricted data to external cloud services.","feed_headline":"Clay imprints let AI identify textile material and technique","feed_subtitle":"On experimental imprints, the best model hit 77.5% material and 85.6% technique accuracy via a no-code platform.","key_machinery":"The central mechanism is AmalthAI's closed human-in-the-loop workflow. A web interface hides dataset management, training, and inference behind guided forms; an orchestration layer runs parallel hyperparameter-search trials and keeps the best checkpoint; inference runs in disposable containers built from the same training environment; and Grad-CAM (a heatmap technique showing which image regions drove the prediction) is fed, together with the original image and prediction metadata, to a vision-language model that writes a plain-language account of where the model looked and why it may have erred. The same storage layer can run fully on-site or synchronize with an external data service, which is what makes restricted artifact data usable in the first place.","core_discovery":"On the paper's own terms, the discovery is that experimentally produced clay imprints carry recoverable label-relevant information about the original textile: misclassifications concentrate in classes that archaeologists themselves find visually hard to separate, which the authors read as evidence that the models are exploiting textile-related properties. The supporting design discovery is that a single graphical platform can carry a non-expert from raw images to trained, reusable classifiers and segmenters; repeated train/test splits yielded similar scores, and the best-performing classification model produced the headline accuracies. The stated scope is deliberately modest: the results are consistent with the hypothesis that imprints preserve such information, not proof that every imprint does, and not a replacement for expert analysis.","pith_inferences":["Beyond the paper: the strongest test of the archaeological claim would be cross-condition generalization — training on imprints made with one clay preparation and lighting regime, then testing on imprints made with different clay and illumination; a collapse to chance would show the classifiers are reading dataset-specific cues rather than textile properties.","Beyond the paper: Grad-CAM plus a vision-language description could be used as a post-hoc shortcut detector for non-experts, but the paper does not measure whether this loop changes expert trust or improves final conclusions.","Beyond the paper: the same no-code workflow could be applied to other indirect archaeological evidence, such as tool marks or residues, whenever experts can define classes on archaeological grounds.","Beyond the paper: the platform's per-user data isolation and content-addressed storage design point toward a general pattern for privacy-preserving ML in heritage settings, but the paper does not evaluate multi-institution collaborative workflows."],"forward_implications":["Archaeologists without programming skills can train, compare, and reuse models for multiple label schemas from a single interface, so the same platform can test several hypotheses on one corpus.","Because all assets can stay on institutional hardware, the workflow is compatible with state-owned or rights-encumbered artifact collections that cannot be uploaded to vendor clouds.","The segmentation layer lets subsequent analysis concentrate on the imprint alone, screening out clay texture, lighting, and background irregularities.","Misclassification patterns across classes give a first-pass signal of which textile distinctions are visually confusable, and can direct experts to inspect those cases with the explanation loop.","Repeated splits with similar scores suggest the results are not an artifact of one favorable data partition, so the platform's default training setup can serve as a repeatable protocol."],"supporting_citations":[{"why":"Supplies one of the classification architectures benchmarked for material and technique accuracy.","marker":"[16]"},{"why":"Supplies the best-performing classification model whose inference results carry the headline 77.53% and 85.57% accuracies.","marker":"[46]"},{"why":"Supplies a segmentation architecture that reached the top mIoU in the imprint-localization experiments.","marker":"[4]"},{"why":"Supplies a second segmentation architecture used for imprint-region localization.","marker":"[39]"},{"why":"Defines Grad-CAM, the heatmap method used to localize each classification prediction before VLM explanation.","marker":"[43]"},{"why":"Provides the vision-language model that converts raw images, heatmaps, and prediction metadata into plain-language explanations.","marker":"[49]"},{"why":"Provides the interactive segmentation model used with the annotation tool to produce ground-truth masks.","marker":"[21]"},{"why":"Provides the annotation tool in which cultural heritage experts created the ground-truth segmentation masks.","marker":"[7]"},{"why":"Documents that similar fiber processing can make textile classes hard to distinguish visually, supporting the observed misclassification pattern.","marker":"[47]"}],"fun_headline_variants":["Open-source AI decodes textile clues from clay imprints","AmalthAI: no-code AI for cultural heritage analysis","Self-hosted AI platform aids non-expert artifact studies","Clay imprints + open-source AI: textile material ID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands or falls on whether the models' accuracy reflects the textile attributes rather than incidental visual cues — clay texture, lighting, and camera settings — that co-vary with the classes in the experimental images.","fun_headline_variants_meta":{"raw":{"variants":["Open-source AI decodes textile clues from clay imprints","AmalthAI: no-code AI for cultural heritage analysis","Self-hosted AI platform aids non-expert artifact studies","Clay imprints + open-source AI: textile material ID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1666,"prompt_tokens":895,"completion_tokens":771,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":704}},"tokens_in":511,"tokens_out":771,"duration_ms":7956,"temperature":1.0,"reasoning_tokens":704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:15:56.164974+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same experimental materials and images, then retrain and test under a protocol where clay preparation, lighting, and camera settings are systematically crossed with material and technique labels; if accuracy drops to near chance when the trained model is evaluated on imprint classes imaged under unseen clay and lighting conditions, or when those covariates are shuffled, the hypothesis that imprints preserve recoverable textile information would be refuted.","supporting_citations":[{"cited_title":"28, 2026","cited_arxiv_id":null,"evidence_quote":"Provides the annotation tool in which cultural heritage experts created the ground-truth segmentation masks."},{"cited_title":"In: Banck-Burgess, J., Marinova, E., Mischka, D","cited_arxiv_id":null,"evidence_quote":"Documents that similar fiber processing can make textile classes hard to distinguish visually, supporting the observed misclassification pattern."}],"review_version":1}