{"id":"227f5ace-0cec-4b9e-9154-97e60014e306","arxiv_id":"1909.02548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A deep network predicts 15 expert handwriting features from \"AND\" images, and two inference modules convert them into explainable same-writer/different-writer confidence scores, backed by a new public dataset.","lead":"This paper introduces XAI-AND, a public dataset of handwritten \"AND\" samples annotated with 15 expert-defined features, and a two-stage system that predicts those features with a deep network and then explains writer-verification decisions using either cosine similarity or a Bayesian likelihood ratio. The goal is to give forensic document examiners a transparent, per-feature report of why two handwriting samples are judged to be from the same writer or from different writers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The explanation is built from FLN feature predictions whose per-feature validation accuracy reaches as low as 65.63% (exit_stroke_d), but the paper never checks whether the resulting feature-similarity report corresponds to the true expert features; the central explanatory claim is therefore…","rationale":"The reader identifies the validity and reliability of the 15 expert-defined features as the weakest assumption. My concern is adjacent but more specific: even if the expert labels are valid, the explanation uses the FLN's predicted features, and several features have substantial prediction error. The paper never measures explanation fidelity, so the central claim that the system provides an interpretable decision report for an FDE is not established. I do not think this forces a rejection: the architecture is coherent, the dataset and accuracy tables are useful, and the gap is addressable with a ground-truth comparison experiment. I therefore keep the reader's CONDITIONAL verdict unchanged rather than escalating, while strengthening the technical reason for the condition. The concrete test proposed would directly measure how much of the explanation is driven by classifier error, and would also provide a way to calibrate confidence in the explanation.","tokens_in":7949,"tokens_out":3129,"duration_ms":36981,"concrete_test":"On Dtest, generate per-feature similarity reports and final COCS/LLR scores two ways: once using the FLN's predicted feature distributions/argmax labels, and once using the ground-truth expert feature labels for the same image pairs. Then compare (a) the per-feature similarity values and their ranks, and (b) the resulting overall verification decision. If the FLN-based and label-based reports disagree on which features drive the score for a nontrivial fraction of pairs (e.g., more than 10%), or if the verification accuracy computed with ground-truth labels differs materially from Table 6, the reported explanations cannot be taken as reflecting the actual expert-defined features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The system's stated goal is to provide a decision report for an FDE to interpret. Both inference modules are explicitly functions of the FLN's predicted features: DAAM computes cosine similarity between softmax probability vectors from the FLN (Eq. 7), and LAAM feeds the argmax of the softmax layer into a Bayesian network over categorical distances (Section 3.2). Consequently, the per-feature explanation shown in Figure 4 inherits every error of the feature learner. Table 5 reports validation accuracies for individual features that are far from reliable: exit_stroke_d is only 65.63%, slantness 71.88%, staff_of_d 85.94%, and staff_of_a 84.38%. If the predicted feature is wrong, the explanation may highlight a 'low similarity' on a feature that is actually similar, or vice versa. The paper provides no experiment relating the FLN-generated explanation to ground-truth expert labels, no analysis of the impact of per-feature classifier error on COCS or LR scores, and no inter-rater reliability measure for the 15 features that would establish that the labels themselves are a stable target. The verification accuracy numbers in Table 6 show the pipeline runs, but they do not validate the explanation's content. For the central claim to hold, the feature similarity report must be a trustworthy decomposition of the verification decision, and that requirement is exactly the unexamined link in this work.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an explainable handwriting verification system composed of a feature learning network (FLN) that maps handwritten 'AND' images to 15 expert-defined discrete features, followed by one of two inference modules: DAAM, which computes per-feature cosine similarity between softmax outputs and a thresholded overall similarity score, and LAAM, which builds a Bayesian network over categorical feature distances and outputs a likelihood ratio. The system outputs a per-feature similarity report intended for forensic document examiners. The authors introduce the XAI-AND dataset of 15,518 labeled image fragments and report verification accuracies up to 95.78% (DAAM_SAE) on seen writers and 94.27% on unseen writers, alongside per-feature validation accuracies for the FLN. The central claim is that the produced decision reports provide useful explanations for handwriting verification decisions.","tokens_in":8209,"tokens_out":3032,"duration_ms":30522,"significance":"If the explanation reports are accurate and trustworthy, the work addresses a real need in forensic handwriting verification, where opaque deep-learning outputs are unacceptable. The public release of a dataset with 15 expert-defined feature labels per sample is a useful resource for the community. The paper also demonstrates a plausible two-stage architecture (feature learning plus a probabilistic or geometric inference module) that could generalize to other verification tasks. However, the significance is conditional: the verification numbers are suspect due to a metric definition error, and the explanatory content is not validated against ground-truth feature comparisons or by any human-subject study. The paper's main contribution is therefore an unverified prototype rather than an established method.","major_comments":[{"comment":"The definitions of the evaluation metrics are incorrect as written. The text states 'Inter Writer Accuracy (Type 2) = FP/Se' and 'Overall Accuracy = (TP+TN)/Se' with Se being the total number of samples. FP/Se is a false-acceptance rate, not an accuracy, and the overall accuracy formula mixes sample counts with pair-based counts. Because every reported verification accuracy in Table 6 depends on these definitions, the quantitative results need to be recomputed with correct pair-level true-positive and true-negative rates, and the text must be corrected.","section":"Section 4, Table 6"},{"comment":"The central explainability claim is not validated. The paper shows qualitative examples of similarity reports but gives no experiment that compares the FLN-based per-feature similarity scores with ground-truth feature comparisons (e.g., whether a 'low similarity' report corresponds to actually different feature values under the expert labels). Given Table 5 reports per-feature validation accuracies as low as 65.63% (exit_stroke_d), the explanation report can highlight the wrong features for a substantial fraction of samples. Without a quantitative or human-subject evaluation, the claim that the system provides a trustworthy decision report for FDEs is unsupported.","section":"Section 4, Figure 4"},{"comment":"The explanation is by construction a deterministic function of the same features that produce the confidence score: COCS is the mean of the per-feature cosine similarities (Eq. (8)), and the LAAM likelihood ratio is computed from the same FLN argmax predictions via Eq. (10). The reports therefore decompose the score rather than provide an independent justification of the decision. This is a limitation that should be acknowledged explicitly; the paper currently presents the decomposition as an explanation without testing whether it aids interpretation or corresponds to forensic reasoning.","section":"Section 3.2, Eq. (7)-(10)"},{"comment":"There is a factual inconsistency in dataset size: the abstract states the XAI-AND dataset contains 13,700 handwritten samples, while Section 2 reports 15,518 'AND' image fragments extracted from the CEDAR Letter dataset. The source of this discrepancy must be resolved. Additionally, the paper reports that labels were collected from 89 external examiners and verified by 2 QD examiners, but provides no inter-rater reliability (e.g., Cohen's kappa) or per-feature label agreement. Since the entire feature-learning and explanation pipeline depends on the reliability of these 15 labels, the absence of any label-quality statistics is a load-bearing gap.","section":"Section 2 and Abstract"}],"minor_comments":[{"comment":"Equation (8) is inconsistent with the text: the text says COCS is the mean of the per-feature similarities, but the equation is a plain sum without division by 15.","section":"Section 3.2, Eq. (8)"},{"comment":"The loss notation y'_i and y_i in Eq. (4) is ambiguous; for the multi-class softmax outputs, the categorical cross-entropy should be written with a single class index rather than a binary per-class formulation.","section":"Section 3.1, Eq. (4)"},{"comment":"The LAAM_SAE row contains the typo 'Intra Inter Writer Accuracy (Type 1)' instead of 'Intra Writer Accuracy (Type 1)'.","section":"Table 6"},{"comment":"The text says 'This is shown in Figure 1 (a)' when describing the skip auto-encoder architecture, but the referenced figure is Figure 2.","section":"Section 3.1, Figure 2"},{"comment":"Several references are incomplete or inconsistent (e.g., [5] lacks year and venue details, [13] gives only authors and title), which will hinder readers in locating the cited works.","section":"References"},{"comment":"The abstract claims the methods extend to face verification and bio-medical comparison, but no experiments or analysis support this generalization; the claim should be softened to a suggestion for future work.","section":"Abstract / Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a workshop-style contribution, and the dataset release is a positive element, but the metric definitions and the lack of explanation validation are serious. The metric error in particular affects all headline numbers and must be corrected before any archival acceptance. I would not reject outright because the architecture and dataset are potentially salvageable, but the authors need to re-run the evaluation and provide a concrete validation of the explanation content. I also recommend asking for a reconciliation of the dataset size and a label-reliability analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: the paper's real contribution is the XAI-AND dataset, 15,518 \"AND\" fragments with 15 expert-annotated features, plus a two-module pipeline that produces a per-feature similarity report. That report is a faithful decomposition of the verification score by construction (COCS is the mean of the 15 cosine similarities, and the LLR is built from the same predicted features), so the \"explanation\" is not post-hoc hand-waving. What is missing is any evidence the explanation helps a forensic document examiner or is even correct at the feature level.\n\nWhat's new and good: the dataset is genuinely new and fills a gap. The authors recruited 89 external examiners and had two QD examiners verify labels. The FLN with a skip auto-encoder to denoise and a multi-task head for 15 features is a reasonable, reproducible architecture. The two inference modules are clean: cosine similarity over soft probabilities and a Bayesian-network likelihood ratio over categorical distance values. The paper is honest that some per-feature accuracies are poor (exit_stroke_d at 65.63%) and comparing DAAM and LAAM on seen/unseen/shuffled splits is standard.\n\nSoft spots, in order: (1) No inter-rater reliability on the 15 labels, so we don't know if the labeling protocol is stable. This is basic for a dataset that is supposed to be a benchmark. (2) The explanation's feature-level similarity scores come from the same predicted features that have low per-class accuracy, and there is no experiment relating the report to ground-truth expert feature comparisons. A low \"exit_stroke_d\" similarity in Figure 4 might reflect a misclassification, not a real difference. (3) No user study or task-based evaluation showing FDEs make better or faster decisions with the report. (4) Section 4's Type 2 accuracy definition is written as FP/Se, which must be a typo, and the abstract says 13,700 samples while Section 2 says 15,518. (5) Code and dataset are claimed to be public but no URLs appear in the manuscript.\n\nNone of these are fatal. The verification accuracies are believable and the pipeline demonstrably runs. The central claim is more modest than the title: it provides a decision report, not a validated explanation. The gaps are fixable.\n\nBottom line: worth a serious referee. The dataset alone justifies it. If the authors add IRR, correct the metric definition, release the data/code, and run a small FDE-based evaluation, this could be a solid benchmark paper.","headline":"A genuinely useful expert-labeled handwriting dataset and a clean pipeline that explains verification scores by construction, but the explainability claim itself is only demonstrated by examples and a user study is missing.","tokens_in":8760,"tokens_out":2362,"would_cite":false,"duration_ms":22568,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mapping handwriting to 15 expert traits explains every verdict","keywords":["handwriting verification","explainable AI","feature learning network","forensic document examination","cosine similarity","Bayesian network likelihood ratio","XAI-AND dataset","deep learning"],"falsifier":"Measure inter-rater agreement between the two questioned-document examiners on a sample of images; if agreement is low, or if a feature's labels do not predict whether two samples share a writer, the explanation loses its ground. Alternatively, compare verification accuracy when the 15 features are replaced with random labels; if accuracy does not drop, the features are not carrying the explanation.","tokens_in":7730,"feed_emoji":"✍️","tokens_out":4243,"duration_ms":37688,"temperature":0.7,"pith_summary":"The paper proposes a handwriting-verification system whose output is a per-feature explanation rather than an opaque confidence score. The system maps each handwriting image to 15 discrete features that forensic document examiners use, then compares two images feature by feature using either cosine similarity or a Bayesian likelihood-ratio model. If the claim holds, a forensic examiner receives a decision report that shows which traits matched and which did not, alongside an overall same-writer/different-writer verdict. On the XAI-AND dataset the best configuration reaches 95.78% overall accuracy on seen writers and 94.27% on unseen writers.","feed_headline":"Mapping handwriting to 15 expert traits explains every verdict","feed_subtitle":"A two-module system scores same- or different-writer pairs feature by feature, reaching 95.8% accuracy on seen writers.","key_machinery":"The central machinery is the Feature Learning Network (FLN): a multi-task network that fans out a shared encoding into 15 branches, each with a hidden layer and softmax, so each branch predicts one expert-defined handwriting feature. A skip auto-encoder (SAE) first reconstructs a denoised, translation-invariant image and provides the encoding, and the FLN's predictions feed two inference modules: cosine similarity (DAAM) and a Bayesian-network likelihood ratio (LAAM). The Bayesian network encodes dependencies among the 15 feature distances and supplies the likelihood-ratio explanation.","core_discovery":"The paper's claim is that explainable handwriting verification can be achieved by a two-module system: a Feature Learning Network (FLN) that maps a 64x64 handwriting image to predicted values of 15 expert-defined features, and an inference module that turns those predictions into a per-feature similarity report. The first inference method (DAAM) computes cosine similarity between the soft class probabilities of corresponding features; the second (LAAM) computes a likelihood ratio from a Bayesian network over categorical distances between features. The authors state that their goal is to provide a decision report for a forensic document examiner to interpret, and they report that the best configuration (DAAM with a skip auto-encoder) achieves 95.78% overall accuracy with seen writers and 94.27% with unseen writers.","pith_inferences":["If the 15 features prove reliable across examiners, the recipe of a multi-task feature learner plus similarity inference could transfer to fingerprint, signature, or other checklist-driven forensic comparisons.","Weighting features by their discriminative power might improve accuracy beyond the reported mean cosine similarity.","The explanation quality could be probed by ablating features and checking whether the likelihood ratio still tracks writer identity."],"forward_implications":["Each verification decision is accompanied by a per-feature similarity report, so an examiner can see which traits matched and which did not.","The same two-module architecture can be transferred to other verification tasks, including face verification and bio-medical comparison, as the paper states.","The XAI-AND dataset, with 15,518 'AND' images and expert feature labels, is released as a public benchmark for explainable handwriting verification.","The system can be extended from a two-pass pipeline to end-to-end training, and to visual explanations via gradient-based localization."],"supporting_citations":[{"why":"Supplies the handwriting-individuality hypothesis and the CEDAR Letter dataset from which XAI-AND images are drawn.","marker":"[12]"},{"why":"Provides the prior hybrid feature learning work whose writer partitioning and baselines the experiments build on.","marker":"[11]"},{"why":"Supplies the likelihood-ratio estimation via similarity and rarity that the LAAM inference module adapts.","marker":"[13]"},{"why":"Provides the symmetric skip-connection auto-encoder architecture used in the best-performing feature learner.","marker":"[8]"},{"why":"Defines the seven fundamental elements of handwriting that structure the 15 expert features.","marker":"[4]"},{"why":"The vantage-writer explanation approach that the proposed per-feature explanation contrasts with.","marker":"[2]"}],"fun_headline_variants":["Explainable handwriting verification using 15 expert features","AI handwriting verification now explains its verdicts via 15 traits","Forensic handwriting checks get explainable AI with 15 feature scores","Two-module net explains handwriting matches with 15 expert traits","95.8% accuracy: explainable handwriting verification via 15 traits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's explanation is only as valid as its assumption that the 15 expert-defined features reliably capture writer individuality and are consistently labeled.","fun_headline_variants_meta":{"raw":{"variants":["Explainable handwriting verification using 15 expert features","AI handwriting verification now explains its verdicts via 15 traits","Forensic handwriting checks get explainable AI with 15 feature scores","Two-module net explains handwriting matches with 15 expert traits","95.8% accuracy: explainable handwriting verification via 15 traits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3554,"prompt_tokens":942,"completion_tokens":2612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2526}},"tokens_in":558,"tokens_out":2612,"duration_ms":17417,"temperature":1.0,"reasoning_tokens":2526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:16:50.638995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure inter-rater agreement between the two questioned-document examiners on a sample of images; if agreement is low, or if a feature's labels do not predict whether two samples share a writer, the explanation loses its ground. Alternatively, compare verification accuracy when the 15 features are replaced with random labels; if accuracy does not drop, the features are not carrying the explanation.","supporting_citations":[{"cited_title":"Individuality of hand- writing: a validation study","cited_arxiv_id":null,"evidence_quote":"Supplies the handwriting-individuality hypothesis and the CEDAR Letter dataset from which XAI-AND images are drawn."},{"cited_title":"Likelihood ratio estimation in forensic identification using similarity and rarity, 2013","cited_arxiv_id":null,"evidence_quote":"Supplies the likelihood-ratio estimation via similarity and rarity that the LAAM inference module adapts."},{"cited_title":"Crepieux-Jamin","cited_arxiv_id":null,"evidence_quote":"Defines the seven fundamental elements of handwriting that structure the 15 expert features."},{"cited_title":"Towards explainable writer veri- fication and identification using vantage writers","cited_arxiv_id":null,"evidence_quote":"The vantage-writer explanation approach that the proposed per-feature explanation contrasts with."}],"review_version":1}