Pith. sign in

REVIEW 4 major objections 6 minor 13 references

Explanation based Handwriting Verification

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mapping handwriting to 15 expert traits explains every verdict

desk verdict A genuinely useful expert-labeled handwriting dataset and a clean pipeline that explains verification scores by construction, but the explainability claim itself is only demonstrated by examples and a user study is missing. read the letter →

arxiv 1909.02548 v1 pith:VTCMJLTX submitted 2019-08-14 cs.CV cs.LG

Mihir Chauhan , Mohammad Abuzar Shaikh , Sargur N. Srihari This is my paper · ORCID
classification cs.CVcs.LG
keywords handwritingverificationexplainableAIfeaturelearningnetworkforensicdocumentexaminationcosinesimilarityBayesianlikelihoodratioXAI-ANDdatasetdeep
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a handwriting-verification system whose output is a per-feature explanation rather than an opaque confidence score. The system maps each handwriting image to 15 discrete features that forensic document examiners use, then compares two images feature by feature using either cosine similarity or a Bayesian likelihood-ratio model. If the claim holds, a forensic examiner receives a decision report that shows which traits matched and which did not, alongside an overall same-writer/different-writer verdict. On the XAI-AND dataset the best configuration reaches 95.78% overall accuracy on seen writers and 94.27% on unseen writers.

What carries the argument

The central machinery is the Feature Learning Network (FLN): a multi-task network that fans out a shared encoding into 15 branches, each with a hidden layer and softmax, so each branch predicts one expert-defined handwriting feature. A skip auto-encoder (SAE) first reconstructs a denoised, translation-invariant image and provides the encoding, and the FLN's predictions feed two inference modules: cosine similarity (DAAM) and a Bayesian-network likelihood ratio (LAAM). The Bayesian network encodes dependencies among the 15 feature distances and supplies the likelihood-ratio explanation.

What would settle it

Measure inter-rater agreement between the two questioned-document examiners on a sample of images; if agreement is low, or if a feature's labels do not predict whether two samples share a writer, the explanation loses its ground. Alternatively, compare verification accuracy when the 15 features are replaced with random labels; if accuracy does not drop, the features are not carrying the explanation.

Watch

Extended reading notes

Core claim

The paper's claim is that explainable handwriting verification can be achieved by a two-module system: a Feature Learning Network (FLN) that maps a 64x64 handwriting image to predicted values of 15 expert-defined features, and an inference module that turns those predictions into a per-feature similarity report. The first inference method (DAAM) computes cosine similarity between the soft class probabilities of corresponding features; the second (LAAM) computes a likelihood ratio from a Bayesian network over categorical distances between features. The authors state that their goal is to provide a decision report for a forensic document examiner to interpret, and they report that the best configuration (DAAM with a skip auto-encoder) achieves 95.78% overall accuracy with seen writers and 94.27% with unseen writers.

Load-bearing premise

The system's explanation is only as valid as its assumption that the 15 expert-defined features reliably capture writer individuality and are consistently labeled.

Editorial extensions

If this is right

  • Each verification decision is accompanied by a per-feature similarity report, so an examiner can see which traits matched and which did not.
  • The same two-module architecture can be transferred to other verification tasks, including face verification and bio-medical comparison, as the paper states.
  • The XAI-AND dataset, with 15,518 'AND' images and expert feature labels, is released as a public benchmark for explainable handwriting verification.
  • The system can be extended from a two-pass pipeline to end-to-end training, and to visual explanations via gradient-based localization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 15 features prove reliable across examiners, the recipe of a multi-task feature learner plus similarity inference could transfer to fingerprint, signature, or other checklist-driven forensic comparisons.
  • Weighting features by their discriminative power might improve accuracy beyond the reported mean cosine similarity.
  • The explanation quality could be probed by ablating features and checking whether the likelihood ratio still tracks writer identity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an explainable handwriting verification system composed of a feature learning network (FLN) that maps handwritten 'AND' images to 15 expert-defined discrete features, followed by one of two inference modules: DAAM, which computes per-feature cosine similarity between softmax outputs and a thresholded overall similarity score, and LAAM, which builds a Bayesian network over categorical feature distances and outputs a likelihood ratio. The system outputs a per-feature similarity report intended for forensic document examiners. The authors introduce the XAI-AND dataset of 15,518 labeled image fragments and report verification accuracies up to 95.78% (DAAM_SAE) on seen writers and 94.27% on unseen writers, alongside per-feature validation accuracies for the FLN. The central claim is that the produced decision reports provide useful explanations for handwriting verification decisions.

Significance. If the explanation reports are accurate and trustworthy, the work addresses a real need in forensic handwriting verification, where opaque deep-learning outputs are unacceptable. The public release of a dataset with 15 expert-defined feature labels per sample is a useful resource for the community. The paper also demonstrates a plausible two-stage architecture (feature learning plus a probabilistic or geometric inference module) that could generalize to other verification tasks. However, the significance is conditional: the verification numbers are suspect due to a metric definition error, and the explanatory content is not validated against ground-truth feature comparisons or by any human-subject study. The paper's main contribution is therefore an unverified prototype rather than an established method.

major comments (4)
  1. [Section 4, Table 6] The definitions of the evaluation metrics are incorrect as written. The text states 'Inter Writer Accuracy (Type 2) = FP/Se' and 'Overall Accuracy = (TP+TN)/Se' with Se being the total number of samples. FP/Se is a false-acceptance rate, not an accuracy, and the overall accuracy formula mixes sample counts with pair-based counts. Because every reported verification accuracy in Table 6 depends on these definitions, the quantitative results need to be recomputed with correct pair-level true-positive and true-negative rates, and the text must be corrected.
  2. [Section 4, Figure 4] The central explainability claim is not validated. The paper shows qualitative examples of similarity reports but gives no experiment that compares the FLN-based per-feature similarity scores with ground-truth feature comparisons (e.g., whether a 'low similarity' report corresponds to actually different feature values under the expert labels). Given Table 5 reports per-feature validation accuracies as low as 65.63% (exit_stroke_d), the explanation report can highlight the wrong features for a substantial fraction of samples. Without a quantitative or human-subject evaluation, the claim that the system provides a trustworthy decision report for FDEs is unsupported.
  3. [Section 3.2, Eq. (7)-(10)] The explanation is by construction a deterministic function of the same features that produce the confidence score: COCS is the mean of the per-feature cosine similarities (Eq. (8)), and the LAAM likelihood ratio is computed from the same FLN argmax predictions via Eq. (10). The reports therefore decompose the score rather than provide an independent justification of the decision. This is a limitation that should be acknowledged explicitly; the paper currently presents the decomposition as an explanation without testing whether it aids interpretation or corresponds to forensic reasoning.
  4. [Section 2 and Abstract] There is a factual inconsistency in dataset size: the abstract states the XAI-AND dataset contains 13,700 handwritten samples, while Section 2 reports 15,518 'AND' image fragments extracted from the CEDAR Letter dataset. The source of this discrepancy must be resolved. Additionally, the paper reports that labels were collected from 89 external examiners and verified by 2 QD examiners, but provides no inter-rater reliability (e.g., Cohen's kappa) or per-feature label agreement. Since the entire feature-learning and explanation pipeline depends on the reliability of these 15 labels, the absence of any label-quality statistics is a load-bearing gap.
minor comments (6)
  1. [Section 3.2, Eq. (8)] Equation (8) is inconsistent with the text: the text says COCS is the mean of the per-feature similarities, but the equation is a plain sum without division by 15.
  2. [Section 3.1, Eq. (4)] The loss notation y'_i and y_i in Eq. (4) is ambiguous; for the multi-class softmax outputs, the categorical cross-entropy should be written with a single class index rather than a binary per-class formulation.
  3. [Table 6] The LAAM_SAE row contains the typo 'Intra Inter Writer Accuracy (Type 1)' instead of 'Intra Writer Accuracy (Type 1)'.
  4. [Section 3.1, Figure 2] The text says 'This is shown in Figure 1 (a)' when describing the skip auto-encoder architecture, but the referenced figure is Figure 2.
  5. [References] Several references are incomplete or inconsistent (e.g., [5] lacks year and venue details, [13] gives only authors and title), which will hinder readers in locating the cited works.
  6. [Abstract / Section 5] The abstract claims the methods extend to face verification and bio-medical comparison, but no experiments or analysis support this generalization; the claim should be softened to a suggestion for future work.

Circularity Check

1 steps flagged · score 3.0 of 10

Per-feature explanation is an arithmetic decomposition of the defined COCS score; verification results remain independently grounded.

  1. self definitional [Section 3.2, "Distance as a measure (DAAM)", Eq. (8); Section 4, discussion of Figure 4]
    "We measure the similarity of the categorical probabilities learned by the FLN soft-max layer. ... We also compute the Overall Cosine Similarity COCS by taking the mean of the sum of Csim across all f. COCS(q_f,k_f)=Σ j=1..15 Csim(q_fj,k_fj) (8) ... The overall similarity score is low (0.3784) because the similarity betweenis_lowercase, sta f f_o f_d and exit_stroke_d is low."

    Eq. (8) defines the verification confidence score COCS as the aggregate of the 15 per-feature cosine similarities Csim(q_fj,k_fj), which are computed from the FLN softmax probabilities. The 'explanation' in Figure 4 says the overall score is low because is_lowercase, staff_of_d, and exit_stroke_d have low similarity. Those are exactly the additive components of COCS, so the explanation is true by construction and cannot serve as an independent test of why the network made the decision. The feature-level report is a restatement of the score's definition, not a separate explanation derived from the network's latent reasoning.

full rationale

Most of the paper's derivation chain is self-contained: the FLN is trained on 15 expert feature labels, the inference threshold T is tuned on Dval and evaluated on Dtest, and verification accuracy is measured against external writer identities from the CEDAR Letter dataset. The self-citations ([11], [13]) supply a data-partitioning convention and a standard likelihood-ratio/Bayesian-network formulation; neither is a load-bearing uniqueness claim that forces the paper's choice. The one qualified circularity is in the explainability claim: DAAM's confidence score is defined as the sum of the per-feature cosine similarities, so the per-feature report in Figure 4 is an accounting identity rather than an independent causal explanation. This is a mild definitional self-consistency issue, not a fabricated prediction, and it does not undermine the dataset or the reported verification accuracies; hence the low score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of 15 expert features, the faithfulness of the learned feature predictors, and modeling choices for the likelihood-ratio network. The only explicitly tuned numeric parameter is the DAAM threshold; the Bayesian network structure is a hand or scoring choice. No new physical or conceptual entities are postulated.

free parameters (2)
  • Decision threshold T (DAAM) = not reported; chosen on Dval where precision is close to recall
    The threshold converts cosine similarity into a same/different-writer decision. It is tuned on validation data, which is standard, but the reported verification accuracy is threshold-dependent and not parameter-free.
  • Bayesian network structure for BN1 and BN2 = not specified; selected using correlation, k2, BDeu, BIC, and domain knowledge
    The LLR explanation depends on the graph structure, but the paper does not report the final structure or the exact selection rule, making the likelihood ratio partly a modeling choice.
assumptions (4)
  • domain assumption Every individual has a unique way of writing (individuality hypothesis).
    Invoked in Section 1 via [12]; this is the premise of forensic writer verification and of the dataset's design, assumed rather than established here.
  • domain assumption The 15 expert-defined features are a sufficient and reliable representation of writer-discriminating characteristics.
    The system's confidence and explanations use only these features. No evidence is provided that they capture all relevant writer information or that labels are consistent across examiners (Section 2).
  • domain assumption Predicted feature probabilities from FLN are faithful estimates of the expert features.
    Explanations are computed from the softmax and argmax outputs of FLN. Table 5 shows some features are predicted at only 65.63% accuracy (exit_stroke_d), so prediction errors propagate into the explanations.
  • domain assumption The Bayesian network structure encodes the true dependencies between feature differences.
    Equation 10 factorizes the joint probability using the graph in Figure 3, but the structure is chosen by scoring and domain knowledge, not derived or validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explanation based Handwriting Verification." pith.science (2026). https://pith.science/paper/VTCMJLTX

@misc{pith2026190902548,
  author       = {Pith},
  title        = {Pith review of: Explanation based Handwriting Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VTCMJLTX}},
  note         = {Machine review of arXiv:1909.02548}
}
read the original abstract

Deep learning system have drawback that their output is not accompanied with ex-planation. In a domain such as forensic handwriting verification it is essential to provideexplanation to jurors. The goal of handwriting verification is to find a measure of confi-dence whether the given handwritten samples are written by the same or different writer.We propose a method to generate explanations for the confidence provided by convolu-tional neural network (CNN) which maps the input image to 15 annotations (features)provided by experts. Our system comprises of: (1) Feature learning network (FLN),a differentiable system, (2) Inference module for providing explanations. Furthermore,inference module provides two types of explanations: (a) Based on cosine similaritybetween categorical probabilities of each feature, (b) Based on Log-Likelihood Ratio(LLR) using directed probabilistic graphical model. We perform experiments using acombination of feature learning network (FLN) and each inference module. We evaluateour system using XAI-AND dataset, containing 13700 handwritten samples and 15 cor-responding expert examined features for each sample. The dataset is released for publicuse and the methods can be extended to provide explanations on other verification taskslike face verification and bio-medical comparison. This dataset can serve as the basis and benchmark for future research in explanation based handwriting verification. The code is available on github.

Figures

Figures reproduced from arXiv: 1909.02548 by the authors.

Figure 1
Figure 1. Explanation with evidence provided to the court of law by a QD examiner. Red [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) shows the Auto Encoder Architecture. (b) shows the expanded form of Features [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Bayesian network structure for difference distribution [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Graph of Similarity Score of Questioned (q) & Known (k) Image Features for (a) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 10 canonical work pages

  1. [1]

    Recognition of handwritten and machine-printed text for postal address interpretation

    Srihari SN. Recognition of handwritten and machine-printed text for postal address interpretation. Pattern Recognition Letters 1993;14: 291-303

    work page 1993

  2. [2]

    Towards explainable writer veri- fication and identification using vantage writers

    Axel Brink, Lambert Schomaker, and Marius Bulacu. Towards explainable writer veri- fication and identification using vantage writers. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, pages 824–828. IEEE, 2007

    work page 2007

  3. [3]

    Writer verification using cnn feature extraction

    Jun Chu, Mohammad Abuzar Shaikh, Mihir Chauhan, Lu Meng, and Sargur Srihari. Writer verification using cnn feature extraction. In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 181–186. IEEE, 2018

    work page 2018

  4. [4]

    Crepieux-Jamin

    J. Crepieux-Jamin. L’ecriture et le caractere. presses univ. de france, 14th edition, 1951

    work page 1951

  5. [5]

    Explainable artificial intelligence (xai)

    David Gunning. Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA), nd Web, 2017

    work page 2017

  6. [6]

    Huber and A.M

    Roy A. Huber and A.M. Headrick. Handwriting identification: facts and fundamentals. 1999

    work page 1999

  7. [7]

    Convolutional networks for images, speech, and time series

    Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995

    work page 1995

  8. [8]

    Image restoration using convolutional auto-encoders with symmetric skip connections

    Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv preprint arXiv:1606.08921, 2016

    arXiv 2016

Show all 13 references
  1. [9]

    Corpus-based and knowledge-based measures of text semantic similarity

    Rada Mihalcea, Courtney Corley, Carlo Strapparava, et al. Corpus-based and knowledge-based measures of text semantic similarity. In AAAI, volume 6, pages 775– 780, 2006

    2006

  2. [10]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618–626, 2017

    2017

  3. [11]

    M. A. Shaikh, M. Chauhan, J. Chu, and S. Srihari. Hybrid feature learning for handwrit- ing verification. In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 187–192, Aug 2018. doi: 10.1109/ICFHR-2018.2018. 00041

    2018

  4. [12]

    Individuality of hand- writing: a validation study

    Sargur N Srihari, Sung-Hyuk Cha, Hina Arora, and Sangjik Lee. Individuality of hand- writing: a validation study. In Proceedings of Sixth International Conference on Doc- ument Analysis and Recognition, pages 106–109. IEEE, 2001

    2001

  5. [13]

    Likelihood ratio estimation in forensic identification using similarity and rarity, 2013

    Sargur Srihari Yi Tang. Likelihood ratio estimation in forensic identification using similarity and rarity, 2013

    2013

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.