{"id":"d3b94d52-6762-4a39-9dc2-63e1448edde0","arxiv_id":"2506.16331","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Pixel-level saliency maps outperform point-specific maps for explaining writer identification and verification networks, though the forensic-support claim rests on a limited qualitative evaluation.","lead":"Two existing explainability methods were tested on neural networks that recognize handwriting authors, with pixel-level maps scoring better than point-specific maps. The work aims to provide forensic handwriting experts with visual support for authorship decisions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deletion/insertion in Sec 3.2 uses the same image as query and retrieval, so the point-specific maps are evaluated as self-similarity maps while pixel-level maps are evaluated against their own gradient objective; the reported superiority may be a protocol artifact.","rationale":"The paper's central quantitative claim is that pixel-level saliency maps outperform point-specific saliency maps, and that claim rests entirely on the deletion/insertion metric of Sec 3.2. The reader flagged the unspecified threshold and the same-image query/retrieval setup. I go further: the same-image setup is not just a missing detail; it changes what each map is being asked to do. Point-specific overall maps are built from self-correlations of the image's feature maps, so deleting their highlighted pixels removes repeated patterns but leaves the embedding relatively stable; low deletion scores are largely predetermined. Pixel-level maps are gradients of cosine distance to a blank white image, so their highlighted pixels are precisely those whose deletion moves the embedding toward the base and lowers self-cosine similarity; high deletion scores are aligned with the map's construction. Thus the comparison is biased before any threshold is set. This is the most load-bearing concern because if the metric is invalid, the headline 'outperforms' is unsupported, and the forensic-support conclusion rests mainly on two manually inspected pages. The proposed cross-image test would settle it. I also note that Table 2 contains exceptions where point-specific scores exceed pixel-level scores (e.g., Firemaker ResNet20 WI: 22.8 vs. 15.6), but the main issue is protocol validity rather than these individual reversals. The reader's conditional verdict remains appropriate; the paper needs the metric protocol corrected and the threshold reported, or the quantitative claim should be dropped.","tokens_in":7831,"tokens_out":8348,"duration_ms":94432,"concrete_test":"Compute deletion/insertion on genuine retrieval pairs: for each query q, use a same-writer retrieved image r+ and a different-writer r− from the held-out test set. For each saliency method, generate the map on (q, r), then iteratively set black pixels of r to white (deletion) or add them to a white base (insertion) in map order, and record cosine(q, altered_r). Average over at least 100 pairs per dataset, report the cosine threshold or sweep it, and include the random-deletion baseline. If pixel-level maps no longer consistently beat point-specific maps on this cross-image metric, the reported superiority is an artifact of the self-similarity protocol; if they still do, the quantitative claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec 3.2 defines deletion/insertion with 'the query and retrieved image are the same' and only black pixels altered. Under this protocol the point-specific overall saliency map is a self-correlation map (Eq. 2 with q=r): it highlights pixels in the image that are similar to other pixels in the same image. Deleting those pixels removes repeated patterns, but the global embedding changes little because other instances of the pattern remain, so low aucd scores are largely predetermined. The pixel-level map, by contrast, is the gradient of cosine distance to a blank white base image (Sec 2.1); its top pixels are exactly the pixels whose deletion moves the embedding toward the base and away from the original image, so high deletion scores are aligned with the map's construction. The comparison therefore stacks the deck before the threshold is applied. The unspecified cosine threshold in Sec 3.2 can further compress or shift the AUC values and makes reproduction impossible. Consequently the headline 'pixel-wise outperforms point-specific' is not established by the current metric; the qualitative two-page analysis does not fix this because it also compares highlight patterns without a defined scoring rule or expert ground truth.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies two transparency techniques to neural networks trained for writer identification and writer verification: pixel-level saliency maps based on gradients of the embedding cosine distance to a blank white image, and point-specific saliency maps based on feature-map inner products that highlight similar image regions. The methods are evaluated on the CVL, Firemaker, and ICDAR2013 datasets with deletion and insertion AUC scores adapted from Hu et al., and qualitatively by inspecting whether highlighted regions repeat across occurrences of the same character. The authors report that pixel-wise saliency maps achieve higher deletion scores than point-specific maps, and conclude that pixel-wise maps outperform point-specific maps and are suitable for supporting forensic handwriting analysis.","tokens_in":8055,"tokens_out":4129,"duration_ms":47707,"significance":"This is the first application of these two transparency techniques to writer identification and verification, and the topic is relevant for forensic document analysis. The quantitative comparison in Table 2 covers multiple architectures and datasets, and the qualitative analysis examines character-level patterns in a domain where such evidence could be useful. If the evaluation were sound, the finding that pixel-level gradients localize writer-specific features more reliably than point-specific self-similarity maps would be a useful contribution. However, the current evaluation protocol has structural asymmetries and missing parameters that undermine the headline claim, so the significance is conditional on a reworked evaluation.","major_comments":[{"comment":"The deletion/insertion protocol sets 'the query and retrieved image are the same.' This makes the point-specific overall saliency map a self-similarity map: with q=r, Eq. (2) highlights pixels that are similar to other pixels in the same image. Deleting such pixels removes repeated patterns, but the global embedding may change little because other instances of the pattern remain, so low deletion scores for the point-specific maps are partly predetermined. In contrast, the pixel-level map is the gradient of the cosine distance to a blank white base image (Sec. 2.1), so its top pixels are by construction the pixels whose deletion moves the embedding away from the original image. The comparison in Table 2 therefore does not establish that pixel-wise maps are intrinsically better; it may reflect a protocol artifact. I recommend re-running the evaluation with query and retrieved images being different test images, or otherwise grounding both methods in the same retrieval task.","section":"Sec. 3.2 and Table 2"},{"comment":"The text says 'The value is set to zero if it falls below this threshold,' but the threshold is never specified. Because the AUC values in Table 2 depend on this threshold, the quantitative results are not reproducible, and the ranking of methods could change with different thresholds. The authors should state the exact threshold used and report sensitivity of the deletion/insertion scores to this parameter.","section":"Sec. 3.2"},{"comment":"The insertion scores are near ceiling for both methods: for the WI networks, all values in Table 2 are above 89%, and for the WV networks all but two are above 90%. This makes insertion scores non-discriminating, so the claim that pixel-wise maps outperform point-specific maps rests entirely on the deletion scores. Given the protocol issue in Sec. 3.2, the deletion-score advantage is not sufficient evidence. A metric that is not saturated, or an explicit argument for why the saturated insertion scores do not matter, is needed.","section":"Sec. 3.4 and Table 2"},{"comment":"The qualitative evaluation is based on only two pages from the CVL dataset, one with successful retrieval and one with failed retrieval. The comparison to 'the areas forensic experts consider' is informal: no forensic expert annotations, no defined scoring rule, and no inter-rater protocol are provided. The claim that the pixel-wise maps are 'suitable for the support of forensic experts' is therefore not supported by the presented evidence. A systematic study with expert-provided ground truth or a predefined highlight-matching criterion would be required.","section":"Sec. 3.5"}],"minor_comments":[{"comment":"The number of random masked variants, n=4, is fixed without any sensitivity analysis or justification. Since SmoothGrad results can depend on n, the robustness of the saliency maps to this choice should be discussed.","section":"Sec. 2.1"},{"comment":"The choice of a white base image is motivated as representing an empty writing surface, but the networks were trained on snippets containing handwriting. The effect of using a blank image as the base for gradient computation is not analyzed; a short discussion or ablation would be helpful.","section":"Sec. 2.1"},{"comment":"The definition of the overall saliency map is ambiguous: after Eq. (2), the summation over (i,j) with a fixed (x,y) is not clearly distinguished from a summation over all spatial locations. Please define the variables and index ranges explicitly.","section":"Sec. 2.2"},{"comment":"The description of the deletion and insertion procedure says only black pixels are altered, but it is not clear how partial deletions or insertions are applied per iteration. A precise algorithmic description would improve reproducibility.","section":"Sec. 3.2"},{"comment":"The text states that for WI networks the insertion scores are high with 'all but one value being above 90%,' which is correct for Table 2, but the corresponding sentence for WV networks should be checked: the ResNet20 CVL insertion score is 84.3%, which is the second exception, not the only one.","section":"Sec. 3.4"}],"recommendation":"major_revision","confidential_remarks":"The evaluation protocol issue in Sec. 3.2 is load-bearing for the central claim. The authors compare a self-similarity map against a gradient-to-blank-base map without specifying the cosine threshold, so the Table 2 result could change substantially under a fair protocol. The qualitative claims also need stronger grounding. I would be willing to reconsider after the authors re-run the evaluation with distinct query/retrieved pairs and provide the threshold value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on the WI/WV transparency paper. The genuinely new piece is the domain transfer: nobody had applied Kobs-style pixel saliency or Zhu-style point-specific maps to writer identification and verification, and the adaptation to binarized handwriting (white base image, edits restricted to black pixels) is a sensible, small extension. The paper also reports results on three datasets, which is more than a toy demonstration.\n\nThe trouble is the quantitative comparison. The deletion/insertion protocol in Sec 3.2 sets query and retrieved image to be the same. For the point-specific overall saliency map, that makes the map a self-correlation map: it highlights pixels similar to other pixels in the same image. Deleting those pixels leaves the global embedding mostly intact because repeated patterns remain. For the pixel-level map, the saliency is the gradient of cosine distance to a blank white base image, so deleting the top pixels drives the embedding toward the base by construction. The deletion score therefore measures two different things, and the reported superiority of pixel-level maps is largely baked into the setup. The unspecified cosine threshold only adds to the problem—there is no way to reproduce the exact AUC values.\n\nOther soft spots: insertion scores are near-ceiling for both methods, so they don't discriminate. There are no error bars or statistical tests across the nine network/dataset combinations, and no code or trained models are released. The qualitative section is two pages from one dataset; the claim that pixel-wise maps are 'suitable for support of forensic experts' goes beyond what that evidence can carry.\n\nThat said, the paper is not sloppy about its stated scope. It explicitly points out that point-specific maps may improve with character-level input, and the qualitative observations about recurring highlights for same characters are plausible. But the central quantitative claim is not established by the current protocol. A careful revision could fix this: use different query and retrieval images, specify the threshold, add error bars, and perhaps validate against expert annotations.\n\nWho is this for? Researchers working on explainability for document analysis and forensic handwriting tools. It deserves a serious referee because the domain application is real and the flaws are fixable, but the paper needs major revision before its comparison can be trusted.","headline":"First application of two saliency methods to writer ID/verification, but the deletion-score comparison that drives the headline is compromised by a same-image protocol that favors the pixel-level maps.","tokens_in":8570,"tokens_out":2950,"would_cite":false,"duration_ms":32804,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pixel-wise saliency maps outperform point-specific saliency maps for explaining writer-identification and writer-verification networks, and can support forensic handwriting analysis.","keywords":["writer identification","writer verification","saliency maps","model transparency","forensic handwriting analysis","deletion and insertion metrics","deep metric learning","handwriting retrieval"],"falsifier":"Run the deletion and insertion protocol with several different cosine-similarity thresholds and with genuine retrieval pairs (two different pages, sometimes by the same writer, sometimes by different writers) instead of the same image; if the pixel-level map's deletion-score advantage over point-specific maps disappears or reverses under any of these settings, the paper's central quantitative claim does not survive. A second check: match saliency peaks against labeled character occurrences on a larger sample; if the repeated 'y' and 'f' patterns do not appear, the qualitative support claim weakens.","tokens_in":7584,"feed_emoji":"✍️","tokens_out":5250,"duration_ms":55435,"temperature":0.7,"pith_summary":"Neural networks identify and verify handwriting authors well, but what they look at inside a page of text stays opaque. This paper is the first to apply two transparency techniques to writer-identification and writer-verification networks: pixel-wise saliency maps and point-specific saliency maps. Using deletion and insertion scores on three handwriting datasets, it finds that pixel-wise maps outperform point-specific maps and highlight the same character forms that forensic experts consider during authorship analysis. The point-specific maps allocate highlights less consistently and are judged not suitable for supporting the analysis process.","feed_headline":"Pixel-level saliency maps outperform point maps for writer ID","feed_subtitle":"First transparency comparison for handwriting-ID networks: gradients localize the character features forensic experts use.","key_machinery":"The load-bearing objects are two gradient-based saliency constructions. The pixel-wise map is $s(I) = \\partial d(x_I, x_{\\text{base}})/\\partial I$, the gradient of the cosine-similarity distance between the embedding of the input image and a white blank-paper base image, averaged over four random white-pixel masks in a smooth-grad style. The point-specific map starts from rewriting cosine similarity between a query image $q$ and a retrieval image $r$ as a sum over all location pairs of dot products of last-layer feature maps, so each pair of points $(i,j)$ and $(x,y)$ receives saliency $\\sum_k A^q_{i,j,k} A^r_{x,y,k}$; an overall map sums these contributions. The evaluation machinery is the deletion/insertion protocol: iteratively remove or insert the map's most significant black pixels, track cosine similarity against a random baseline, and summarize with AUC scores.","core_discovery":"The paper claims that for embedding-based writer-identification and writer-verification networks, pixel-wise saliency maps derived from the gradient of cosine similarity between the input embedding and a white base image are more faithful than point-specific saliency maps derived from decomposing cosine similarity over feature-map locations. Faithfulness is measured by deletion and insertion AUC scores: deleting the most-highlighted pixels should lower similarity faster than random deletion, and inserting them should raise it faster than random insertion. Across ResNet18, ResNet20, and ResNet50 backbones and the CVL, Firemaker, and ICDAR2013 datasets, the pixel-level maps achieve substantially higher deletion scores than the point-specific maps, while insertion scores are high for both. Qualitatively, pixel-level maps show repeated highlighting patterns across occurrences of the same character, whereas point-specific maps often place peak highlights on the wrong character or spread highlights over many characters, making them less suitable for forensic use.","pith_inferences":["Editorial: Because the paper uses the same image as query and retrieved image, the deletion/insertion metric measures self-consistency rather than retrieval relevance; testing with genuine retrieval pairs would clarify whether pixel-level maps also explain real identification decisions.","Editorial: The cosine-similarity threshold used to zero scores is never stated, so a simple threshold sweep would show whether the reported pixel-level advantage is robust.","Editorial: The paper hints that single-character inputs could improve point-specific maps; a quantitative comparison of highlight placement on cropped characters would turn that suggestion into a testable claim.","Editorial: The point-to-image maps' tendency to highlight similarly shaped letter parts suggests the networks encode sub-character grapheme fragments, which could become a vocabulary for forensic comparison if confirmed on larger samples."],"forward_implications":["If pixel-wise maps are faithful, forensic experts can use them to locate writer-specific character forms without reading model internals.","Deletion and insertion scores can serve as a standard quantitative check for saliency methods in document-analysis networks.","Deletion scores depend strongly on network architecture, so transparency quality is a property of the network and loss as much as of the explanation method.","Repeated highlighting of the same characters suggests these networks learn allographic letter-shape features, which could inform better writer-identification training.","Point-specific maps may become interpretable only with character-level or tighter image crops, a direction the paper identifies for future work."],"supporting_citations":[{"why":"Provides the ResNet-based writer-identification training setup that the transparency methods are applied to.","marker":"[7]"},{"why":"Supplies the deletion and insertion score protocol that the paper adapts for binarized handwriting images.","marker":"[14]"},{"why":"Introduces the point-specific saliency maps for image pairs that are evaluated here.","marker":"[16]"},{"why":"One of the three handwriting datasets used for training and evaluation.","marker":"[17]"},{"why":"Second handwriting dataset used for training and evaluation of the transparency techniques.","marker":"[18]"},{"why":"Third handwriting dataset, with a smaller snippet size, used for training and evaluation.","marker":"[19]"},{"why":"Introduces the pixel-level saliency map gradient technique with smooth-grad averaging.","marker":"[20]"},{"why":"Provides the SigNet contrastive architecture used for the writer-verification networks.","marker":"[22]"}],"fun_headline_variants":["Pixel saliency maps beat point maps for writer ID","Gradients clarify handwriting: pixel maps top point maps","For handwriting AI, pixel-level heatmaps outshine point heatmaps","Pixel maps reveal handwriting features point maps miss","Transparency for writer ID: pixel saliency wins over point"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the deletion/insertion metric, which uses the same image as query and retrieved image and zeroes similarities below an unspecified threshold, correctly measures how well a saliency map explains the network's decisions.","fun_headline_variants_meta":{"raw":{"variants":["Pixel saliency maps beat point maps for writer ID","Gradients clarify handwriting: pixel maps top point maps","For handwriting AI, pixel-level heatmaps outshine point heatmaps","Pixel maps reveal handwriting features point maps miss","Transparency for writer ID: pixel saliency wins over point"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1218,"prompt_tokens":892,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":508,"tokens_out":326,"duration_ms":3807,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:43:49.516197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the deletion and insertion protocol with several different cosine-similarity thresholds and with genuine retrieval pairs (two different pages, sometimes by the same writer, sometimes by different writers) instead of the same image; if the pixel-level map's deletion-score advantage over point-specific maps disappears or reverses under any of these settings, the paper's central quantitative claim does not survive. A second check: match saliency peaks against labeled character occurrences on a larger sample; if the repeated 'y' and 'f' patterns do not appear, the qualitative support claim weakens.","supporting_citations":[{"cited_title":"Towards End-to-End Deep Learning-based Writer Identification","cited_arxiv_id":null,"evidence_quote":"Provides the ResNet-based writer-identification training setup that the transparency methods are applied to."},{"cited_title":"X-MIR: EXplainable Medical Image Retrieval","cited_arxiv_id":null,"evidence_quote":"Supplies the deletion and insertion score protocol that the paper adapts for binarized handwriting images."},{"cited_title":"Visual Explanation for Deep Metric Learning","cited_arxiv_id":null,"evidence_quote":"Introduces the point-specific saliency maps for image pairs that are evaluated here."},{"cited_title":"CVL-Database: An Off-line Database for Writer Retrieval, Writer Identification and Word Spotting","cited_arxiv_id":null,"evidence_quote":"One of the three handwriting datasets used for training and evaluation."},{"cited_title":"Writer Identification Using Edge-Based Direc- tional Features","cited_arxiv_id":null,"evidence_quote":"Second handwriting dataset used for training and evaluation of the transparency techniques."},{"cited_title":"ICDAR 2013 Com- petition on Writer Identification","cited_arxiv_id":null,"evidence_quote":"Third handwriting dataset, with a smaller snippet size, used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the pixel-level saliency map gradient technique with smooth-grad averaging."},{"cited_title":"SigNet: Convolutional Siamese Network for Writer Independent Offline Signature Verification","cited_arxiv_id":"1707.02131","evidence_quote":"Provides the SigNet contrastive architecture used for the writer-verification networks."}],"review_version":1}