{"id":"4de3fc76-b076-4709-8d61-e1b91528317a","arxiv_id":"2507.06029","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FGNS ranks neighbor examples by weighted distance to a class prototype, letting non-experts detect AI misclassifications faster and more accurately than with standard k-NN explanations.","lead":"Researchers invented a new way to explain an AI's image prediction to non-experts: show the most typical examples of the predicted class, not just the most similar-looking ones. In a test with 98 people and unfamiliar Kannada script, this helped them spot the AI's mistakes faster and more accurately, without over-trusting correct answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Judgment-accuracy claims rest on an unspecified Likert-to-binary threshold; without the threshold and sensitivity analyses, the reported true/false judgment improvements are not reproducible or interpretable.","rationale":"The paper's primary contribution is the user-study demonstration that FGNS helps non-experts detect errors. That demonstration has two dependent measures. The 5-point correctness rating interaction is reported with full statistics and is plausible, and the authors honestly report null effects on helpfulness, trust, and satisfaction. The binary judgment-accuracy measure, however, is presented as a headline accuracy result, yet its construction is unspecified. The conversion from an ordinal rating to a binary correct/incorrect judgment requires an explicit threshold; different thresholds are equally defensible and can change both the means and the significance of the reported interaction. Since the manuscript provides neither the raw data nor the analysis code, the threshold cannot be reconstructed. This is the weakest point because it directly undermines the 'more accurate decisions' claim, not merely a secondary metric. The quantitative prototype-distance analyses in Section 4.1 are also circular—they measure the same distance-to-prototype objective FGNS optimizes in Eq. 2—but they do not affect the user-study comparison, so the threshold issue is more load-bearing for the central claim. I therefore agree with the reader's weakest assumption and recommend keeping the verdict at CONDITIONAL until the threshold and robustness checks are reported.","tokens_in":10794,"tokens_out":9904,"duration_ms":114014,"concrete_test":"Recompute Judgment Accuracy from the raw 5-point ratings for each participant under every plausible split (correct = rating >=3, >3, >=4, and with 3 excluded or assigned to one side), and report the Explanation x Judgment interaction and the four means for each split. If the FGNS advantage (0.70 vs 0.65 true judgments; 0.17 vs 0.21 false judgments) or its significance is not stable across splits, the judgment-accuracy claim is an artifact of the unspecified threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FGNS improves non-experts' ability to identify model errors rests on two user-study measures: the raw 5-point correctness rating interaction (F(1,98)=9.11, p=.003) and the 'Judgment Accuracy' results (True M=0.70 vs 0.65; False M=0.17 vs 0.21), where the latter is called 'an implicit measure' in Section 4.2. The paper never states how Likert responses were converted into binary True/False judgments: no threshold (e.g., ratings >=4 vs >3), no rule for neutral 3s, and no exclusion/inclusion criterion. Because the reported means and SDs are aggregates, even plausible thresholds can change the hit and false-alarm rates substantially; the interaction F(1,98)=7.35, p=.008 could shift in either direction. This matters because the abstract and discussion explicitly claim 'more accurate decisions'. Without code or data, a reader cannot determine whether the judgment-accuracy effect is robust or an artifact of a particular, possibly post hoc, binarization. The raw interaction alone is not enough to establish the binary-accuracy claim, which is the headline benefit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Feature-Guided Neighbor Selection (FGNS), a post hoc explanation method that re-ranks k-NN candidate neighbors using a feature-mask-weighted distance to a class prototype, where the masks come from aggregating LIME explanations and validating them with SAGE global importance. The method is evaluated on Kannada-MNIST with two complementary analyses: a quantitative comparison of FGNS versus k-NN neighbor properties (distances to query and prototype, clustering, variance) and a between-subjects user study (N = 98) in which non-experts rated classification correctness, judged correctness as true/false, responded under time measurement, and reported helpfulness, trust, and satisfaction. The headline user-study result is a significant Explanation Type x Classification Type interaction on correctness ratings, F(1,98) = 9.11, p = .003, with FGNS participants rating misclassifications as less correct than k-NN participants did, while ratings for correct classifications were nearly identical. The paper also reports faster response times for FGNS and no significant differences in helpfulness, trust, or satisfaction.","tokens_in":11005,"tokens_out":4367,"duration_ms":55426,"significance":"If the user-study findings hold, FGNS would be a practical, low-cost method for supporting non-expert validation of model predictions in unfamiliar domains, and the paper addresses a real gap in example-based XAI. The manuscript has several strengths: it reports all five user-study measures including null results, it provides effect sizes, it includes a limitations section, and it clearly describes the algorithm pipeline. However, the quantitative evidence for RQ1 is largely circular because the analysis checks a property that FGNS optimizes by construction, and the judgment-accuracy measure rests on an undisclosed Likert-to-binary conversion. These issues weaken the support for the abstract's claim that FGNS produces 'more accurate decisions,' although the raw Likert-scale interaction is still a meaningful result. The central idea is defensible, but the current evaluation does not yet establish the stronger claim.","major_comments":[{"comment":"The Distance Analysis does not independently validate RQ1. Equation (2) defines FGNS selection as minimizing rho * sum_i ||M_i ⊙ (N_candidate - P_c)||^2, and Section 4.1 then shows that FGNS neighbors are closer to P_c under unweighted L2 distance (M = 4.14 vs. 5.55, t(298) = -19.37, p < .001). Because P_c appears directly in the selection objective, closer prototype distance is an expected consequence of the ranking rule rather than an empirical discovery that FGNS neighbors 'better reflect class characteristics.' The Cluster Analysis and Variance Analysis in the same section are restatements of the same distance-to-prototype comparison, not independent evidence. I recommend re-framing this analysis as a sanity check, or adding an external criterion such as expert-rated class representativeness or performance on held-out classes, to support RQ1.","section":"§4.1, Eq. (2)"},{"comment":"The binary True/False judgment measure is derived from the 5-point Likert correctness ratings, but the conversion rule is never specified. The paper calls the measure 'implicit' but does not state the threshold (e.g., whether ratings of 4 and 5 count as 'correct'), how neutral ratings of 3 are assigned, or whether the threshold was chosen before or after seeing the data. Because only aggregate means and standard deviations are reported, the reader cannot reconstruct the binary outcomes, and the reported interaction F(1,98) = 7.35, p = .008, as well as the means (True M = 0.70 vs. 0.65; False M = 0.17 vs. 0.21), may depend heavily on the chosen threshold. Since the abstract and discussion explicitly claim 'more accurate decisions,' the authors should report the exact binarization rule and provide a sensitivity analysis over plausible thresholds (e.g., >=4 vs. >3, with neutral ratings excluded or assigned to one category).","section":"§4.2, Judgment Accuracy"},{"comment":"The response-time analysis reports F(1,93) whereas all other analyses use F(1,98), indicating that five participants or data points were excluded as 'outliers,' but the paper never states how many outliers were removed, by what criterion, or whether exclusion was decided before or after inspecting the data. The speed advantage of FGNS (M = 12.93s vs. 15.68s) is a secondary but non-negligible claim, and the current reporting does not allow the reader to assess whether the exclusion criterion materially affected the result. Please report the exclusion rule and the number of excluded observations.","section":"§4.2 and Table 1, Response Time"}],"minor_comments":[{"comment":"The reference formatting for 'L. Alfeo and Cimino [2024]' is inconsistent with the author-year citation style used throughout the rest of the paper.","section":"§2.2"},{"comment":"The y-axis of Figure 4 is labeled 'Frequency,' but the figure shows density-like distributions of Euclidean distances, and no units or binning details are provided; a box plot or density plot with labeled axes would be clearer.","section":"§4.1, Figure 4"},{"comment":"The discussion describes the lower False Judgment rate as reducing 'false positives,' but the operationalization counts incorrect correctness judgments, not false positives in the signal-detection sense; using 'false judgments' throughout would avoid confusion.","section":"§5"},{"comment":"The symbol k is used both for the number of LIME superpixels and for the number of K-means clusters (k = 7); these are distinct quantities and should be renamed or explicitly differentiated.","section":"Algorithm 1 and §3.3"},{"comment":"The manuscript does not include a data or code availability statement; given that the judgment-accuracy measure depends on an undisclosed threshold, releasing anonymized item-level ratings would substantially aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a workshop-length contribution (accepted at an IJCAI 2025 workshop) being considered for a fuller venue. The core idea is sound and the user-study interaction on correctness ratings is a genuine result, but the circular quantitative validation and the underspecified judgment-accuracy conversion are load-bearing issues that can be fixed within the manuscript's scope. I recommend major revision rather than rejection, and I would encourage the editor to require the sensitivity analysis for the binarization threshold before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, FGNS is a real, modest contribution: it re-ranks k-NN explanation candidates using a weighted distance to a class prototype, where the weights come from LIME local masks filtered by SAGE global importance. That combination is new, and the user study is more honest than many XAI papers—it reports null results on helpfulness, trust, and satisfaction, and the central interaction on correctness ratings is significant (F(1,98)=9.11, p=.003) with a small effect size. Second, the paper's claim that users made 'more accurate decisions' relies on a judgment-accuracy measure whose Likert-to-binary threshold is never stated. The paper calls it 'implicit' but gives no rule. Those 0.70 vs 0.65 and 0.17 vs 0.21 numbers could shift under different binning, and the reader cannot reproduce them. That is a genuine problem for the headline, though not for the raw rating interaction.\n\nThe quantitative Section 4.1 is also partly circular. Showing FGNS neighbors are closer to the class prototype is exactly the objective in Eq. 2. So it confirms the implementation works, but it is not independent evidence that the selected neighbors are 'better' in any human-meaningful sense. The cluster and variance analyses are just aggregations of the same distance. Fine as a sanity check, not as validation.\n\nOther soft spots: one clean-image dataset, hand-set hyperparameters (k=7, rho=1.0, IoU thresholds), no code or data, and effect sizes are modest—2.49 vs 2.83 on a 5-point scale is about half a point. These are not fatal, but they mean the external validity is narrow.\n\nWhat the paper does well: the method is clearly specified, the procedure is concrete, and the authors situate FGNS against Delaney et al. and the COLE-HP line. The honest reporting of non-significant trust/satisfaction results is a credit. The judgment-accuracy threshold and the missing materials are fixable in revision.\n\nWho this is for: XAI researchers interested in example-based explanations and non-expert evaluation. It deserves a serious referee. My recommendation: send it to peer review, but with a firm request for the binarization rule, sensitivity analysis, and ideally the analysis scripts. Without those, the binary-accuracy claims should be treated as unverified.","headline":"FGNS is a plausible, modestly supported method for helping non-experts spot misclassifications, but the paper's quantitative validation is partly circular and the judgment-accuracy analysis has an unspecified Likert-to-binary threshold.","tokens_in":11653,"tokens_out":2930,"would_cite":false,"duration_ms":31351,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting class-prototype-aligned explanation examples helps non-experts spot AI model errors.","keywords":["Explainable AI","example-based explanations","feature importance","k-nearest neighbors","class prototypes","non-expert users","Kannada-MNIST","model evaluation"],"falsifier":"Re-analyse or replicate the user study with an explicit, pre-registered binary threshold (for example, ratings 1-2 count as 'incorrect' and 3-5 as 'correct') and check whether FGNS still beats k-NN on judgment accuracy and response time; alternatively, run the same paradigm on natural images with less well-defined class prototypes and see whether the error-detection advantage disappears.","tokens_in":10477,"feed_emoji":"🎯","tokens_out":6050,"duration_ms":54375,"temperature":0.7,"pith_summary":"This paper introduces Feature-Guided Neighbor Selection (FGNS), a post hoc explanation method that decides which example images to show beside a model prediction. Instead of choosing the nearest neighbors in feature space, FGNS re-ranks candidates by how far they deviate from a prototype of the predicted class, weighting the comparison with features that LIME highlights locally and SAGE validates globally. In a study with 98 non-experts judging Kannada digit predictions, participants given FGNS examples rated misclassifications as less correct and made faster, more accurate right/wrong judgments than participants given ordinary k-NN examples, while rating correct predictions almost identically. The paper claims that class-representative, prototype-aligned neighbors support people without domain expertise in validating AI decisions, and that this performance gain does not automatically raise perceived trust.","feed_headline":"Prototype-guided neighbors help non-experts catch AI mistakes","feed_subtitle":"FGNS explanations improved error detection in a user study on unfamiliar script recognition, without inflating trust.","key_machinery":"The central object is the feature-guided ranking loss $L_{\\text{feature}} = \\rho \\sum_{i=1}^k \\| M_i \\odot (I_{\\text{candidate}} - P_c) \\|_2^2$, where $P_c$ is the pixel-wise median prototype of the predicted class $c$, $M_i$ are binary feature masks obtained by aggregating LIME's local explanations and validating them with SAGE's global importance, and $\\odot$ is element-wise multiplication. This loss measures how much a candidate training image deviates from the class prototype in the regions the model treats as characteristic, and FGNS selects the three lowest-loss neighbors as the explanation. The six-step pipeline (local LIME extraction, class-level aggregation, SAGE validation, clustering and diversity filtering, prototype construction, and re-ranked selection) turns the abstract idea of class representativeness into a concrete, computable neighbor ordering.","core_discovery":"The central claim is that FGNS improves non-expert evaluation of model predictions by selecting neighbors that reflect class characteristics rather than merely minimizing feature-space distance. FGNS constructs a class prototype from the pixel-wise median of training images and integrates LIME's local superpixel attributions with SAGE's global importance scores into binary feature masks. It then ranks training instances for the predicted class by the L2 distance from each candidate to the prototype, weighted by those masks, and displays the three lowest-loss examples. In the user study this produced a significant Explanation Type $\\times$ Classification Type interaction on correctness ratings, $F(1,98)=9.11$, $p=.003$, with FGNS users rating misclassifications as less correct ($M=2.49$ vs $2.83$) while rating correct classifications nearly identically ($M=4.33$ vs $4.35$); it also yielded better true/false judgment rates ($0.70$ vs $0.65$ true, $0.17$ vs $0.21$ false) and faster responses ($12.93$s vs $15.68$s per item). Quantitatively, FGNS neighbors sat closer to class prototypes and clustered more tightly than k-NN neighbors, supporting the interpretation that prototype alignment, not proximity, drives the user-level effect.","pith_inferences":["If prototype-aligned neighbors improve non-expert error detection on unfamiliar script, the same design principle may extend to other unfamiliar perceptual domains such as medical or satellite imagery, where class prototypes can be estimated from training data; this is an untested implication.","The observed dissociation between improved task performance and unchanged self-reported trust suggests trust questionnaires may be insensitive to explanation quality; a testable follow-up would measure behavioural reliance, such as how often users accept or override predictions.","Because FGNS depends on well-formed prototypes and clean superpixels, its advantage could shrink on natural images with high intra-class variation; a direct comparison on a more diverse dataset would map that boundary."],"forward_implications":["For image classification tasks where users lack domain expertise, explanation examples should be selected for prototype alignment rather than feature-space proximity.","FGNS-style neighbor selection can replace proximity-based k-NN ranking in twin-system XAI architectures without retraining the predictor.","Prototype-guided explanations improve error detection without inflating trust or satisfaction, potentially reducing overreliance while keeping users appropriately skeptical.","FGNS neighbor sets are more consistent across queries (lower variance around the class prototype), making explanation output more stable and interpretable."],"supporting_citations":[{"why":"Supplies LIME local feature attributions that FGNS aggregates into superpixel masks.","marker":"[Ribeiro et al., 2016]"},{"why":"Supplies SAGE global importance scores used to validate and filter the aggregated masks.","marker":"[Covert et al., 2020]"},{"why":"Provides the twin-system CNN plus k-NN explanation framework that FGNS modifies by re-ranking neighbors.","marker":"[Kenny et al., 2021]"},{"why":"Documents the non-expert difficulty with k-NN explanations that motivates FGNS.","marker":"[Ford and Keane, 2023]"},{"why":"Supports the principle that users prefer prototype-aligned examples, grounding FGNS's design rationale.","marker":"[Delaney et al., 2023]"},{"why":"Earlier case-based reasoning evidence that examples near class boundaries offer more insight, informing the choice of class-representative neighbors.","marker":"[Doyle et al., 2004]"}],"fun_headline_variants":["Feature-guided neighbors help non-experts spot AI errors faster","Prototype-weighted neighbors improve non-expert error detection","FGNS: feature-aware neighbors make AI errors clearer to novices","Neighbors selected by feature importance reveal AI missteps","Class-prototype neighbors beat k-NN for non-expert AI checks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline user-study results depend on an unstated rule for converting 5-point correctness ratings into binary right/wrong judgments, so the reported accuracy gains over k-NN hinge on a threshold the paper never specifies.","fun_headline_variants_meta":{"raw":{"variants":["Feature-guided neighbors help non-experts spot AI errors faster","Prototype-weighted neighbors improve non-expert error detection","FGNS: feature-aware neighbors make AI errors clearer to novices","Neighbors selected by feature importance reveal AI missteps","Class-prototype neighbors beat k-NN for non-expert AI checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":2982,"prompt_tokens":959,"completion_tokens":2023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1939}},"tokens_in":575,"tokens_out":2023,"duration_ms":16833,"temperature":1.0,"reasoning_tokens":1939,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:12:36.789215+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-analyse or replicate the user study with an explicit, pre-registered binary threshold (for example, ratings 1-2 count as 'incorrect' and 3-5 as 'correct') and check whether FGNS still beats k-NN on judgment accuracy and response time; alternatively, run the same paradigm on natural images with less well-defined class prototypes and see whether the error-detection advantage disappears.","supporting_citations":[{"cited_title":"”why should i trust you?”: Ex- plaining the predictions of any classifier","cited_arxiv_id":null,"evidence_quote":"Supplies LIME local feature attributions that FGNS aggregates into superpixel masks."},{"cited_title":"Covert, Scott Lundberg, and Su- In Lee","cited_arxiv_id":null,"evidence_quote":"Supplies SAGE global importance scores used to validate and filter the aggregated masks."},{"cited_title":"Explaining black-box classi- fiers using post-hoc explanations-by-example: The effect of explanations and error-rates in xai user studies","cited_arxiv_id":null,"evidence_quote":"Provides the twin-system CNN plus k-NN explanation framework that FGNS modifies by re-ranking neighbors."},{"cited_title":"Explanation oriented retrieval","cited_arxiv_id":null,"evidence_quote":"Earlier case-based reasoning evidence that examples near class boundaries offer more insight, informing the choice of class-representative neighbors."}],"review_version":1}