{"id":"5d1d34df-1b57-4614-a31b-2383feb39165","arxiv_id":"2501.02042","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Many similarity measures used to judge adversarial text-XAI attacks are too sensitive, and synonymity-weighted variants change the measured success rates substantially.","lead":"This paper tests how the choice of similarity measure changes whether an adversarial attack on text explanations is judged successful. It proposes weighting explanation similarity by word synonymity, claiming this gives a truer picture of how stable LIME explanations are.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'more accurate' claim rests on using the same GloVe embedding for both candidate generation and synonymity scoring, with no external semantic ground truth; the observed reductions in attack success may be partly by construction.","rationale":"The reader identified the load-bearing weakness as the unvalidated synonymity function: GloVe cosine similarity is used both to generate candidate replacements and to score synonymity, with no human or independent semantic baseline. I agree with that assessment. The paper's positive contribution is the empirical demonstration that similarity-measure choice dramatically changes attack success rates, which is a real and useful observation. However, the stronger claim, that synonymity-weighted measures provide 'more accurate estimates of the actual weakness of XAI methods to adversarial examples,' is not established by the experiments as designed. Because the perturbation search selects words that are close in the same embedding space used by the weighted similarity measure, the large reductions in attack success are at least partly a tautological consequence of the evaluation metric rewarding the search's own preferences. This is not a claim that the authors acted improperly; it is a methodological gap that could be closed with an external ground truth. The paper also mislabels Eq. 2 as Kendall's tau, which weakens the specific Kendall-related conclusions but does not change the overall verdict. Given that the issue is addressable by adding human or independent semantic baselines and by clarifying/correcting the measure definitions, the appropriate outcome remains conditional acceptance rather than rejection. The reader's verdict already captures this, so no adjustment is needed.","tokens_in":12716,"tokens_out":3056,"duration_ms":32740,"concrete_test":"Take a sample of, say, 100 perturbed explanations generated by the current pipeline that are recorded as successful attacks at tau=0.5 under standard Jaccard but failed under weighted Jaccard. Obtain human semantic-equivalence ratings for each original-perturbed pair, or, failing that, scores from a held-out contextual metric such as BERTScore or SBERT cosine. Recompute weighted attack success using the independent score as the synonymity function. If the independent score correlates only weakly with GloVe cosine (e.g., rho < 0.5) or if human raters judge that failed attacks do change meaning, then the weighted measure is rejecting attacks that genuinely alter the explanation, and the accuracy claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that synonymity weighting yields more accurate stability estimates requires an external criterion for explanation-level semantic equivalence, and the paper supplies none. Empirically, Section 4.6 defines Syn as cosine similarity on GloVe Twitter 27B-25d, while Section 3.1 selects perturbations as the n nearest neighbors in some embedding space, apparently the same family of embeddings. Thus a replacement is chosen precisely because its GloVe cosine to the original word is high, and it is then scored as highly similar by the same function. The observed collapse of Jaccard and Spearman attack success under weighting (Section 4.7) is therefore partly built into the pipeline: it demonstrates self-consistency between the generator's embedding and the evaluator's embedding more than it demonstrates semantic accuracy. No human ratings, no independent embedding (e.g., contextual embeddings or BERTScore), and no definition of 'meaning preserved' are provided. The abstract's 'more accurate estimates' is therefore an unsupported superlative. A secondary internal inconsistency: Eq. 2 is labelled Kendall's tau but actually counts position mismatches, not pairwise inversions, so the Kendall sensitivity conclusions should be re-derived from the correct statistic. The paper's descriptive finding, that measure choice swings success rates from 0 to 1 (Table 3), is solid and useful; it is the accuracy/validity component that is under-supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies how the choice of similarity measure affects estimates of adversarial stability for LIME explanations on text. The authors run a greedy perturbation search over two text datasets under nine ranked-list similarity measures and four success thresholds, and they report attack success rates for each combination. They then propose synonymity-weighted variants of Jaccard, Kendall, Spearman, and RBO, scoring substituted words by GloVe cosine similarity, and show that weighting dramatically lowers the reported attack success rates for Jaccard and Spearman. Their descriptive claim is that standard measures disagree strongly — Kendall appears nearly always 'successful' while Jaccard and RBO0.9 are very conservative. Their prescriptive claim is that the weighted measures give 'more accurate estimates' of the true weakness of XAI methods to adversarial examples.","tokens_in":12986,"tokens_out":8142,"duration_ms":82310,"significance":"If the accuracy claim were supported, this would be a useful methodological contribution to adversarial text-XAI evaluation: it would show that rank-list measures can overstate instability and that synonymity-aware variants are preferable. The descriptive comparison is the paper's real strength: Tables 2 and 3 show cleanly that the choice of measure can move attack success rates from near zero to near one, and the experimental protocol is described in enough detail to be replicated. The Limitations section is candid about several threats to generality. However, the central 'more accurate' claim currently lacks an external semantic ground truth; the only evidence is the weighted measures' own reduction in success rates, and the synonymity function appears to come from the same embedding family used in perturbation search. The paper is therefore best viewed as a solid sensitivity study whose validity conclusion needs additional support or careful reframing.","major_comments":[{"comment":"The central claim that synonymity weighting yields 'more accurate estimates' is not externally grounded. Section 4.6 defines Syn as cosine similarity on the GloVe Twitter 27B-25d embedding, while Section 3.1 says perturbations are chosen as the n nearest neighbors in 'some embedding space' without naming it. If that embedding space is GloVe, then a replacement is selected precisely because its GloVe cosine to the original word is high, and it is then scored as highly similar by the same function; the reductions in attack success in Table 3 are then partly a self-consistency artifact, not evidence of semantic accuracy. The Limitations section itself concedes that 'superior estimates of synonymity can probably be obtained with alternative methods.' To support the abstract's 'more accurate estimates,' the authors should either specify the perturbation embedding and show it is independent of the evaluation embedding, or validate the weighted measures against human judgments of meaning preservation, an independent contextual-embedding baseline such as BERTScore, or another external criterion. Without such validation, the paper can claim a measure-sensitivity effect but not accuracy.","section":"§3.2, Eq. (2); §3.4; §4.5, Eq. (6)"},{"comment":"Equation (2) is not Kendall's tau. The prose says the measure 'counts the number of pairwise inversions,' but the displayed formula sums position mismatches, 1[A[i] ≠ B[i]], plus a size-difference term. That is a Hamming distance over aligned positions, not the number of discordant pairs. Because the near-100% attack success rates for 'Kendall' in Table 2 and the sensitivity conclusions in Section 3.4 are based on this misdefined statistic, the authors should re-run the analysis with the true Kendall tau distance or rename the measure consistently. The weighted extension in Eq. (6) inherits the same problem, so the 'Kendall w' results in Table 2 should also be revisited.","section":"§3.2, Eq. (2); §3.4; §4.5, Eq. (6)"},{"comment":"The weighted measure definitions are incomplete for two cases that occur in the reported experiments. Section 4.2 says that elements in A may be mapped to the null mapping, but Eq. (5) evaluates Syn(A[i], M(A[i])) with no branch for Syn(a, null), so the numerator is undefined for disjoint elements. In Eq. (7), the term |i−j|/Syn(a,b) divides by Syn(a,b); the paper constrains Syn to [0,1] but gives no rule for converting raw GloVe cosine values, which can be negative or zero, into this interval, and a zero synonymity score would make the quotient undefined. Because the Jaccard-w and Spearman-w results in Table 3 depend on these unspecified choices, the implementation is not fully reproducible as written. Please add explicit penalty branches for null mappings and a clamping or rescaling rule for Syn.","section":"§4.2, §4.5, Eqs. (5) and (7)"}],"minor_comments":[{"comment":"The phrase 'some embedding space' is needlessly vague and is directly relevant to the circularity concern; please name the embedding used for nearest-neighbor perturbation generation.","section":"§3.1"},{"comment":"There are typos in this section: 'Jaccad Index' should be 'Jaccard Index,' and 'synonymity weighting„' contains a stray double comma.","section":"§4.6"},{"comment":"In Eq. (6), the variables a and b in the summation are not bound; they should be A[i] and B[i] to match the indicator and the penalty term.","section":"§4.5, Eq. (6)"},{"comment":"The attack-success rates are reported without confidence intervals or the raw counts from which they are computed; given that each batch consists of 50 examples, reporting counts or intervals would help readers judge the stability of the rates.","section":"§3.3, Tables 2–3"},{"comment":"The discussion of RBO says 'the lack of overall success associated with RBO0.9 makes a firm judgment on its usefulness premature'; this is appropriately cautious, but the same caution is not consistently applied to the weighted Jaccard and Spearman results, which are discussed as demonstrating inaccuracy of the base measures.","section":"§4.7"}],"recommendation":"major_revision","confidential_remarks":"I want to flag the same-embedding circularity as the main correctness risk; the paper needs either an independent semantic validation or a substantial weakening of the accuracy claim. The descriptive measure-comparison contribution is solid and worth publishing, so I would not recommend rejection. I would also ask the handling editor to ensure that the authors either correct the Kendall definition or stop calling it Kendall's tau, since the current equation contradicts the prose."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the arXiv:2501.02042. The headline: the descriptive comparison of similarity measures for adversarial text XAI is genuinely useful and should be published; the central claim that synonymity weighting gives 'more accurate' stability estimates is not supported by the evidence as written.\n\nWhat's new: a systematic comparison of nine rank-list similarity measures (Jaccard, Kendall, Spearman, RBO at several persistence values) on LIME attacks for two text datasets. The key empirical observation is that attack success swings from near zero to near one depending on the measure (Table 2). That is a real methodological warning for anyone reporting XAI robustness results. The synonymity-weighted variants of Jaccard, Spearman, and RBO are a reasonable adaptation of element-weighted similarity, and the authors credit prior work honestly. The finding that RBO is largely insensitive to weighting is interesting.\n\nSoft spots, in proportion:\n\n1. The 'more accurate' claim is the load-bearing weakness. Perturbations are generated as embedding nearest neighbors (Sec 3.1) and synonymity is scored with the same embedding family (GloVe cosine, Sec 4.6). So the pipeline is self-consistent by construction: a replacement that survives the generator is already close under the evaluator. The dramatic collapse of Jaccard/Spearman success under weighting (Table 3) likely reflects that self-consistency more than semantic equivalence. No human ratings, no independent semantic baseline (e.g., BERTScore or a different embedding), no definition of 'meaning preserved.' This is exactly the kind of circularity that the abstract's 'more accurate' cannot carry.\n\n2. Equation 2 is mislabeled as Kendall's tau. As written it is a per-position mismatch count plus a size penalty, not the number of pairwise inversions. The Kendall sensitivity conclusions need to be re-derived with the correct statistic.\n\n3. Speck of dust: the weighted Spearman formula divides by Syn(a,b), which is undefined for Syn=0. The paper does not state how this case is handled. Minor, but fixable.\n\n4. Code is not provided despite the abstract promise.\n\nWho it is for: researchers studying robustness of local surrogate explanations, especially adversarial text. The sensitivity map across measures is a useful reference and a good basis for discussion.\n\nRecommendation: send to peer review. A serious referee should require an external semantic validation (even a small human study or an alternate embedding), a corrected Kendall equation, and a softened accuracy claim. The descriptive contribution stands on its own and warrants referee time.","headline":"Useful sensitivity map for text-XAI similarity measures, but the 'more accurate' synonymity-weighting claim rests on a circular embedding choice and an unvalidated semantic assumption.","tokens_in":13506,"tokens_out":2666,"would_cite":true,"duration_ms":24464,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LIME's apparent instability across adversarial text examples is largely a side effect of overly sensitive similarity measures, and synonymity-weighted versions of those measures give a more accurate picture of robustness.","keywords":["explainable AI","adversarial attacks on XAI","LIME stability","similarity measures","ranked lists","synonymity weighting","text classification","GloVe embeddings"],"falsifier":"Recompute the synonymity weighting with an independent semantic scoring source, for example a thesaurus-based synonymy relation or a contextual embedding model not used to generate the candidate replacements, on the same GB and S2D attack batches. If attack success rates under the weighted measures return to near their unweighted levels, the reported accuracy gain is an artifact of GloVe cosine similarity rather than a true reflection of semantic preservation.","tokens_in":12496,"feed_emoji":"🔤","tokens_out":8402,"duration_ms":79053,"temperature":0.7,"pith_summary":"This paper tries to establish that the similarity measure used to score adversarial text attacks on local surrogate explainers like LIME is a primary driver of reported instability, not a neutral meter. It shows that rank-sensitive measures such as Kendall's tau treat small synonym substitutions as near-total explanation failure, producing attack success rates close to 100 percent even when the original and perturbed documents remain semantically close. It then introduces synonymity weighting, in which each replaced word contributes cosine similarity to its replacement rather than strict equality, and reports that weighted versions of Jaccard and Spearman drop most attack successes to zero or near zero. The intended upshot is that prior estimates of LIME's vulnerability on text are inflated by the choice of measure, and that practitioners should calibrate sensitivity or adopt weighted measures before judging an XAI method weak.","feed_headline":"Synonymity weighting shows reported LIME attacks are inflated","feed_subtitle":"Rank-list measures like Kendall's tau call synonym swaps failures; word-similarity scores flip many reported attacks.","key_machinery":"The carrying mechanism is a synonymity function $\\mathrm{Syn}(a,b)$, defined as the cosine similarity between GloVe word vectors for the original feature and its replacement, constrained to $\\mathrm{Syn}(x,x)=1$, embedded into rank-list measures through a mapping from each perturbed word back to its original. Applied to Jaccard, it replaces the strict intersection numerator with a sum of pairwise synonymity scores; applied to Kendall's tau it scales a dissonant pair's penalty by $1-\\mathrm{Syn}(a,b)$; applied to Spearman's footrule it divides the positional distance by $\\mathrm{Syn}(a,b)$ up to the maximum footrule distance; and applied to RBO it enlarges each depth-$d$ intersection by the synonymity of mapped disjoint pairs. This softens the discontinuity between a word and a near-synonym, and it is what turns many reported attack successes into failures.","core_discovery":"The paper's central claim is that many standard similarity measures for ranked lists are overly sensitive for text-based adversarial XAI, and this sensitivity, not the surrogate itself, accounts for most apparent instability. Experimentally, the same greedy attack against LIME explanations on two text datasets with a DistilBERT classifier is scored under nine measures: Kendall's tau reports near-certain attack success at every threshold, Jaccard and Spearman rise to near-certain success at high similarity thresholds, while RBO gives moderate rates. When the same attacks are rescored with synonymity-weighted variants, Jaccard and Spearman success collapses, leading the authors to state that conclusions about XAI instability drawn with unweighted Jaccard or Spearman should be viewed with suspicion. Kendall remains too sensitive even weighted, and RBO's own depth weighting already absorbs much of the synonymity effect.","pith_inferences":["The same measure-sensitivity problem should appear outside text: any XAI stability evaluation that scores explanations as discrete feature lists will overstate instability when deletion or replacement changes token identity but not meaning, so synonymity or perceptual-similarity weighting is a general remedy rather than a text-only one.","Because the attack's replacement selection and the synonymity score both use GloVe cosine similarity, a circularity check is missing: the weighted measure may simply reward the search for choosing words close in the same embedding space, and a human rating study or a thesaurus-based synonymy function would separate semantic preservation from embedding-space proximity.","The paper's Limitations section concedes that transferability to other XAI methods, untested measures, and better synonymity estimates are open; the accuracy result should therefore be read as conditional on GloVe-based cosine synonymity and on LIME-style ranked lists.","A practical testable extension is to choose the success threshold $\\tau$ per measure based on the measure's own sensitivity, so that a random synonym substitution is not automatically classified as a success; calibrating thresholds this way would make cross-measure robustness comparisons meaningful."],"forward_implications":["Reanalyzing prior adversarial-text-XAI results with synonymity-weighted Jaccard or Spearman can convert many reported successes into failures, so published instability rates for LIME should be re-examined before being used to disqualify the method.","Kendall's tau, even in weighted form, remains too sensitive for text XAI stability work and should not be trusted for attack-success judgments.","RBO with $p = 0.7$ or $p = 0.9$ is the most defensible standard measure: it changes little under synonymity weighting, indicating its depth-based weighting already compensates for the sensitivity problem.","Synonymity weighting costs negligible computation because explanation generation dominates, so adopting a weighted measure does not slow the adversarial pipeline."],"supporting_citations":[{"why":"Supplies the greedy attack algorithm, the LIME sampling instability analysis, and the experimental setup that this paper extends by varying similarity measures and success thresholds.","marker":"(Burger et al., 2023)"},{"why":"Defines Rank-biased Overlap (RBO), the baseline measure used throughout and the target of one synonymity-weighted extension.","marker":"(Webber et al., 2010)"},{"why":"Defines LIME, the local surrogate method whose ranked-list explanations are the objects being compared for stability.","marker":"(Ribeiro et al., 2016)"},{"why":"Supplies DistilBERT, the explained classifier used for both experimental datasets.","marker":"(Sanh et al., 2019)"},{"why":"Provides the Twitter gender bias dataset used for the GB experimental results.","marker":"(Dinan et al., 2020)"},{"why":"Provides the symptoms-to-diagnosis dataset used for the S2D experimental results.","marker":"(Kaggle)"},{"why":"Establishes the fragility-of-interpretations problem that motivates treating explanation similarity as the object of investigation.","marker":"(Ghorbani et al., 2019)"},{"why":"Earlier demonstration that LIME and SHAP can be fooled, the line of work whose conclusions this paper argues were measure-dependent.","marker":"(Slack et al., 2020)"}],"fun_headline_variants":["Standard rank metrics overstate LIME instability","Synonym-aware scoring deflates reported XAI attacks","Jaccard and Spearman misjudge text explanation stability","Kendall's tau too sensitive for text explainability attacks","Synonymity weight rescues LIME stability estimates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GloVe cosine similarity between a replaced word and its replacement is a valid measure of whether the explanation's meaning actually stayed the same, and that using the same embeddings to choose replacements and to score them does not systematically bias the result.","fun_headline_variants_meta":{"raw":{"variants":["Standard rank metrics overstate LIME instability","Synonym-aware scoring deflates reported XAI attacks","Jaccard and Spearman misjudge text explanation stability","Kendall's tau too sensitive for text explainability attacks","Synonymity weight rescues LIME stability estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1480,"prompt_tokens":956,"completion_tokens":524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":572,"tokens_out":524,"duration_ms":6009,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:15:10.684202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the synonymity weighting with an independent semantic scoring source, for example a thesaurus-based synonymy relation or a contextual embedding model not used to generate the candidate replacements, on the same GB and S2D attack batches. If attack success rates under the weighted measures return to near their unweighted levels, the reported accuracy gain is an artifact of GloVe cosine similarity rather than a true reflection of semantic preservation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the greedy attack algorithm, the LIME sampling instability analysis, and the experimental setup that this paper extends by varying similarity measures and success thresholds."}],"review_version":1}