{"id":"cffa96ba-0e04-4c7e-acac-33e9e8bcde6e","arxiv_id":"2501.09006","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A genetic algorithm finds smaller adversarial perturbations of LIME text explanations than a greedy baseline in some settings, but the stability estimates lack statistical support.","lead":"This paper tests whether a genetic algorithm can find smaller word-level changes that break LIME's text explanations compared with the greedy search used in prior work. It reports some smaller attacks, but the evidence rests on only 20 examples per dataset with no error bars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed improvement in minimum-perturbation estimates rests on single stochastic runs with no variance reporting; repeat-seed tests are needed before Section 7's conclusion can be accepted.","rationale":"The reader's weakest assumption correctly identifies that a single unseeded run of a small genetic algorithm on 20 examples cannot reliably estimate minimum perturbation counts. My reading of the paper confirms this as the most load-bearing weakness. The genetic algorithm is explicitly described as restrictive in Section 4, and Section 5 gives no indication of repeated trials or statistical testing. The central claim in Section 7, framed as a demonstrated improvement in stability estimates, therefore lacks the evidentiary base needed to distinguish algorithmic efficacy from sampling noise.\n\nThat said, the paper is not without merit. The qualitative examples in Figures 1–3 show that the GA can occasionally find semantically better single-word replacements than the greedy search, and some Table 5 entries (e.g., S2D RBO0.5 at 30%: 1 vs 2; S2D Jaccard w at 60%: 3 vs 6) suggest a plausible direction. These are existence results, not established population-level improvements. The absence of target classifier, LIME configuration, and embedding details also prevents independent reproduction, which is a further gap but secondary to the variance issue.\n\nMy proposed test directly targets the claim: if the GA's lower minima do not persist across a majority of seeds, the 'demonstrated effectiveness' language in Section 7 and the abstract should be weakened to a report of single-instance improvements. If the minima do persist, the claim would be substantially strengthened. Since the reader already rejected on insufficient support and my concern aligns, I recommend no change to the verdict.","tokens_in":11075,"tokens_out":3845,"duration_ms":38924,"concrete_test":"Run the GA with 20 distinct random seeds on the same 20 GB and S2D examples for every cell in Tables 2–5 where GA improved on GS (notably GB Spearman w 30%, S2D Jaccard w 60%, S2D Spearman w 30%). Record the distribution of minima and success rates; the central claim holds only if the GA's median minimum is at least one perturbation lower than GS and the improvement appears in a majority of seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 7) is that the alternate genetic search 'demonstrated effectiveness in both reducing the number of perturbations' and matching or exceeding greedy search. This rests on comparisons in Tables 2–5, e.g., GB Spearman w success at τ=30% rising from 0 to 0.24 and S2D Spearman w minimum dropping from 7 to 3. But Section 4 fixes population size 10 and 10 generations, and Section 5 reports 'twenty examples' with no mention of repeated runs, seeds, or variance. A stochastic GA of this size can produce a lucky low-perturbation individual in a single run; without a distribution over seeds, the observed differences may be noise. The tables also contain malformed entries (e.g., Table 2 '0.3 5', '0 .82') and missing cells ('-'), which complicates verification. The search is not exhaustive, so Table 5's 'minimum' is the best single-run value, not an established lower bound. Since the abstract claims 'superior comparisons of the stability of explainability methods,' the lack of repeated trials is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the greedy word-replacement search used in earlier adversarial attacks on LIME's text explanations with a genetic algorithm, with the goal of finding the smallest number of perturbations needed to cross a fixed explanation-similarity threshold. The formal setup includes constraints for preserving the target model's prediction, semantic similarity of the document, a perturbation cap, and protection of top-k explanation features. The authors compare the genetic algorithm against their prior greedy search on 20 examples from each of two datasets, reporting success rates, mean similarities, perturbation rates, and minimum perturbation counts across nine similarity measures. The central claim, stated in Section 7, is that the genetic algorithm reduces the number of perturbations while matching or exceeding the greedy search in attack success and perturbed document quality.","tokens_in":11242,"tokens_out":10919,"duration_ms":104099,"significance":"If the empirical claims were reliable, the paper would contribute a useful methodological step: moving adversarial XAI research from demonstrating the existence of LIME instability toward quantifying the minimal perturbation needed to expose a given level of instability. The idea of using threshold-based similarity as a stopping criterion is reasonable, and the comparison of search strategies is a natural way to improve lower-bound estimates. However, the current evidence is too thin for the stated conclusions: the experiments are a single run of a small stochastic genetic algorithm on 20 examples per dataset, with no repeated seeds, no error bars, and no statistical tests. The target model, LIME settings, and replacement embedding are not reported, and several tables contain malformed or inconsistent entries. The paper's value therefore depends on a substantial empirical revision rather than on the current results.","major_comments":[{"comment":"The central comparison rests on a single run of a stochastic genetic algorithm (population size 10, 10 generations) over 20 examples per dataset, with no random seeds, repeated trials, confidence intervals, or statistical tests. Since the GA is stochastic and LIME itself uses randomized sampling, a point difference such as the GB Spearman w success rate at τ=30% (0 for GS, 0.24 for GA) or the S2D Spearman w minimum perturbation count (7 vs 3) could be sampling noise. This is load-bearing because Section 7's conclusion is stated directly from these point estimates.","section":"§5, Tables 2–5"},{"comment":"The experimental protocol is not reproducible. The target classifier f is never identified; LIME's sampling parameters (number of perturbed samples, feature set size, kernel width, surrogate hyperparameters) are not reported; and the embedding space and candidate replacement set used by both searches are not specified. These choices determine which perturbations are possible and which explanations are generated, so the comparisons in Tables 2–5 cannot be independently verified or meaningfully interpreted.","section":"§3.3, §5"},{"comment":"The formal objective is internally inconsistent. Equation (5) treats a successful attack as reaching similarity at or below the threshold τ, but Equation (7) states Sime(edb, edp) > γ, and the relationship between γ and τ is never defined. Moreover, Equation (7) is a feasibility constraint set, not an optimization of the number of perturbations, even though Section 3.3 frames the goal as finding minimal perturbations and Table 5 is labeled 'Minimum Perturbation(s)'. Since the GA fitness function is said to be Objective Function (7), it is unclear how the algorithm prefers shorter perturbation sets over merely feasible ones. Equation (7) also contains the typo i ≤ ε|f| instead of i ≤ ε|db|.","section":"§3.1, Eq. (7)"},{"comment":"Several table entries are malformed or internally inconsistent, which blocks verification. Examples include '0.3 5', '0 .82', and '0 .65' in Table 2, and in Table 3 the GS Jaccard w cell at GB τ=50% and 60% is shown as 0 even though Table 2 reports a 0% success rate for the same cells, conflicting with the '-' convention used elsewhere for 'no successful attack'. The tables need to be regenerated with a consistent format, with all empty cells explicitly marked.","section":"Tables 2–5"},{"comment":"The claim that the GA 'has the capacity to meet or exceed the original greedy search in terms of perturbed document quality' is supported only by three qualitative examples, one of which (Figure 3) shows the greedy search producing a better perturbation. There is no quantitative document-quality metric or systematic evaluation, and Section 4 admits the GA parameters are restrictive while Section 5 states that larger datasets could not be tested. The conclusion in Section 7 therefore overstates the strength and generality of the evidence.","section":"§6, §7"}],"minor_comments":[{"comment":"The sentence 'This list ensures out criteria of generalizability' contains a typo; it should read 'our criteria'.","section":"§2"},{"comment":"The phrase 'solutions to are composed of a population' is ungrammatical and should be rewritten.","section":"§4"},{"comment":"The four thresholds (30–60%) and the chosen similarity measures are taken from Burger et al. (2024) without definitions; the paper should include a self-contained summary of the measures or at least define the weighted variants, especially since the weighted measures are central to several reported improvements.","section":"§3.2"},{"comment":"Equation (7) uses |f| in the perturbation cap constraint, whereas Equation (4) uses |db|; the notation should be made consistent.","section":"§3.1"},{"comment":"The paper does not provide a code or data availability statement; given that the experiments are not reproducible from the text alone, the authors should release code and seeds or at least describe the exact experimental pipeline.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a preliminary study whose conclusions exceed the evidence. The core idea is reasonable and the identified problems are in principle addressable, so I do not recommend outright rejection, but any revision would need to be substantial: multiple seeded runs, full reporting of the target model and LIME configuration, corrected and self-consistent tables, and a more cautious framing of 'minimum' as 'best found'. The reliance on the authors' own concurrent work for thresholds and similarity measures should also be made explicit in the contribution statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Frankly, this paper has a real idea buried under a too-thin experiment. The contribution is the use of a genetic algorithm (borrowed from Alzantot et al.) to search for minimum perturbations that break LIME text explanations, instead of the greedy search used in prior work. That is a legitimate extension of the authors' own framework, and the qualitative examples in Figures 1–3 do show cases where the GA finds more natural replacements. The comparison against the greedy baseline is the right way to frame the question. So the direction is plausible, and the paper is honest about its computational limits (population 10, 10 generations, 20 examples per dataset).\n\nBut the evidence does not support the abstract's claim of 'superior comparisons.' A single run of a small stochastic GA on 20 examples per dataset, with no seeds, no error bars, and no statistical tests, is not a reliable estimate of minimum perturbation counts. The minimums in Table 5 are best-of-20 values, not lower bounds. Several table entries are malformed ('0.3 5', '0 .82', '0 .65'), which makes verification harder. The paper also omits the target classifier, LIME configuration, and replacement embedding, so the experiment cannot be reproduced from the text alone. These are load-bearing omissions, not cosmetic ones.\n\nThe circularity concern the reader raised does not actually land: the success thresholds and similarity measures come from the authors' own prior work, but the GA comparison is a measurement, not a derivation that assumes its conclusion. So the core idea is not circular. The citation pattern is likewise reasonable—self-citations point to the framework being extended.\n\nWho is this for? Researchers working on adversarial XAI for text, especially those comparing stability across explanation methods. They will read this as a proof-of-concept for a minimum-perturbation benchmark, not as a definitive result.\n\nMy recommendation: send it to peer review, but with the expectation of a major revision. Ask for repeated seeds, variance reporting, the full experimental setup, and code/data release. If the authors supply those, the paper could become a solid contribution. As it stands, the claim outruns the evidence.","headline":"A plausible GA-based approach to estimating minimum perturbations for LIME stability, undermined by a single-run 20-example experiment with no variance reporting.","tokens_in":11811,"tokens_out":2046,"would_cite":false,"duration_ms":20265,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the greedy search with a genetic algorithm finds LIME text explanations that break under fewer word replacements, suggesting previous stability estimates were optimistic.","keywords":["Explainability","Interpretability","XAI","Stability","Robustness","Genetic algorithm","LIME","Adversarial attack"],"falsifier":"Run the same genetic-algorithm search on the same 20 examples many times with different random seeds and compare the distribution of minimum perturbations to the greedy search's minima; if the genetic minima frequently equal or exceed the greedy minima, or if single-run minima swing widely between seeds, the claim that this search improves LIME stability estimates is not supported.","tokens_in":10771,"feed_emoji":"🧬","tokens_out":8301,"duration_ms":80474,"temperature":0.7,"pith_summary":"This paper asks how unstable LIME's text explanations really are, measured by the fewest word replacements needed to push the explanation past a fixed similarity threshold. Prior attacks only showed that instability exists; the authors argue that the minimum number of perturbations is the quantity that should be compared across explainers. They replace the usual greedy search with a genetic algorithm, keeping the constraints that the model's prediction and the document's meaning are preserved. The paper reports that the genetic search matches or beats the greedy search in attack success and perturbed-document quality while often finding successful attacks at lower minimum perturbation counts. If correct, LIME has been shown to be less stable than earlier estimates suggested.","feed_headline":"Attack search shows LIME breaks with fewer word changes than thought","feed_subtitle":"A genetic-algorithm search lowers the minimum edits needed to break text explanations, sharpening stability estimates.","key_machinery":"The load-bearing object is the minimum-viable-perturbation search: minimize explanation similarity $S_{\\mathrm{ime}}(e_{d_b}, e_{d_p})$ subject to $f(d_b)=f(d_p)$, semantic similarity above a threshold, a cap on the total fraction of words changed, and preservation of the top-$k$ features of the original explanation. The search mechanism is a genetic algorithm with a population of 10 chromosomes evolved over 10 generations, using mutation that replaces a word at a valid index with a nearby embedding-space alternative and crossover that swaps document tails; fitness is the objective function itself. Success is judged by nine similarity measures (rank-biased overlap, a weighted overlap favoring top ranks, at three persistence weights; Jaccard index; Kendall's tau rank distance; Spearman's footrule; and weighted variants of the non-RBO measures) at thresholds of 30%, 40%, 50%, and 60% explanation similarity. The genetic algorithm's wider exploration of the replacement space is what lets the paper estimate lower bounds on perturbations rather than merely demonstrating that some perturbation exists.","core_discovery":"The paper's central claim is that a genetic-algorithm search can locate minimum viable perturbations—the fewest word replacements that drive explanation similarity below a chosen threshold $\\tau$—more effectively than the greedy search used in prior work. On the Gender Bias and Symptoms-to-Diagnosis datasets, the genetic algorithm produced successful attacks where the greedy search found none (for example, the weighted Spearman measure at the 30% threshold on Gender Bias rose from 0% to 24% success, and weighted Jaccard moved from 0% to 6% at 50% and 29% at 60%), and it lowered the minimum perturbation count for several similarity measures while keeping average perturbation rates and ending similarities close to the greedy baseline. The paper therefore claims that stability estimates for LIME based on greedy search are incomplete: the true minimum perturbation needed to reach a given instability threshold is often smaller than previously reported, and that gap matters for judging which explanation methods are trustworthy.","pith_inferences":["If these minima reproduce across repeated runs, LIME's practical stability is lower than the literature suggests, and any deployment relying on LIME explanations should treat a single-word adversarial edit as a realistic threat.","The same setup could become a comparative benchmark that pits LIME against other ranked-list explainers, using minimum perturbation counts at matched thresholds as the ranking criterion.","Because the GA was limited to 10 chromosomes, 10 generations, and 20 examples per dataset, the reported improvements are point estimates; a random-seed variance study would show whether the method's advantage is robust or partly sampling noise.","The emphasis on inconspicuous replacements suggests a natural extension: optimize perturbed-document quality as a second objective alongside perturbation count, rather than only checking semantic similarity after the attack."],"forward_implications":["Stability of explanation methods should be reported as a minimum-perturbation curve across similarity thresholds, not as a binary found/not-found result.","Previous greedy-search-based estimates of LIME's stability are upper bounds on the true minimum, and the true values can be smaller.","The same threshold-and-minimum framework transfers to any explainer that returns a ranked feature list, giving a common yardstick for comparing stability across XAI methods.","Because a deliberately small genetic algorithm already lowers the minima, further increases in search effort or better mutation strategies are likely to find even smaller perturbation counts.","Perturbation quality matters jointly with count: the genetic search sometimes preserves meaning better at the same perturbation rate, though the greedy search also wins on some examples."],"supporting_citations":[{"why":"Supplies the definition of stability used to frame the search objective.","marker":"Alvarez-Melis & Jaakkola, 2018"},{"why":"Introduces LIME, the explanation method whose explanations are attacked.","marker":"Ribeiro et al., 2016"},{"why":"Provides the greedy search baseline, datasets, and prior evidence of LIME instability in text.","marker":"Burger et al., 2023"},{"why":"Defines the similarity measures and the 30/40/50/60% success thresholds used in the experiments.","marker":"Burger et al., 2024"},{"why":"Motivates the genetic-algorithm approach for generating natural-language perturbations.","marker":"Alzantot et al., 2018"},{"why":"Supplies TextFooler, the embedding-based word replacement strategy used for candidate perturbations.","marker":"Jin et al., 2020"},{"why":"Demonstrates that post hoc explanation methods can be fooled, establishing the existence of the instability being measured.","marker":"Slack et al., 2020"}],"fun_headline_variants":["Genetic search finds smaller attacks on LIME than greedy method","LIME's true instability exposed by genetic-algorithm attack search","Fewer word changes needed to fool LIME than previous estimates","Genetic algorithm sharpens LIME stability tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the minimum perturbation counts reported are reliable estimates of LIME's instability, even though they come from one run of a small genetic algorithm (population size 10, 10 generations) on only 20 examples per dataset, with no variance or significance testing.","fun_headline_variants_meta":{"raw":{"variants":["Genetic search finds smaller attacks on LIME than greedy method","LIME's true instability exposed by genetic-algorithm attack search","Fewer word changes needed to fool LIME than previous estimates","Genetic algorithm sharpens LIME stability tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1285,"prompt_tokens":874,"completion_tokens":411,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":490,"tokens_out":411,"duration_ms":4371,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:11:28.552883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same genetic-algorithm search on the same 20 examples many times with different random seeds and compare the distribution of minimum perturbations to the greedy search's minima; if the genetic minima frequently equal or exceed the greedy minima, or if single-run minima swing widely between seeds, the claim that this search improves LIME stability estimates is not supported.","supporting_citations":[{"cited_title":"The Effect of Similarity Measures on Accurate Stability Estimates for Local Surrogate Models in Text-based Explainable AI","cited_arxiv_id":"2406.15839","evidence_quote":"Supplies the definition of stability used to frame the search objective."}],"review_version":1}