{"id":"fbd7803f-21e3-4b5c-a4e7-d71ebf9fb68d","arxiv_id":"2411.16251","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"XPROB builds explanation neighborhoods for text classifiers by explicitly editing counterfactual texts with local word-context probabilities, matching generator-based explainers in quality while running faster and staying auditable.","lead":"This paper introduces XPROB, a way to explain text-classifier decisions by editing nearby texts word by word using simple word-context statistics, instead of using a neural text generator. It aims to make the explanation process transparent while keeping explanations as accurate as those from generator-based methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2's product of one-sided n-gram conditionals is an unvalidated proxy for sequence likelihood; if it selects unnatural edits, the reported competitive performance may reflect spliced sentiment words rather than faithful local neighborhoods.","rationale":"The paper's central empirical claim is that probability-based editing produces neighborhoods competitive with generator-based explainers. The reader identified the n-gram probability estimates and the product of one-sided conditionals as the weakest assumption; I agree that this is the most load-bearing point, but I would emphasize the mathematical invalidity of Eq. 2 as a likelihood proxy rather than corpus sparsity. The paper treats Ppre(w | left) * Psuc(w | right) as if it were proportional to the probability of inserting w between those contexts, which is only true under a strong conditional-independence assumption that is neither stated nor tested. Since all experiments use n=1, the method relies on bigram statistics that cannot capture joint fit, and the qualitative examples provide direct evidence that the resulting texts are often ungrammatical. This matters because the quantitative metrics in Table 3 measure confidence drops after manipulating attributed words; if the neighborhoods are simply ungrammatical strings with obvious sentiment words spliced in, high completeness and compactness scores may not indicate faithful local explanations. I do not think this warrants rejection, because the method is clearly specified, reproducible, and the empirical comparison is plausible; however, the paper should either validate the factorization or soften the claims about realistic and grammatical neighborhoods. The stability comparison is not the primary issue: although ABELE has a lower raw adjective deviation in Table 6, that comparison is confounded by scale, and a coefficient-of-variation view favors XPROB, so I would not hinge the verdict on that point. The concrete test with perplexity or a full-sequence likelihood comparison would settle the factorization concern directly.","tokens_in":15490,"tokens_out":6596,"duration_ms":83836,"concrete_test":"Generate 200 matched-length neighbors from XPROB, XPROAX, and LIME on a random sample of the Amazon and Yelp test sets, and compute GPT-2 (or a held-out 5-gram) perplexity for each set. If XPROB's mean perplexity is substantially higher than XPROAX's (e.g., more than 2x), the Eq. 2 neighborhoods are off-manifold and the Table 3 results should be reinterpreted as artifacts of the edit objective. Alternatively, for 100 word-prototype pairs, compare the argmax edit chosen by Eq. 2 with the argmax edit under a strong language model's estimate of the full joint probability; if the rankings disagree on more than 20% of pairs, the factorization is the cause of the unnatural edits.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that XPROB is a competitive, transparent replacement for generator-based explainers depends on the edit objective in Eq. 2: the argmax over insertion/replacement positions of Ppre(w | left context) * Psuc(w | right context) / e^(j-i). This is not a valid estimate of the full-sequence probability in Eq. 1 unless the left and right contexts are conditionally independent given w and the rest of the sequence is ignored. With n=1, which is used in all experiments (Section 4.1), Ppre and Psuc are single-word bigram estimates; the objective cannot distinguish whether w fits jointly between the two contexts, only whether w is frequent after the left word and before the right word. The paper's own qualitative examples show ungrammatical outputs, e.g., 'excellent a beginning french students' and 'the were very excellent desserts', suggesting the factorization does fail in practice. If XPROB's neighborhoods are off-manifold, the completeness and compactness scores in Table 3 may be inflated by inserting obvious sentiment words into counterfactuals, rather than reflecting the classifier's local decision boundary. The claim that probability-based editing 'ensures grammatical correctness' (Section 3.1) is therefore unsupported, and the 'competitive performance' conclusion rests on an unvalidated independence assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes XPROB, a model-agnostic explanation method for text classifiers that constructs local neighborhoods by recursive probability-based editing. Instead of using a neural generator to synthesize neighboring texts, XPROB integrates tokens from the explicand into real counterfactual prototypes under a local n-gram objective (Eq. 2), with prototypes selected by tf-idf closeness from an external corpus Xp. The method is evaluated on Yelp and Amazon review datasets with BERT and LSTM classifiers, comparing against LIME, XSPELLS, ABELE, and XPROAX. Reported results show competitive fidelity, completeness, and compactness, lower time cost, and a stability experiment suggesting low attribution variance. The paper argues that the deterministic, rule-based editing process is more transparent and controllable than generator-based neighborhood construction.","tokens_in":15726,"tokens_out":5835,"duration_ms":52585,"significance":"If the empirical claims hold, the contribution is valuable: XPROB offers a simple, deterministic, and considerably faster alternative to generator-based neighborhood construction while producing explanations of comparable quality to the stronger baseline XPROAX. The method is reproducible in principle, and the authors provide source code. The evaluation is broad in scope—two real-world datasets, two classifiers, four baselines, and several complementary metrics—and the dependency analysis (Section 4.5) is a thoughtful addition. The main advertised benefits are transparency and stability, and these are only partially supported by the evidence. The paper does not contain machine-checked proofs or parameter-free derivations, but the algorithmic construction is explicit and auditable. The central 'competitive performance' claim is defensible from the reported tables, whereas the stronger 'superior stability' and 'grammatical correctness' claims need revision.","major_comments":[{"comment":"The objective in Eq. (2) is not a valid estimate of the sequence probability defined in Eq. (1). With n=1, which is used in all experiments (Section 4.1), Ppre(w|left) is a forward bigram conditional and Psuc(w|right) is a backward bigram conditional; their product does not equal the joint probability of the manipulated sequence, even under a Markov assumption. Consequently, the claim that 'Implementing manipulation as outlined in Eq. 1 ensures grammatical correctness' is unsupported. The qualitative examples in Table 2 actually show ungrammatical outputs, e.g., 'excellent a beginning french students' and 'the were very excellent desserts', which suggest the factorization can select off-manifold edits. The paper should either temper the grammaticality claim, justify the product as a heuristic with empirical validation, or add a human/automatic grammaticality evaluation of the generated neighborhoods.","section":"Section 3.1, Eq. (2)"},{"comment":"The 'superior stability' claim is not fully supported by the reported data. In Table 6, ABELE has a lower averaged adjective deviation (0.062) than XPROB (0.082), and the paper's explanation that ABELE's low deviation comes from the smaller scale of its attribution scores is post hoc; no scale-normalized comparison is provided. XPROB does show the lowest noun deviation (0.019) and competitive adjective deviation, but no statistical significance tests accompany the comparisons. The conclusion should be revised to a more modest statement about low attribution deviation, and the analysis should include a scale-adjusted stability measure or a significance test.","section":"Section 4.4, Table 6"},{"comment":"The dependency comparison between XPROAX and XPROB is not controlled. XPROAX is trained on a 200k-entry corpus while XPROB uses 5k–80k entries, and the analysis is limited to one dataset/model pair (Amazon LSTM) and two metrics. The statement that XPROB exhibits 'direct but less sensitive dependency' on the volume of the corpus is not directly supported, because no comparison shows how XPROAX's performance degrades with corpus size under the same evaluation. To substantiate the 'more controllable' and 'less sensitive' claims, the authors should either measure XPROAX with varying generator training sizes or explicitly reframe the results as demonstrating resource efficiency rather than differential sensitivity.","section":"Section 4.5, Tables 7–9"},{"comment":"The paper fixes several hyperparameters—n=1, epsilon, the edit-length penalty base e, neighborhood population p=400, prototype count k=80, locality kernel width sigma, and diversity weight lambda—and notes that hyperparameter impact studies are omitted due to space limitations. Since the method's selling point is a transparent and controllable construction process, the absence of any sensitivity analysis for these choices leaves the robustness of the reported results unexamined. In particular, n controls the context size and is set to the smallest possible value; a brief study varying n would strengthen the practical claims.","section":"Section 4.1, hyperparameters"}],"minor_comments":[{"comment":"The acronym XPROB is used in the abstract but defined only in the introduction; consider defining it at first mention.","section":"Abstract"},{"comment":"In the second input row, the classifier is labeled 'b(·)' but the text elsewhere uses f(·); this is likely a typo and should be corrected.","section":"Table 2"},{"comment":"The phrase 'Albeit not explicitly given' is awkward; suggest rewording to 'Although not written explicitly' or similar.","section":"Section 3.1"},{"comment":"The source code URL contains a duplicated scheme: 'https://https://github.com/caiy0220/XPROB' should be 'https://github.com/caiy0220/XPROB'.","section":"Footnote 1"},{"comment":"Reference [43] appears corrupted: the proceedings name includes 'frye2020shapley' and the entry should be repaired.","section":"References"},{"comment":"The set update on line 8, 'S = Snew − Snew ∩ N (x)', is not standard notation; please use set difference and intersection symbols explicitly, e.g., S ← Snew \\ (Snew ∩ N(x)).","section":"Algorithm 1"},{"comment":"The claim that XPROB's near-zero noun attributions 'accurately reflect their non-informative nature' relies on the selection criterion (confidence near 0.5 for single-word inputs) as a proxy; this connection should be acknowledged more explicitly.","section":"Section 4.4"},{"comment":"The generator training hyperparameters (epochs, learning rate, etc.) are not reported; the paper states they follow the original paper, but including them would improve reproducibility for the baseline comparisons.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's fit to the journal is good and the self-citation pattern is appropriate (only the XPROAX baseline is from the same group). The main issues are overclaiming on grammatical correctness and stability, and the lack of controlled sensitivity/dependency analyses. These are fixable with reanalysis and rewriting, so I do not see a need for rejection, but the current version overstates support for some of its central claims. The source code availability is a strength, but the URL typo should be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"XPROB is a genuinely transparent alternative to generator-based text explainers: it builds neighborhoods by explicit n-gram edits of counterfactual prototypes, with a deterministic objective and a linear surrogate. The central empirical claim—competitive completeness and compactness at roughly a 10x speedup over XPROAX—holds up from the tables. I also credit the authors for shipping code, testing two model architectures, and including a dependency analysis that actually demonstrates the generator's brittleness.\n\nThe soft spots are real but not fatal. Equation 2 multiplies a left bigram and a right bigram probability; with n=1 this is a crude proxy for whether a word fits jointly in context. The paper's own examples betray the claim: \"excellent a beginning french students\" and \"the were very excellent desserts\" are not grammatical, so the assertion that editing \"ensures grammatical correctness\" is unsupported. That matters because it weakens the \"fully transparent\" narrative, but it does not sink the main result—completeness and compactness are competitive, and those metrics do not hinge on grammaticality. A more careful paper would use a true sequence probability estimator or restrict the claim to context-plausibility.\n\nThe stability claim is also overstated. Table 6 shows ABELE with lower adjective deviation (0.062 vs 0.082), and the explanation that ABELE's attributions are smaller in scale is plausible but untested. Noun deviations are nicely low, but without significance testing or reported hyperparameters (sigma, lambda, edit-penalty base), I would call the stability edge suggestive rather than demonstrated.\n\nThe main missing piece is an analysis of Eq. 2's failure modes—when the corpus lacks relevant n-grams, how often do edits become infeasible or unnatural? The paper acknowledges prototype dependence but does not quantify the off-manifold risk. That is the natural referee request.\n\nOverall: a solid, useful contribution to text-XAI. It deserves a serious referee, and I would accept it with major revision requests for the above points. I would bring it to a reading group only if someone is actively working on neighborhood construction.","headline":"A transparent, fast n-gram editing alternative to generator-based text explainers, with solid competitive results, an overclaimed stability edge, and a valid but non-fatal concern about the edit objective's grammar claims.","tokens_in":16306,"tokens_out":3537,"would_cite":true,"duration_ms":139580,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"XPROB constructs text-explanation neighborhoods by recursive, transparent probability-based editing rather than black-box generators, and reports competitive quality with superior stability.","keywords":["explainable AI","text classification explanation","neighborhood construction","model-agnostic explanations","probability-based editing","n-gram contexts","generator-based explainers","feature attribution"],"falsifier":"Run XPROB with a deliberately small or domain-mismatched corpus $X_p$ (say, 2,000 texts of unrelated genre) on explicands full of rare words, and compare surrogate fidelity against XPROAX on the same inputs; if the fidelity gap widens in proportion to the explicand's missing n-gram coverage, the claim that probability-based editing preserves neighborhood quality at small corpus sizes would fail.","tokens_in":15246,"feed_emoji":"🔍","tokens_out":6645,"duration_ms":65843,"temperature":0.7,"pith_summary":"The paper claims that transparent, deterministic text editing can replace black-box neural generators in building the synthetic neighborhoods that model-agnostic text explainers rely on. Its proposed method, XPROB, inserts or replaces words from the text being explained into counterfactual prototypes at positions chosen by local n-gram probabilities estimated from a plain-text corpus, then recursively applies the same edit rule. The authors argue that this substitution is not a sacrifice: XPROB matches the explanation quality of the best generator-based competitor on correctness, completeness, and compactness across two datasets and two classifier architectures, while being dozens to hundreds of times faster and more stable across similar inputs. Because every generated neighbor is the result of an explicit, auditable edit, the explanation pipeline itself stays transparent.","feed_headline":"N-gram editing explains text classifiers without neural generators","feed_subtitle":"XPROB matches the best generator-based explainers and is more stable, at a fraction of the compute cost.","key_machinery":"The central object is the probability-based edit operation $O(w, \\hat{x})$ of Eq. 2, which selects an insertion or replacement position by maximizing the product of a left-context conditional probability $P_{\\text{pre}}$ and a right-context conditional probability $P_{\\text{suc}}$, estimated by n-gram counts in the retained corpus $X_p$, divided by an edit-length penalty $e^{j-i}$ to favor small changes. It is what carries the argument because it makes every generated neighbor a deterministic, auditable function of explicit corpus statistics, eliminating the black-box generator while keeping neighbors close to the natural text manifold. Around it sits the recursive editing scheme of Algorithm 1, which iterates edited texts as new prototypes until the neighborhood is filled, and the tf-idf prototype selection that enforces locality.","core_discovery":"The paper's central claim is that substituting the generator-based construction process with recursive probability-based editing yields an explainer, XPROB, whose explanation quality is competitive with neural-generator explainers and whose neighborhood construction is fully transparent and more controllable. The decisive mechanism is the edit objective: for each word $w$ of the explicand and each prototype counterfactual $\\hat{x}$, XPROB chooses the operation that maximizes $P_{\\text{pre}}(w|\\hat{x}^i_{i-n+1})\\,P_{\\text{suc}}(w|\\hat{x}^{j+n-1}_{j})\\,/\\,e^{j-i}$, where the probabilities are corpus-count estimates of the word's compatibility with the local left and right n-gram contexts, and the denominator penalizes long deletions. Recursively applying this rule transitions from counterfactual to explicand and yields realistic neighbors without training or running a generator. The paper reports that on Yelp and Amazon review data, for both BERT and LSTM classifiers, XPROB matches XPROAX, the strongest generator-based competitor, on completeness and compactness, and that its deterministic construction produces lower attribution variance across similar contexts, which the authors credit to its controllability.","pith_inferences":["A testable extension the paper does not pursue: interpolated n-gram smoothing in place of raw corpus counts could reduce the corpus size XPROB needs, since Eq. 2 currently gives any unseen context the same minimal epsilon.","The transparency claim implies a direct human audit: users could be shown pairs of XPROB edits and generator outputs and asked which operation is easier to verify, a comparison the paper does not run.","Because the edit rule operates on any discrete sequence with local contexts, XPROB's scheme should transfer to code, log lines, or tabular text fields; the paper only demonstrates it on short review texts.","The paper's binary-class assumption is not a hard limit: redefining counterfactuals as low-confidence instances, as the conclusion notes, would let the same machinery serve multi-class settings, but that adaptation is left untested."],"forward_implications":["Explanation neighborhoods can be constructed without training or running a generator, cutting per-entry time cost to about one second versus five to several hundred seconds for the generator-based competitors.","Every synthetic neighbor is traceable to an explicit edit operation and to corpus counts, making the construction process auditable in a way that latent-space sampling is not.","XPROB's deterministic edits yield more stable attributions across similar contexts: on the stability test, its averaged attribution deviations for adjectives and nouns are the lowest among the compared explainers.","Completeness and compactness match XPROAX, the best generator-based competitor, and clearly exceed LIME, showing that transparent editing does not sacrifice explanation quality.","XPROB's explanation quality depends on its external corpus in a direct and predictable way, and it remains competitive with a 5,000-text corpus even when the generator baseline needs 200,000 texts.","The recurring editing rule transfers to any discrete sequence with local contexts, so the same machinery could build neighborhoods for code, log lines, or multi-class text settings, with counterfactuals redefined as low-confidence instances."],"supporting_citations":[{"why":"Provides the input-perturbation baseline (LIME) that XPROB must beat and that motivates the need for manifold-respecting neighborhoods.","marker":"[5]"},{"why":"Introduces XPROAX, the progressive generator-based neighborhood approximation that is both the main competitor and the inspiration for recursive editing.","marker":"[9]"},{"why":"Defines XSPELLS, a generator-based text explainer using latent decision trees, used as a competitor in the evaluation.","marker":"[17]"},{"why":"Introduces ABELE, a generator-based explainer originally for images that the paper adapts to text as a competitor.","marker":"[19]"},{"why":"Supplies the DAAE generator used by all generator-based competitors, so the comparison rests on this autoencoder's locality-preserving latent space.","marker":"[36]"},{"why":"Supplies the VAE generator used in the dependency analysis to show that generator choice and capacity change explanation quality.","marker":"[18]"},{"why":"Provides the n-gram language-modeling background that the probability-based editing objective is built on.","marker":"[26]"},{"why":"Supplies the tf-idf representation used for prototype selection and distance weighting in the locality constraint.","marker":"[27]"},{"why":"Provides the text-augmentation approach used to construct the stability test cases.","marker":"[42]"}],"fun_headline_variants":["No generators, just probability edits: transparent text AI explanations","XPROB matches neural explainers with fully transparent recursive edits","Transparent probability edits replace black-box generators for text explainability","Probability-based editing: transparent text explainer without neural nets","Recursive probability edits make text classifier explanations transparent and stable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that n-gram counts from a 20,000-text corpus faithfully represent natural text contexts, so the edit objective's product of left and right conditional probabilities really picks the most natural place for a word; where those contexts are missing, edits become infeasible or unnatural and the neighborhood no longer reflects the classifier's local behavior.","fun_headline_variants_meta":{"raw":{"variants":["No generators, just probability edits: transparent text AI explanations","XPROB matches neural explainers with fully transparent recursive edits","Transparent probability edits replace black-box generators for text explainability","Probability-based editing: transparent text explainer without neural nets","Recursive probability edits make text classifier explanations transparent and stable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00049,"raw_usage":{"total_tokens":2415,"prompt_tokens":952,"completion_tokens":1463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":1380}},"tokens_in":568,"tokens_out":1463,"duration_ms":10784,"temperature":1.0,"reasoning_tokens":1380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:22:00.450303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run XPROB with a deliberately small or domain-mismatched corpus $X_p$ (say, 2,000 texts of unrelated genre) on explicands full of rare words, and compare surrogate fidelity against XPROAX on the same inputs; if the fidelity gap widens in proportion to the explicand's missing n-gram coverage, the claim that probability-based editing preserves neighborhood quality at small corpus sizes would fail.","supporting_citations":[{"cited_title":"“Why should I trust you?","cited_arxiv_id":null,"evidence_quote":"Provides the input-perturbation baseline (LIME) that XPROB must beat and that motivates the need for manifold-respecting neighborhoods."},{"cited_title":"XPROAX-local explanations for text classification with progressive neigh- borhood approximation,","cited_arxiv_id":null,"evidence_quote":"Introduces XPROAX, the progressive generator-based neighborhood approximation that is both the main competitor and the inspiration for recursive editing."},{"cited_title":"Explaining sentiment classification with synthetic exemplars and counter-exemplars,","cited_arxiv_id":null,"evidence_quote":"Defines XSPELLS, a generator-based text explainer using latent decision trees, used as a competitor in the evaluation."},{"cited_title":"Black box explanation by learning image exemplars in the latent feature space,","cited_arxiv_id":null,"evidence_quote":"Introduces ABELE, a generator-based explainer originally for images that the paper adapts to text as a competitor."},{"cited_title":"Educating text autoencoders: Latent representation guidance via denoising,","cited_arxiv_id":null,"evidence_quote":"Supplies the DAAE generator used by all generator-based competitors, so the comparison rests on this autoencoder's locality-preserving latent space."},{"cited_title":"Generating sentences from a con- tinuous space,","cited_arxiv_id":null,"evidence_quote":"Supplies the VAE generator used in the dependency analysis to show that generator choice and capacity change explanation quality."},{"cited_title":"Jurafsky, Speech & language processing","cited_arxiv_id":null,"evidence_quote":"Provides the n-gram language-modeling background that the probability-based editing objective is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the tf-idf representation used for prototype selection and distance weighting in the locality constraint."},{"cited_title":"Gender bias in coreference resolution,","cited_arxiv_id":null,"evidence_quote":"Provides the text-augmentation approach used to construct the stability test cases."}],"review_version":1}