{"id":"279f3285-2296-41e2-ba4a-ebd1e7730d08","arxiv_id":"2501.00777","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ZeroCF and FitCF generate label-flipping text counterfactuals from BERT feature attributions, and FitCF outperforms Polyjuice, BAE, and FIZLE on AG News and SST2.","lead":"This paper builds a two-step pipeline for automatic counterfactual text generation: find the words a BERT classifier relies on, then prompt an LLM to edit those words, verify the edit flips the label, and use verified edits as few-shot examples. It reports better label-flip rates than three existing methods on news and sentiment classification, though the main evaluation metric is computed by the same LLM that wrote the counterfactuals.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SLFR is computed by the same LLM that generates the counterfactuals (Appendix G), so the headline flip-rate gains may reflect self-consistency rather than genuine label flips; this directly threatens the claim that FitCF outperforms FIZLE and other baselines.","rationale":"The reader's weakest assumption is exactly the concern I find most load-bearing: the Soft Label Flip Rate is computed by the same LLM that generates the counterfactuals. This threatens the paper's primary empirical claim that FitCF consistently outperforms FIZLE, POLYJUICE, and BAE. I agree with the reader's assessment that this is a serious validity issue, and I also note the paper's own justification in Appendix G.1 is based on a weak aggregate-accuracy analogy rather than a direct test. In addition, the perfect 100% flip rates on SST2 with Llama3-8B are a red flag that the metric may be trivially satisfiable by the generator. I considered whether the faithfulness-quality correlation (Section 6.4) based on four data points is an even weaker link, but the correlation is a secondary claim; the headline contribution is the outperformance result, which is directly undercut by the metric-bias concern. Therefore the reader's CONDITIONAL verdict is appropriate, and no change to the verdict is needed, provided the condition is understood as requiring an independent evaluation. The proposed test -- recomputing SLFR with the BERT models already used in the pipeline -- is a feasible and concrete way to determine whether the concern actually lands.","tokens_in":17362,"tokens_out":7366,"duration_ms":67999,"concrete_test":"Recompute SLFR for all rows of Table 1 using the fine-tuned BERT classifiers from Table 6 (textattack/bert-base-uncased-ag-news and gchhablani/bert-base-cased-finetuned-sst2) as the label-flip judge, exactly as used in the demonstration-verification step of Section 3.2. If FITCF's margin over FIZLE and ZeroCF shrinks, reverses, or loses statistical significance (report 95% confidence intervals over at least 3 runs), then the headline performance claim is an artifact of the shared generator-evaluator.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 defines SLFR and notes 'we use the same LLM for both counterfactual generation and classification.' Appendix G confirms that the same LLM serves as both flip label verifier and generator. The paper's only justification (Appendix G.1) is that the LLM and BERT have similar classification accuracy (Table 6 vs Table 10), so 'the tendency of SLFR will be consistent.' This is not a valid inference: similar aggregate accuracy does not imply similar per-instance decisions, especially for adversarially edited texts. Moreover, the evaluation prompt (Appendix G.2) asks the generation model itself whether the original and edited instances have different predicted classifications; it is well documented that LLMs are subject to self-preference and can rate their own outputs leniently. The risk is concrete: on SST2, FITCF with Llama3-8B attains 100% SLFR for IG, LIME, and SHAP, even though this same model's zero-shot classification accuracy on SST2 is 89.75%. A perfect flip rate in a binary task is implausible as a genuine property of the generated text and suggests the model is exploiting its own decision boundary and then confirming the flip. Because the framework's demonstration-construction step (Section 3.2) uses BERT as the label-flip verifier, while the final evaluation uses the LLM, the paper never reports whether these counterfactuals flip the target model's predictions. Thus the central quantitative claim that FitCF 'consistently outperforms' FIZLE and other baselines is not established unless the evaluation is repeated with an independent classifier.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ZeroCF, a zero-shot counterfactual generation method that feeds top words from BERT-based feature attribution (Gradient, IG, LIME, SHAP) into an LLM prompt, and FitCF, a few-shot extension that automatically constructs demonstrations by running ZeroCF on cluster-selected examples, filtering them through BERT label-flip verification, and then using these verified pairs as in-context demonstrations. The authors evaluate on AG News and SST2 with three LLMs (Llama3-8B, Qwen2.5-32B, Qwen2.5-72B) against baselines POLYJUICE, BAE, and FIZLE, using SLFR, perplexity, and text similarity. They also ablate the three FitCF components (important words, number of demonstrations, label-flip verification) and report a correlation between faithfulness of attribution scores and counterfactual quality.","tokens_in":17700,"tokens_out":3135,"duration_ms":28533,"significance":"If the empirical claims hold, the paper makes two useful contributions: it provides a practical pipeline that replaces hallucination-prone LLM-generated important words with BERT-derived attribution words, and it shows that automatically verified demonstrations can improve few-shot counterfactual generation. The study is systematic: it covers multiple attribution methods, multiple LLMs, two datasets, and includes ablation studies. The code is publicly available, and the paper is transparent about many experimental details. However, the core quantitative claim—that FitCF 'consistently outperforms' the baselines—rests almost entirely on SLFR as computed by the same LLM that generated the counterfactuals. The manuscript's own appendix acknowledges this design, and the justification provided (similar aggregate accuracy between LLM and BERT) is not sufficient to establish the validity of the metric. The correlation analysis in Section 6.4 also relies on only four attribution methods, making the strong correlation claim fragile. These issues are central to the paper's conclusions and require substantive revision.","major_comments":[{"comment":"The primary evaluation metric, Soft Label Flip Rate, is computed by the same LLM that generated the counterfactuals: Section 5.1 states 'we use the same LLM for both counterfactual generation and classification,' and Appendix G confirms this. The only justification given, in Appendix G.1, is that the LLM and BERT have similar classification accuracy on the two datasets, so 'the tendency of SLFR will be consistent.' This is not a valid inference: similar aggregate accuracy does not imply per-instance agreement, and per-instance divergence is especially likely for adversarially edited texts near the decision boundary. The risk becomes concrete in Table 1: with Llama3-8B on SST2, FitCF attains 100% SLFR for IG, LIME, and SHAP, even though the same model's zero-shot accuracy on SST2 is only 89.75%. A perfect flip rate in a binary task is implausible as a measure of genuine label change and instead suggests that the generator is exploiting its own decision boundary and the evaluator is confirming self-consistency. Since the headline claim that FitCF 'consistently outperforms' FIZLE and other baselines is based on SLFR, this is a load-bearing issue. The authors should report flip rates using the BERT classifier (the target model used for feature attribution and verification) or another held-out classifier, and discuss any discrepancies.","section":"§5.1"},{"comment":"No error bars, confidence intervals, or significance tests accompany the automatic evaluation results. The SLFR values are reported in increments of 0.5%, suggesting a test set of about 200 instances per dataset; under that assumption, differences of a few percentage points (e.g., 94.5% vs. 95.5% for AG News with Llama3-8B) are well within sampling noise. The claim that FitCF 'consistently outperforms' all baselines across all settings needs a quantitative basis; otherwise, the observed differences may not be reliable. The authors should provide at least bootstrap confidence intervals or multiple runs with different seeds, and preferably a paired significance test for the main FitCF-vs-FIZLE and FitCF-vs-ZeroCF comparisons.","section":"Table 1"},{"comment":"The correlation analysis between faithfulness of attribution scores and counterfactual quality uses only four attribution methods (Gradient, IG, LIME, SHAP) per dataset and model. With n=4, a Kendall's tau of 1.0—reported for AG News—can arise by chance with non-negligible probability, and the manuscript gives no significance value or confidence interval for these correlations. Figure 3 appears to show only the tau values, not the underlying scatterplots, so the reader cannot assess whether the correlation is driven by a single outlier or is stable across the data. The authors should either include more attribution methods, bootstrap over instances to obtain confidence intervals, or at minimum explicitly state that the finding is based on four points and should be interpreted as preliminary.","section":"§6.4"}],"minor_comments":[{"comment":"The relationship between the number of clusters k and the number of demonstrations ℓ is not defined in the method section; Section 5.2.2 states that ℓ = 2k, but this choice should be introduced in §3.2 where the demonstration sampling is described.","section":"§3.2"},{"comment":"The caption says 'Bold faced values indicate for each approach, which feature importance method is the best performing'—but the table also includes baselines (POLYJUICE, BAE) that have no feature importance method. The caption should clarify how bolding applies to those rows.","section":"Table 1"},{"comment":"The arrow notation in the ablation tables is confusing: for example, '↑0.63' for TS in Table 2 appears to indicate an increase, but since lower TS is better, an upward arrow is not clearly an improvement. The convention is stated in §6.2, but it is easy to misread; consider using explicit 'better/worse' labels or separate improvement/deterioration columns.","section":"§6.2"},{"comment":"The sentence 'Table 9 shows inference time for ZEROCF and FITCF using Llama3-8B, Qwen2.5-32B and Qwen2.5-32B across AG News and SST2 datasets' repeats 'Qwen2.5-32B'; the second occurrence should be 'Qwen2.5-72B'.","section":"Appendix F.2"},{"comment":"The sentence 'as selected LLMs and BERT perform similarly on the two datasets, we can assume that the tendency of SLFR will be consistent' is a non-sequitur; the authors should explicitly acknowledge that this is an assumption, not a derivation, and discuss its limitations.","section":"Appendix G.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-structured and the code release is a positive feature, but the central evaluation metric is compromised by using the generator as the evaluator. The authors should be encouraged to rerun or complement the evaluation with BERT-based flip rates and add uncertainty quantification. The correlation analysis in Section 6.4 should also be tempered unless more data points are added. I would not recommend rejection, as the framework itself is plausible and the issues appear addressable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper builds a useful, clearly described pipeline: ZeroCF replaces LLM-hallucinated important words with BERT attribution scores, and FitCF verifies those counterfactuals with a BERT label-flip check and uses them as few-shot demonstrations. Second, the headline result is not established as stated, because the same LLM both generates the counterfactuals and evaluates the label flip (Section 5.1, Appendix G). The 100% SLFR on SST2 for Llama3-8B is the tell: that model's own zero-shot accuracy on SST2 is 89.75%, so a perfect flip rate is implausible as a property of the generated text; it smells like the generator confirming its own edits. The paper's defense (similar aggregate accuracy between LLM and BERT implies consistent tendency) does not hold—aggregate accuracy does not imply per-instance agreement, especially on adversarially edited inputs.\n\nWhat is genuinely good: the comparison against FIZLE, POLYJUICE, and BAE is appropriate, the ablation study separates the three components cleanly and gives a clear ranking (number of demonstrations matters most), and the authors are transparent about their choices—the Limitations section acknowledges the English-only scope and the reliance on a single BERT verifier. Code is released. The faithfulness-quality correlation is an interesting observation, but it rests on only four attribution methods and is essentially a ranking of two families (LIME/SHAP vs Gradient/IG); I would not call that a strong finding yet.\n\nSoft spots, in proportion. The evaluation metric issue is load-bearing. No error bars or significance tests accompany Table 1, which is a problem when differences between methods are often a few percentage points. The correlation analysis uses four data points and no uncertainty; it is suggestive only. Minor: the conclusion text says 'LIME and Integrated Gradients are the most effective' while Section 6.3 and the abstract say LIME and SHAP—an inconsistency that should be caught.\n\nWho this is for: anyone building on zero-shot or few-shot counterfactual generation in text classification, and researchers combining feature attribution with generation. It deserves a serious referee, but the evaluation protocol needs to be redone with an independent classifier, ideally with per-example agreement and confidence intervals, before the central claim of consistent outperformance can be taken at face value.","headline":"A sensible, clearly described framework for feature-importance-guided counterfactual generation, but the headline result is undermined by the same-LLM evaluation protocol and the lack of uncertainty estimates.","tokens_in":18209,"tokens_out":1789,"would_cite":true,"duration_ms":16560,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Counterfactual text generation becomes more reliable when the edited words come from BERT feature attributions instead of the LLM itself, and automatically verified label-flipping demonstrations improve few-shot prompting.","keywords":["counterfactual generation","feature attribution","few-shot prompting","label flip verification","faithfulness","text classification","large language models"],"falsifier":"Recompute Soft Label Flip Rate on the same AG News and SST2 instances using an independent classifier—a different BERT variant or human annotators—rather than the LLM that generated the counterfactuals. If FitCF's flip-rate advantage over FIZLE shrinks or vanishes, the reported gains are at least partly self-consistency rather than genuine label flips.","tokens_in":17204,"feed_emoji":"🔄","tokens_out":9579,"duration_ms":83840,"temperature":0.7,"pith_summary":"The paper tries to establish that counterfactual examples—minimal edits that flip a classifier's prediction—are generated more reliably when the words an LLM edits are supplied by BERT feature attributions rather than by the LLM's own guesses, and that feeding verified examples back as few-shot demonstrations improves quality further. It introduces ZeroCF for the zero-shot step and FitCF for the verified-demonstration framework. The central payoffs are practical: no human-written counterfactuals are needed, and the faithfulness of the attribution method becomes a usable predictor of generation quality. The paper reports that FitCF consistently beats three state-of-the-art baselines on AG News and SST2, with the number of demonstrations the most influential component.","feed_headline":"BERT attribution beats LLM self-guessing for counterfactuals","feed_subtitle":"BERT-derived important words and verified flips beat three counterfactual baselines.","key_machinery":"The carrying object is the verified demonstration. ZeroCF first extracts the top-attributed words from a BERT classifier (using Gradient, Integrated Gradients, LIME, or SHAP) and prompts an LLM to edit the input in a zero-shot setting. FitCF then uses the same BERT model as a label-flip verifier: a candidate counterfactual is kept as a demonstration only if the BERT prediction on the edited text differs from the prediction on the original. These verified input–counterfactual pairs are selected from clusters of sentence embeddings and inserted into the final few-shot prompt, so the LLM generates the target counterfactual by analogy to examples that are known to flip the label. The verifier is the gate that prevents flawed zero-shot outputs from poisoning the demonstrations.","core_discovery":"On the paper's own terms, the central discovery is that feature attribution scores from a fine-tuned BERT model can serve as trustworthy edit cues for counterfactual generation, and that label-flip verification turns those zero-shot outputs into demonstrations that make few-shot prompting stronger. The authors report that ZeroCF often outperforms the zero-shot baseline FIZLE, and that FitCF consistently outperforms POLYJUICE, BAE, FIZLE, and ZeroCF on both datasets across all evaluated LLMs. Ablations show each of the three components—important words, demonstrations, and verification—contributes, with the number of demonstrations dominating. The paper also reports a strong correlation between faithfulness of the attribution scores and quality of the generated counterfactuals, up to Kendall's $\\tau = 1$ for sufficiency on AG News, which it frames as a useful finding for future work.","pith_inferences":["Editorial inference: if the faithfulness–quality correlation generalizes, a cheap pre-filter emerges: evaluate attribution faithfulness on a small validation set and pick the most faithful method, saving the compute of generating counterfactuals for every candidate attribution method.","Editorial inference: because the Soft Label Flip Rate uses the same LLM for generation and verification, some of the reported advantage may be self-agreement; a held-out classifier or human judgment would separate genuine flips from consistent errors.","Editorial inference: the verified-demonstration loop should transfer beyond text classification—question answering and open-ended generation would need a different correctness oracle, but the verify-then-demonstrate pattern is the same.","Editorial inference: using a single BERT model for both attribution and verification means the findings are anchored to that explainee; retraining or swapping the target classifier could change which words matter and therefore which counterfactuals pass verification."],"forward_implications":["If correct, counterfactual datasets for classification can be assembled automatically, since verified LLM edits replace human-crafted demonstrations.","If correct, attribution faithfulness becomes a practical selection criterion: choosing a more faithful method such as LIME or SHAP should yield better counterfactuals.","If correct, the number of verified demonstrations is the highest-leverage tuning knob in the pipeline, more so than the exact attribution method.","If correct, the framework transfers beyond BERT because the paper's design only requires any effective classifier as the attribution and verification backbone."],"supporting_citations":[{"why":"FIZLE, the closest zero-shot baseline; the paper replaces its LLM-generated important words with BERT attribution words.","marker":"Bhattacharjee et al. (2024a)"},{"why":"POLYJUICE, a counterfactual generation baseline FitCF must beat.","marker":"Wu et al. (2021)"},{"why":"BAE, a BERT-based perturbation baseline FitCF must beat.","marker":"Garg and Ramakrishnan (2020)"},{"why":"Auto-CoT, the source of the idea of automatically constructed demonstrations for few-shot prompting.","marker":"Zhang et al. (2023)"},{"why":"LIME, one of the feature attribution methods whose scores select important words.","marker":"Ribeiro et al. (2016)"},{"why":"SHAP, another attribution method used and found more faithful.","marker":"Lundberg and Lee (2017)"},{"why":"Supplies the faithfulness metrics (comprehensiveness, sufficiency, Kendall's tau) used in the correlation analysis.","marker":"Attanasio et al. (2023)"},{"why":"BERT, the model that provides feature attributions and label-flip verification.","marker":"Devlin et al. (2019)"}],"fun_headline_variants":["Feature importance guides better counterfactual examples","Label-flip verification boosts few-shot counterfactual generation","Attribution faithfulness predicts counterfactual quality","BERT's important words beat LLM self-guessing for counterfactuals","Verified flips turn zero-shot counterfactuals into few-shot wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a label flip counted by the same LLM that wrote the counterfactual is a true label flip; if generator and evaluator share the same errors, high flip rates can just show the model agrees with itself.","fun_headline_variants_meta":{"raw":{"variants":["Feature importance guides better counterfactual examples","Label-flip verification boosts few-shot counterfactual generation","Attribution faithfulness predicts counterfactual quality","BERT's important words beat LLM self-guessing for counterfactuals","Verified flips turn zero-shot counterfactuals into few-shot wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3060,"prompt_tokens":943,"completion_tokens":2117,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2047}},"tokens_in":559,"tokens_out":2117,"duration_ms":15201,"temperature":1.0,"reasoning_tokens":2047,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:42:08.363810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute Soft Label Flip Rate on the same AG News and SST2 instances using an independent classifier—a different BERT variant or human annotators—rather than the LLM that generated the counterfactuals. If FitCF's flip-rate advantage over FIZLE shrinks or vanishes, the reported gains are at least partly self-consistency rather than genuine label flips.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the faithfulness metrics (comprehensiveness, sufficiency, Kendall's tau) used in the correlation analysis."}],"review_version":1}