REVIEW 7 cited by
Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite alarm over the reliance of machine learning systems on so-called spurious patterns, the term lacks coherent meaning in standard statistical frameworks. However, the language of causality offers clarity: spurious associations are due to confounding (e.g., a common cause), but not direct or indirect causal effects. In this paper, we focus on natural language processing, introducing methods and resources for training models less sensitive to spurious patterns. Given documents and their initial labels, we task humans with revising each document so that it (i) accords with a counterfactual target label; (ii) retains internal coherence; and (iii) avoids unnecessary changes. Interestingly, on sentiment analysis and natural language inference tasks, classifiers trained on original data fail on their counterfactually-revised counterparts and vice versa. Classifiers trained on combined datasets perform remarkably well, just shy of those specialized to either domain. While classifiers trained on either original or manipulated data alone are sensitive to spurious features (e.g., mentions of genre), models trained on the combined data are less sensitive to this signal. Both datasets are publicly available.
Forward citations
Cited by 7 Pith papers
-
Auditing Evidence Use in Medical LLM Diagnosis
Behavioral auditing of five medical LLMs shows most mined evidence interactions are clinically plausible, while adjudicated shortcut-like failures concentrate in negated or absent findings and clinically local evidence.
-
The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators
Fixed activation probes keep near-ceiling accuracy on harmful-vs-benign corpus contrasts but fall to AUROC 0.59-0.69 on topic- and surface-matched harmful/benign pairs, so they behave as broad-risk detectors, not cont...
-
Understanding and evaluating computer vision models through the lens of counterfactuals
Counterfactual-based methods for concept attribution in classifiers and for dynamic bias evaluation and mitigation in text-to-image models.
-
VISION: Robust and Interpretable Code Vulnerability Detection Leveraging Counterfactual Augmentation
LLM-generated counterfactual code pairs with flipped vulnerability labels, used to train a GNN, sharply improve CWE-20 detection and attribution on the released CWE-20-CFA benchmark.
-
Entangled by Design: Spurious Intra-Variable Signal Routing in Tabular In-Context Learners
In-context learners route predictions through a spurious component inside a composite feature whenever that component correlates with the label, and the routing persists as context grows.
-
Position: Stop Reactively Patching Your Model Every Time and Start Proactive Test-Driven AI Development
In a stylized model, a proactive flywheel that fixes whole groups of related scenarios needs Θ(K log K) update rounds versus Θ(M log M) for reactive patching.
-
Actively evaluating and learning the distinctions that matter: Vaccine safety signal detection from emergency triage notes
An active-learning pipeline with counterfactual data augmentation achieved F1 0.97 for detecting potential vaccine adverse events in emergency triage notes, but the evaluation was not independent of model training.
Discussion (0). Sign in to comment.