REVIEW 13 cited by
Annotation Artifacts in Natural Language Inference Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et. al, 2015) and 53% of MultiNLI (Williams et. al, 2017). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.
Forward citations
Cited by 13 Pith papers
-
When Local Monitors Miss Compositional Harm: Diagnosing Distributed Backdoors in Multi-Agent Systems
Once attack fragments are locally benign and ε-indistinguishable from benign traffic, no local monitor can separate them (TPR−FPR≤ε); the signal reappears only in the right assembled representation.
-
Language Models as Knowledge Bases?
BERT stores relational knowledge extractable via cloze queries without fine-tuning and matches supervised baselines on open-domain QA tasks.
-
idSCD: Identifying Training Datasets through Semantic Correlation Descriptors
idSCD uses semantic correlation descriptors to perform dataset membership inference by comparing learned semantic structures, outperforming baselines in NLI, emotion, and medical text experiments.
-
The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inference
Across four clinical inference tasks, six LLMs answer paired knowledge probes at 92% accuracy but the main reasoning tasks at 25%, indicating a systematic knowledge-reasoning gap.
-
Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization
A unified moment-alignment theory bounds target-domain error by cross-domain differences in loss derivatives, and the new CMA algorithm implements exact gradient and Hessian matching in closed form.
-
ART: Automatic multi-step reasoning and tool-use for large language models
ART automatically generates multi-step reasoning programs with tool integration for LLMs, yielding substantial gains over few-shot and auto-CoT prompting on BigBench and MMLU while matching hand-crafted CoT on most tasks.
-
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets
Annotator identity functions as a shortcut in crowdsourced NLU benchmarks, and models often fail to generalize to examples from annotators absent from training.
-
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
Benchmark items are heterogeneous; the authors annotate them with three LLM judges and use the labels to build filtered subsets, but the validation evidence is weak and the comparison method is flawed.
-
PROBE: Benchmarking Code Generation in Large Language Models
A multi-language evaluation framework measuring correctness, solution proximity, and code quality finds current LLMs pass at most ~0.70 per language and worsen sharply with problem difficulty.
-
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge
MEMAT combines MEMIT weight edits with optimized attention-head corrections, improving cross-lingual success and magnitude metrics over MEMIT in English and Catalan.
-
Knowledge Enhanced Attention for Robust Natural Language Inference
Injecting WordNet lexical relations as bias terms into multi-head attention improves accuracy on the adversarial SNLI test set, with BERT reaching 94.1%, equal to estimated human performance.
-
Investigating Biases in Textual Entailment Datasets
Hypothesis-only classification reaches 64% accuracy on SNLI, revealing dataset biases in SNLI and MultiNLI that the authors quantify and propose a simple mitigation for.
-
Unpacking the Resilience of SNLI Contradiction Examples to Attacks
Adding a single universal trigger word to SNLI hypotheses crashes ELECTRA's accuracy on entailment and neutral classes, barely affects contradiction, and trigger-augmented fine-tuning recovers the lost accuracy.
Discussion (0). Continue with ORCID to comment.