Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics
Reviewed by Pithpith:HISKM452open to challenge →
read the original abstract
Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.
This paper has not been read by Pith yet.
Forward citations
Cited by 3 Pith papers
-
DIPBox: A Multi-scale Testing Framework for Tracking Dataset Regeneration
DIPBox is the first multi-scale testing framework for detecting adversarial dataset regeneration via four similarity metrics, backed by learning-theoretic analysis of utility-divergence trade-offs.
-
On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference
An attack aligns differently shuffled intermediate activations from secure Transformer inference queries to recover model weights with low error using roughly one dollar of queries.
-
WARBERT: A Hierarchical BERT-based Model for Web API Recommendation
WARBERT is a hierarchical BERT-based model that combines recommendation-style filtering and similarity matching to improve Web API recommendations for mashups on the ProgrammableWeb dataset.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.