A schema-level diagnostic uses multi-annotator criterion judgments to separate unstable criteria from systematic category overlaps in subjective NLP annotation prior to gold-label creation.
Confident learning: Estimating uncertainty in dataset labels.Journal of Artificial Intelligence Research, 70:1373–1411, 2021a
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
representative citing papers
FrontierMath is a new benchmark of hundreds of original hard math problems that current AI models solve less than 2% of.
Wake Vision pipeline produces a 6M-image person detection dataset for TinyML with 2.2% label error, improving model accuracy up to 6.6% over prior VWW benchmark across architectures and subsets.
MMGist filters 23,250 items from 18 benchmarks down to 7,262 using three-stage pipeline, preserving model rankings (Spearman ρ=0.98) while cutting items 69% and raising discrimination 78%.
An automated AEB annotation framework uses data augmentation and noise suppression to achieve 80% recall improvement and 50% workload reduction for rare delayed/false triggers under class imbalance and asymmetric label noise.
Representational alignment varies monotonically with SNR and non-monotonically with sample size (minimized near interpolation threshold) across linear and nonlinear networks, and is decoupled from generalization error.
A procedurally generated benchmark decomposes spatial-reasoning tests into four compositional axes and shows LLMs degrade sharply with reasoning depth.
A validation-free metric combining neighbor-consistency and effective rank to estimate face recognition dataset quality for downstream model performance.
A scoping review organizes decades of NLP evaluation debates into a taxonomy of recurring concerns and trade-offs with a structured checklist for better evaluation design.
citing papers explorer
-
Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks
A schema-level diagnostic uses multi-annotator criterion judgments to separate unstable criteria from systematic category overlaps in subjective NLP annotation prior to gold-label creation.
-
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
FrontierMath is a new benchmark of hundreds of original hard math problems that current AI models solve less than 2% of.
-
Wake Vision: A Tailored Dataset and Benchmark Suite for TinyML Computer Vision Applications
Wake Vision pipeline produces a 6M-image person detection dataset for TinyML with 2.2% label error, improving model accuracy up to 6.6% over prior VWW benchmark across architectures and subsets.
-
MMGist: A Comprehensive Multimodal Benchmark for 2027
MMGist filters 23,250 items from 18 benchmarks down to 7,262 using three-stage pipeline, preserving model rankings (Spearman ρ=0.98) while cutting items 69% and raising discrimination 78%.
-
Learning to Annotate Delayed and False AEB Events: A Practical System for Extreme Class Imbalance and Asymmetric Label Noise
An automated AEB annotation framework uses data augmentation and noise suppression to achieve 80% recall improvement and 50% workload reduction for rare delayed/false triggers under class imbalance and asymmetric label noise.
-
Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networks
Representational alignment varies monotonically with SNR and non-monotonically with sample size (minimized near interpolation threshold) across linear and nonlinear networks, and is decoupled from generalization error.
-
DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
A procedurally generated benchmark decomposes spatial-reasoning tests into four compositional axes and shows LLMs degrade sharply with reasoning depth.
-
Efficient, Validation-Free Intrinsic Quality Estimation for Large-Scale Face Recognition Datasets
A validation-free metric combining neighbor-consistency and effective rank to estimate face recognition dataset quality for downstream model performance.
-
Evaluation Revisited: A Taxonomy of Evaluation Concerns in Natural Language Processing
A scoping review organizes decades of NLP evaluation debates into a taxonomy of recurring concerns and trade-offs with a structured checklist for better evaluation design.
- Semantic Trimming and Auxiliary Multi-step Prediction for Generative Recommendation