REVIEW 3 major objections 4 minor 50 references
Detecting Mislabeled and Corrupted Data via Pointwise Mutual Information
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that rating each training sample by its pointwise mutual information with its label separates corrupted from clean data, so training on only high-scoring samples improves accuracy under label noise.
desk verdict Plausible PMI-based data filter, but the abstract's 15% claim is unverifiable from the corrupted text; needs a referee to check the estimator. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the per-sample pointwise mutual information $\mathrm{PMI}(x,y)=\log\left(p(x,y)/(p(x)p(y))\right)$, which decomposes the dataset's total mutual information into per-sample contributions. The method estimates this quantity from the (possibly corrupted) training set, ranks samples by it, and discards the lowest-ranked samples before training. The ranking is the load-bearing device: the paper's claim is that low pointwise contribution co-occurs with label noise and input corruption, so no separate clean-validation set is required.
What would settle it
Take MNIST with a known corruption mask: flip 20% of labels randomly and add noise to 20% of images, compute each sample's PMI from the corrupted training set, then measure precision at the bottom of the ranking against the mask. If the lowest-PMI samples are not enriched for the injected corruptions beyond chance, or if clean but unusual digits are consistently ranked below corrupted typical digits, the central claim is false.
Extended reading notes
Core claim
The paper proposes that every training example can be assigned a pointwise mutual information score $\log\left(p(x,y)/(p(x)p(y))\right)$, and that this score separates clean from corrupted data: mislabeled and noisy inputs have systematically lower contributions to the overall mutual information. On MNIST with synthetic label noise and input corruption, the paper reports that keeping only high-PMI samples for training raises classification accuracy by up to 15% compared to random sampling, and that benign input modifications that preserve the semantic label do not get filtered out, while truly corrupted samples do.
Load-bearing premise
The method's ranking is only as good as the estimate of pointwise mutual information made from the corrupted training data itself; if that estimate is biased by the noise, or if rare but clean inputs naturally have low scores, the filter will remove the wrong samples.
Editorial extensions
If this is right
- Under label corruption, discarding low-PMI samples before training should improve test accuracy compared with training on the same number of randomly selected samples.
- Because the score is computed per sample, the method can act as a preprocessing filter for any downstream classifier, not only for the network used to estimate the score.
- Benign input modifications that preserve the label should keep high PMI, so the filter should remove corrupted inputs without discarding semantically valid augmentations.
- The ranking itself provides a label-quality audit: low-ranked samples are candidates for human review or relabeling.
Reading between the lines
- Beyond the paper's MNIST experiments, the same per-sample PMI ranking could be applied to non-image modalities such as text or tabular data, wherever a tractable estimate of $p(x,y)$ exists.
- If the estimator is fitted on the corrupted set itself, the method may be vulnerable to the very noise it detects—for example, a mislabeled sample that is highly typical of its wrong class could earn a high PMI score; a careful reader should check whether the reported gains come mostly from removing outliers rather than correcting labels.
- Pointwise MI could be combined with loss-based filter methods: samples that are both low-PMI and high-loss are stronger corruption candidates than either signal alone, a testable extension the paper does not run.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a data-selection framework for neural network training under hybrid label and input noise. The central idea is to rank training samples by their pointwise contribution to the overall mutual information between inputs and labels, and to discard samples with low pointwise mutual information (PMI). The abstract claims that lower PMI identifies noisy or mislabeled instances, that training on high-PMI samples improves classification accuracy by up to 15% over random sampling under label corruption, and that the method is robust to benign input modifications. The full text as provided is heavily corrupted by character-encoding errors, so the detailed method, equations, tables, and figures are not readable. The assessment is therefore based primarily on the abstract and on a few equation fragments that appear in the corrupted text.
Significance. If the claim holds, the method could offer a simple, interpretable, information-theoretic criterion for training-data filtering that goes beyond ad-hoc loss-based heuristics. The abstract's claim is falsifiable and the proposed PMI ranking is a plausible extension of existing small-loss and confidence-based selection methods. However, the current manuscript does not allow verification of the central claim: no estimator is specified, no baseline comparisons or error bars are visible, and only one dataset with synthetic noise is mentioned. The significance is therefore conditional; at present the evidence is too thin to establish a substantive advance over standard noisy-label baselines.
major comments (3)
- [Method, as described in the Abstract] The abstract states that "we compute each sample's pointwise contribution to the overall mutual information," but it never specifies how p(x,y) or p(y|x) is estimated. Since PMI(x;y) = log p(x,y)/(p(x)p(y)) = log p(y|x) - log p(y), any implementation requires an estimate of the conditional label distribution. If p(y|x) is obtained from a neural network trained on the same noisy labels, then the ranking reduces to a model-confidence/loss ranking and is vulnerable to the memorization effect: after convergence, corrupted samples receive high p(y|x) and hence high PMI, while rare or underfit clean samples may receive low PMI. This confound is unaddressed. Please specify the estimator, and provide a control experiment that separates PMI from model confidence—for example, compare against the small-loss selection and report results at both early and late training epochs.
- [Results/Abstract] The central empirical claim is "improves classification accuracy by up to 15% compared to random sampling." As stated, this is based on a single dataset (MNIST), synthetic noise, and a best-case framing, with no error bars and no named strong baselines. The abstract-level evidence cannot support a general claim about the method's effectiveness. Please report mean and standard deviation over multiple seeds and noise realizations, show full learning curves, and compare against standard noisy-label methods (e.g., small-loss selection, Co-teaching, DivideMix) at matched retention fractions. Without this, the accuracy improvement cannot be attributed to PMI-based filtering rather than to simply discarding difficult examples.
- [Method/Interpretation] For uniform random label corruption, a corrupted sample has a label that is independent of its input, so its PMI is near zero by construction. The statement "lower contributions indicate noisy or mislabeled instances" is therefore partly tautological. The substantive claim is that retaining high-MI samples improves learned classification accuracy, and that this improvement is not obtained by dropping hard or rare clean inputs. This distinction should be made explicit, and the experiments should show that the bottom of the PMI ranking is enriched for corruptions rather than for rare or atypical clean examples.
minor comments (4)
- [Notation] Provide formal notation for PMI and the empirical estimator. The manuscript should clearly define the sample-level pointwise mutual information, how the marginals and joint are estimated, and how the retention fraction or PMI threshold is chosen.
- [Full text / figures] The submitted full text is corrupted by character-encoding errors; most of the body is unreadable replacement characters, including the method description and result tables/figures. The PDF or source must be re-uploaded with correct encoding so that the derivation and the empirical results can be assessed.
- [Related work] The manuscript cites related work only implicitly. Please add explicit comparisons and references to the small-loss selection literature, confident-learning approaches, and noise-robust training methods, so that the contribution is clearly positioned.
- [Input-noise experiments] Clarify what is meant by "benign input modifications" and "truly corrupted samples." Define the corruption types, their severity, and the evaluation metric used for the input-filtering claim.
Circularity Check
Low-PMI-means-corruption is definitional, but the filtering/accuracy evaluation is external.
-
self definitional
[Abstract; Section 2 (Preliminaries), definition of pointwise mutual information]
"We compute each sample's pointwise contribution to the overall mutual information and find that lower contributions indicate noisy or mislabeled instances."
The 'pointwise contribution' is the pointwise mutual information pmi(x,y)=log p(x,y)/(p(x)p(y)). Under the paper's synthetic label-corruption model, a corrupted label is drawn independently of x, so p(x,y)=p(x)p(y) and pmi=0 (or near zero in finite-sample estimates), while clean labels have positive pmi in expectation. Therefore 'lower contributions indicate noisy labels' is not an empirical finding but a restatement of the definition of pointwise MI plus the assumed independence of the corruption process. The claim is true by construction; the independent content of the paper rests on the later MNIST test-accuracy comparison after filtering.
full rationale
The abstract's sentence conflates a definitional consequence with an empirical discovery. However, the paper's data-selection claim is validated by retraining on the high-MI subset and measuring test accuracy, which is an external benchmark not encoded in the PMI definition. Thus the circularity is partial and localized to the wording of the central 'indication' claim; no self-citation or imported uniqueness issues are present. Score 4.
Assumptions & free parameters
free parameters (2)
- MI ranking threshold / retention fraction
- MI estimator configuration
assumptions (3)
- domain assumption The empirical joint distribution p(x,y) estimated from the corrupted training set is accurate enough for per-sample MI contributions to rank samples by true label quality.
- domain assumption Synthetic noise used in the MNIST experiments is representative of real-world label and input corruption.
- domain assumption The additive decomposition of mutual information into per-sample contributions is the correct sample-level quality score.
Cite this review
Pith. "Pith review of Detecting Mislabeled and Corrupted Data via Pointwise Mutual Information." pith.science (2026). https://pith.science/paper/OMLE4V37
@misc{pith2026250807713,
author = {Pith},
title = {Pith review of: Detecting Mislabeled and Corrupted Data via Pointwise Mutual Information},
year = {2026},
howpublished = {\url{https://pith.science/paper/OMLE4V37}},
note = {Machine review of arXiv:2508.07713}
}
read the original abstract
Deep neural networks can memorize corrupted labels, making data quality critical for model performance, yet real-world datasets are frequently compromised by both label noise and input noise. This paper proposes a mutual information-based framework for data selection under hybrid noise scenarios that quantifies statistical dependencies between inputs and labels. We compute each sample's pointwise contribution to the overall mutual information and find that lower contributions indicate noisy or mislabeled instances. Empirical validation on MNIST with different synthetic noise settings demonstrates that the method effectively filters low-quality samples. Under label corruption, training on high-MI samples improves classification accuracy by up to 15\% compared to random sampling. Furthermore, the method exhibits robustness to benign input modifications, preserving semantically valid data while filtering truly corrupted samples.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Unsupervised label noise modeling and loss correction
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In Proc. ICML, 2019
work page 2019
-
[3]
A closer look at memorization in deep networks
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In Proc. ICML, pages 233--242, 2017
work page 2017
-
[4]
Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise. In Proc. AAAI, 2021
work page 2021
-
[5]
BERT : P re-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : P re-training of deep bidirectional transformers for language understanding. In Proc. ACL, pages 4171--4186, 2019
work page 2019
-
[6]
Classification in the presence of label noise: A survey
Beno \^ t Fr \'e nay and Michel Verleysen. Classification in the presence of label noise: A survey. IEEE Transaction on Neural Networks and Learning Systems, 25 0 (5): 0 845--869, 2013
work page 2013
-
[7]
Robust loss functions under label noise for deep neural networks
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Proc. AAAI, 2017
work page 2017
-
[8]
Co-teaching: R obust training of deep neural networks with extremely noisy labels
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: R obust training of deep neural networks with extremely noisy labels. In Proc. NeurIPS, pages 8527--8537, 2018
work page 2018
Show all 50 references
-
[9]
Robot data curation with mutual information estimators
Joey Hejna, Suvir Mirchandani, Ashwin Balakrishna, Annie Xie, Ayzaan Wahid, Jonathan Tompson, Pannag Sanketi, Dhruv Shah, Coline Devin, and Dorsa Sadigh. Robot data curation with mutual information estimators. arXiv preprint arXiv:2502.08623, 2025
2025 arXiv
-
[10]
Using trusted data to train deep networks on labels corrupted by severe noise
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In Proc. NeurIPS, pages 10456--10465, 2018
2018
-
[11]
Using pre-training can improve model robustness and uncertainty
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Proc. ICML, 2019
2019
-
[12]
Universal language model fine-tuning for text classification
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proc. ACL, pages 328--339, 2018
2018
-
[13]
Mentor N et: L earning data-driven curriculum for very deep neural networks on corrupted labels
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentor N et: L earning data-driven curriculum for very deep neural networks on corrupted labels. In Proc. ICML, 2018
2018
-
[14]
Understanding black-box predictions via influence functions
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885--1894. PMLR, 2017
2017
-
[15]
Submodular mutual information for targeted data subset selection
Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Submodular mutual information for targeted data subset selection. arXiv preprint arXiv:2105.00043, 2021
2021 arXiv
-
[16]
Estimating mutual information
Alexander Kraskov, Harald St \"o gbauer, and Peter Grassberger. Estimating mutual information. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 69 0 (6): 0 066138, 2004
2004
-
[17]
Image N et classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Image N et classification with deep convolutional neural networks. In Proc. NeurIPS, pages 1097--1105, 2012
2012
-
[18]
Clean N et: T ransfer learning for scalable image classifier training with label noise
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Clean N et: T ransfer learning for scalable image classifier training with label noise. In Proc. CVPR, pages 5447--5456, 2018
2018
-
[19]
Divide M ix: L earning with noisy labels as semi-supervised learning
Junnan Li, Richard Socher, and Steven CH Hoi. Divide M ix: L earning with noisy labels as semi-supervised learning. In Proc. ICLR, 2020
2020
-
[20]
Webvision database: V isual learning and understanding from web data
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: V isual learning and understanding from web data. arXiv preprint arXiv:1708.02862, 2017
2017 arXiv
-
[21]
Early-learning regularization prevents memorization of noisy labels
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. In Proc. NeurIPS, 2020
2020
-
[22]
Observer variation in the diagnosis of follicular variant of papillary thyroid carcinoma
Ricardo V Lloyd, Lori A Erickson, Mary B Casey, King Y Lam, Christine M Lohse, Sylvia L Asa, John KC Chan, Ronald A DeLellis, H Ruben Harach, Kennichi Kakudo, et al. Observer variation in the diagnosis of follicular variant of papillary thyroid carcinoma. The American Journal ...
2004
-
[23]
Does label smoothing mitigate label noise? In Proc
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In Proc. ICLR, pages 6448--6458, 2020
2020
-
[24]
Statistical undersampling with mutual information and support points
Alex Mak, Shubham Sahoo, Shivani Pandey, Yidan Yue, and Linglong Kong. Statistical undersampling with mutual information and support points. arXiv preprint arXiv:2412.14527, 2024
2024 arXiv
-
[25]
Conducting behavioral research on amazon’s mechanical turk
Winter Mason and Siddharth Suri. Conducting behavioral research on amazon’s mechanical turk. Behavior Research Methods, 44 0 (1): 0 1--23, 2012
2012
-
[26]
Neural information retrieval: A t the end of the early years
Kezban Dilek Onal, Ye Zhang, Ismail Sengor Altingovde, Md Mustafizur Rahman, Pinar Karagoz, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quinten McNamara, et al. Neural information retrieval: A t the end of the early years. Information Retrieval Journal, 21 0 (2-3): 0...
2018
-
[27]
Deeprank: A new deep architecture for relevance ranking in information retrieval
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Jingfang Xu, and Xueqi Cheng. Deeprank: A new deep architecture for relevance ranking in information retrieval. In Proc. CIKM, pages 257--266, 2017
2017
-
[28]
Running experiments on amazon mechanical turk
Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. Running experiments on amazon mechanical turk. Judgment and Decision Making, 5 0 (5): 0 411--419, 2010
2010
-
[29]
Making deep neural networks robust to label noise: A loss correction approach
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proc. CVPR, pages 1944--1952, 2017
1944
-
[30]
You only look once: U nified, real-time object detection
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: U nified, real-time object detection. In Proc. CVPR, pages 779--788, 2016
2016
-
[31]
Twitter sentiment analysis with deep convolutional neural networks
Aliaksei Severyn and Alessandro Moschitti. Twitter sentiment analysis with deep convolutional neural networks. In Proc. ACL, pages 959--962, 2015
2015
-
[32]
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27 0 (3): 0 379--423, 1948
1948
-
[33]
Meta- W eight- N et: L earning an explicit mapping for sample weighting
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- W eight- N et: L earning an explicit mapping for sample weighting. In Proc. NeurIPS, pages 1917--1928, 2019
1917
-
[34]
SELFIE : R efurbishing unclean samples for robust deep learning
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE : R efurbishing unclean samples for robust deep learning. In Proc. ICML, pages 5907--5915, 2019
2019
-
[35]
Learning from noisy labels with deep neural networks: A survey
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems, 34 0 (11): 0 8135--8153, 2022
2022
-
[36]
Interactive label cleaning with example-based explanations
Stefano Teso, Andrea Bontempelli, Fausto Giunchiglia, and Andrea Passerini. Interactive label cleaning with example-based explanations. Advances in Neural Information Processing Systems, 34: 0 12966--12977, 2021
2021
-
[37]
Symmetric cross entropy for robust learning with noisy labels
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proc. ICCV, pages 322--330, 2019
2019
-
[38]
To smooth or not? when label smoothing meets noisy labels
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. arXiv preprint arXiv:2106.04149, 2021
2021 arXiv
-
[39]
Are anchor points really indispensable in label-noise learning? In Proc
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In Proc. NeurIPS, 2019
2019
-
[40]
Robust early-learning: Hindering the memorization of noisy labels
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In Proc. ICLR, 2021
2021
-
[41]
Adversarial label flips attack on support vector machines
Han Xiao, Huang Xiao, and Claudia Eckert. Adversarial label flips attack on support vector machines. In Proc. ECAI, pages 870--875, 2012
2012
-
[42]
Learning from massive noisy labeled data for image classification
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proc. CVPR, pages 2691--2699, 2015
2015
-
[43]
Relabeling minimal training subset to flip a prediction
Jinghan Yang, Linjie Xu, and Lequan Yu. Relabeling minimal training subset to flip a prediction. arXiv preprint arXiv:2305.12809, 2023
2023 arXiv
-
[44]
Dual T : Reducing estimation error for transition matrix in label-noise learning
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual T : Reducing estimation error for transition matrix in label-noise learning. In Proc. NeurIPS, 2020
2020
-
[45]
Mutual information based data selection in gaussian processes for people tracking
Zulkarnain Zainudin, Sarath Kodagoda, and LV Nguyen. Mutual information based data selection in gaussian processes for people tracking. In Proc. Australian Conference on Robotics and Automation, Wellington, New Zealand, 2012
2012
-
[46]
Understanding deep learning requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In Proc. ICLR, 2017 a
2017
-
[47]
mixup: Beyond empirical risk minimization
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017 b
2017 arXiv
-
[48]
Deep learning over multi-field categorical data
Weinan Zhang, Tianming Du, and Jun Wang. Deep learning over multi-field categorical data. In Proc. ECIR, pages 45--57, 2016
2016
-
[49]
Generalized cross entropy loss for training deep neural networks with noisy labels
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proc. NeurIPS, pages 8778--8788, 2018
2018
-
[50]
Class noise vs
Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: A quantitative study. Artificial Intelligence Review, 22 0 (3): 0 177--210, 2004
2004
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.