{"id":"e09b32e0-ec7e-490e-b1ce-9918a675356a","arxiv_id":"2508.07713","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Samples with low pointwise mutual information between image and label are mostly mislabeled or corrupted, and dropping them before training improves MNIST accuracy by up to 15%.","lead":"Training data often contains wrong labels or corrupted images, which hurts learning. This paper ranks each image by how much it helps the statistical link between images and labels, removes the low-scoring ones, and reports up to 15% better accuracy than random sampling on corrupted MNIST data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Low PMI scores may just reflect a memorizing classifier's confidence rather than label corruption; the estimator is not specified and this confound is unaddressed.","rationale":"The reader's weakest assumption concerns estimator reliability; I agree. The mathematical identity PMI(x;y)=log p(y|x)-log p(y) shows that p(x) drops out, so the more precise load-bearing condition is the estimation of p(y|x), not the full joint density. If that estimate comes from a model trained on the corrupted labels, the method inherits memorization dynamics and can fail exactly when corrupted labels are memorized. The proposed AUROC test separates the two failure modes: label corruption vs. model confidence/rarity. I recommend CONDITIONAL rather than REJECT because the claim may be salvageable with a specified estimator plus a memorization guard (e.g., early stopping or a robust loss); as written, the paper does not rule out the confound. No code, machine-checked proof, or independent artifacts are visible in the supplied fragments to offset this gap.","tokens_in":15740,"tokens_out":10897,"duration_ms":129339,"concrete_test":"From the readable PDF, implement the exact PMI estimator (Section 3) and run it on MNIST with 40% symmetric label noise. Train the estimator in two regimes: (i) early stopping with the paper's stated criterion, and (ii) full convergence (100% training accuracy). For each regime, compute the AUROC of the PMI scores against the known corruption mask. Also report the fraction of clean labels among the top-60% PMI samples. If the convergence regime's AUROC is near 0.5 or the top-60% set is not enriched for clean labels, the PMI ranking is an artifact of underfitting, and the central claim needs an explicit memorization guard.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the per-sample 'pointwise contribution' to MI can be computed from the corrupted training set in a way that is not itself distorted by the corruption. Since PMI(x;y) = log p(x,y)/(p(x)p(y)) = log p(y|x) - log p(y), any tractable implementation needs an estimate of p(y|x). The abstract and the readable fragments do not state which estimator is used. If p(y|x) is obtained from a neural network trained on the same noisy labels, then the ranking is exactly a confidence/loss ranking up to the label marginal. Such a ranking separates easy clean examples from hard corrupted ones only during the early phase of training; after convergence the network memorizes corrupted labels and assigns them high p(y|x), so their PMI is no longer low. In that regime, the bottom of the PMI ranking is dominated by rare, novel, or underfit clean inputs, not by mislabeled ones. The paper's 'up to 15%' accuracy gain could then come from discarding hard/rare examples rather than from detecting corruption. This is a load-bearing gap because the whole method is the estimator: without knowing it, the claim 'lower contributions indicate noisy or mislabeled instances' cannot be distinguished from 'lower contributions indicate low model confidence.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data-selection framework for neural network training under hybrid label and input noise. The central idea is to rank training samples by their pointwise contribution to the overall mutual information between inputs and labels, and to discard samples with low pointwise mutual information (PMI). The abstract claims that lower PMI identifies noisy or mislabeled instances, that training on high-PMI samples improves classification accuracy by up to 15% over random sampling under label corruption, and that the method is robust to benign input modifications. The full text as provided is heavily corrupted by character-encoding errors, so the detailed method, equations, tables, and figures are not readable. The assessment is therefore based primarily on the abstract and on a few equation fragments that appear in the corrupted text.","tokens_in":15898,"tokens_out":3893,"duration_ms":50671,"significance":"If the claim holds, the method could offer a simple, interpretable, information-theoretic criterion for training-data filtering that goes beyond ad-hoc loss-based heuristics. The abstract's claim is falsifiable and the proposed PMI ranking is a plausible extension of existing small-loss and confidence-based selection methods. However, the current manuscript does not allow verification of the central claim: no estimator is specified, no baseline comparisons or error bars are visible, and only one dataset with synthetic noise is mentioned. The significance is therefore conditional; at present the evidence is too thin to establish a substantive advance over standard noisy-label baselines.","major_comments":[{"comment":"The abstract states that \"we compute each sample's pointwise contribution to the overall mutual information,\" but it never specifies how p(x,y) or p(y|x) is estimated. Since PMI(x;y) = log p(x,y)/(p(x)p(y)) = log p(y|x) - log p(y), any implementation requires an estimate of the conditional label distribution. If p(y|x) is obtained from a neural network trained on the same noisy labels, then the ranking reduces to a model-confidence/loss ranking and is vulnerable to the memorization effect: after convergence, corrupted samples receive high p(y|x) and hence high PMI, while rare or underfit clean samples may receive low PMI. This confound is unaddressed. Please specify the estimator, and provide a control experiment that separates PMI from model confidence—for example, compare against the small-loss selection and report results at both early and late training epochs.","section":"Method, as described in the Abstract"},{"comment":"The central empirical claim is \"improves classification accuracy by up to 15% compared to random sampling.\" As stated, this is based on a single dataset (MNIST), synthetic noise, and a best-case framing, with no error bars and no named strong baselines. The abstract-level evidence cannot support a general claim about the method's effectiveness. Please report mean and standard deviation over multiple seeds and noise realizations, show full learning curves, and compare against standard noisy-label methods (e.g., small-loss selection, Co-teaching, DivideMix) at matched retention fractions. Without this, the accuracy improvement cannot be attributed to PMI-based filtering rather than to simply discarding difficult examples.","section":"Results/Abstract"},{"comment":"For uniform random label corruption, a corrupted sample has a label that is independent of its input, so its PMI is near zero by construction. The statement \"lower contributions indicate noisy or mislabeled instances\" is therefore partly tautological. The substantive claim is that retaining high-MI samples improves learned classification accuracy, and that this improvement is not obtained by dropping hard or rare clean inputs. This distinction should be made explicit, and the experiments should show that the bottom of the PMI ranking is enriched for corruptions rather than for rare or atypical clean examples.","section":"Method/Interpretation"}],"minor_comments":[{"comment":"Provide formal notation for PMI and the empirical estimator. The manuscript should clearly define the sample-level pointwise mutual information, how the marginals and joint are estimated, and how the retention fraction or PMI threshold is chosen.","section":"Notation"},{"comment":"The submitted full text is corrupted by character-encoding errors; most of the body is unreadable replacement characters, including the method description and result tables/figures. The PDF or source must be re-uploaded with correct encoding so that the derivation and the empirical results can be assessed.","section":"Full text / figures"},{"comment":"The manuscript cites related work only implicitly. Please add explicit comparisons and references to the small-loss selection literature, confident-learning approaches, and noise-robust training methods, so that the contribution is clearly positioned.","section":"Related work"},{"comment":"Clarify what is meant by \"benign input modifications\" and \"truly corrupted samples.\" Define the corruption types, their severity, and the evaluation metric used for the input-filtering claim.","section":"Input-noise experiments"}],"recommendation":"major_revision","confidential_remarks":"The full text is unreadable in the provided file, so I could not verify whether the estimator for p(y|x) is actually specified in the hidden or garbled portion. If the corrupted text is only an artifact of the submission format, the authors should be asked to resubmit a readable version before a substantive decision is made. In addition, the abstract-level claim is close to the well-known small-loss heuristic for noisy-label selection; the novelty and the empirical comparisons need to be strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the abstract promises a cheap PMI-based filter for noisy and corrupted samples, with a 15% accuracy gain on MNIST under label noise. That is plausible and worth taking seriously, but the text I have is too corrupted to verify the estimator, the baselines, or the error bars, so I am not ready to trust the headline number.\n\nWhat is new and what the paper does well: per-sample pointwise contributions to mutual information are a natural way to score label-input dependence, and extending the idea to joint label and input corruption is a useful niche. The empirical direction matches the small-loss/confidence-margin literature, so the effect is believable. The paper also frames the method as a pre-training filter, which is a practical angle.\n\nThe soft spots: the stress-test concern is real and load-bearing. If p(y|x) is estimated by a network trained on the same corrupted labels, then low PMI is nearly the same as low confidence, and after memorization the ranking stops separating corrupted from clean. The abstract does not say which estimator is used, and the corrupt full text does not let me check. A referee must see the estimator and, ideally, an ablation comparing it to small-loss selection. The circularity point is minor: for uniform random label corruption, near-zero PMI is built into the definition, but the external accuracy evaluation is independent evidence, so that is not a fatal flaw on its own. The evaluation is thin: one dataset, synthetic noise, \"up to 15%\" best-case framing, no error bars or named baselines in the abstract. These are the things I would need a careful referee to check, not reasons to dismiss the paper.\n\nRecommendation: this deserves peer review, not a desk reject, provided the full text is readable and the estimator is specified. The idea is not groundbreaking but it is a solid extension of a known tool, and the data-quality subfield can use a cheap pre-training filter. I would send it to a referee with a specific request to address the estimator confound and to compare against existing label-noise baselines.","headline":"Plausible PMI-based data filter, but the abstract's 15% claim is unverifiable from the corrupted text; needs a referee to check the estimator.","tokens_in":16479,"tokens_out":2466,"would_cite":false,"duration_ms":27935,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that rating each training sample by its pointwise mutual information with its label separates corrupted from clean data, so training on only high-scoring samples improves accuracy under label noise.","keywords":["pointwise mutual information","data selection","label noise","input noise","mislabeled data detection","MNIST","data quality","deep learning"],"falsifier":"Take MNIST with a known corruption mask: flip 20% of labels randomly and add noise to 20% of images, compute each sample's PMI from the corrupted training set, then measure precision at the bottom of the ranking against the mask. If the lowest-PMI samples are not enriched for the injected corruptions beyond chance, or if clean but unusual digits are consistently ranked below corrupted typical digits, the central claim is false.","tokens_in":15506,"feed_emoji":"","tokens_out":4974,"duration_ms":57583,"temperature":0.7,"pith_summary":"This paper is trying to establish a simple data-cleaning principle: a training sample's pointwise contribution to the mutual information between inputs and labels is a reliable signal of data quality. Samples whose input and label share little statistical dependence—low pointwise MI—are the ones most likely to be mislabeled or corrupted, while samples with high pointwise MI are worth keeping. If this is right, it gives practitioners a parameter-free way to select training data under both label noise and input noise, without needing a clean validation set. The paper demonstrates the mechanism on MNIST under synthetic noise, reporting accuracy gains up to 15% over random sampling when training on high-MI samples only.","feed_headline":"Mutual information scores flag bad labels, lift accuracy 15%","feed_subtitle":"On MNIST, training only on high-score samples beats random sampling under label corruption.","key_machinery":"The central object is the per-sample pointwise mutual information $\\mathrm{PMI}(x,y)=\\log\\left(p(x,y)/(p(x)p(y))\\right)$, which decomposes the dataset's total mutual information into per-sample contributions. The method estimates this quantity from the (possibly corrupted) training set, ranks samples by it, and discards the lowest-ranked samples before training. The ranking is the load-bearing device: the paper's claim is that low pointwise contribution co-occurs with label noise and input corruption, so no separate clean-validation set is required.","core_discovery":"The paper proposes that every training example can be assigned a pointwise mutual information score $\\log\\left(p(x,y)/(p(x)p(y))\\right)$, and that this score separates clean from corrupted data: mislabeled and noisy inputs have systematically lower contributions to the overall mutual information. On MNIST with synthetic label noise and input corruption, the paper reports that keeping only high-PMI samples for training raises classification accuracy by up to 15% compared to random sampling, and that benign input modifications that preserve the semantic label do not get filtered out, while truly corrupted samples do.","pith_inferences":["Beyond the paper's MNIST experiments, the same per-sample PMI ranking could be applied to non-image modalities such as text or tabular data, wherever a tractable estimate of $p(x,y)$ exists.","If the estimator is fitted on the corrupted set itself, the method may be vulnerable to the very noise it detects—for example, a mislabeled sample that is highly typical of its wrong class could earn a high PMI score; a careful reader should check whether the reported gains come mostly from removing outliers rather than correcting labels.","Pointwise MI could be combined with loss-based filter methods: samples that are both low-PMI and high-loss are stronger corruption candidates than either signal alone, a testable extension the paper does not run."],"forward_implications":["Under label corruption, discarding low-PMI samples before training should improve test accuracy compared with training on the same number of randomly selected samples.","Because the score is computed per sample, the method can act as a preprocessing filter for any downstream classifier, not only for the network used to estimate the score.","Benign input modifications that preserve the label should keep high PMI, so the filter should remove corrupted inputs without discarding semantically valid augmentations.","The ranking itself provides a label-quality audit: low-ranked samples are candidates for human review or relabeling."],"supporting_citations":[],"fun_headline_variants":["PMI scores spot bad labels, boost MNIST accuracy 15%","Mutual info per sample finds corrupted data","Use pointwise MI to filter noise, gain 15% on MNIST","Data quality: PMI separates clean from corrupted","Low mutual info? Your label might be wrong"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The method's ranking is only as good as the estimate of pointwise mutual information made from the corrupted training data itself; if that estimate is biased by the noise, or if rare but clean inputs naturally have low scores, the filter will remove the wrong samples.","fun_headline_variants_meta":{"raw":{"variants":["PMI scores spot bad labels, boost MNIST accuracy 15%","Mutual info per sample finds corrupted data","Use pointwise MI to filter noise, gain 15% on MNIST","Data quality: PMI separates clean from corrupted","Low mutual info? Your label might be wrong"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1034,"prompt_tokens":612,"completion_tokens":422,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":356,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":356,"tokens_out":422,"duration_ms":4678,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:55:19.254727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take MNIST with a known corruption mask: flip 20% of labels randomly and add noise to 20% of images, compute each sample's PMI from the corrupted training set, then measure precision at the bottom of the ranking against the mask. If the lowest-PMI samples are not enriched for the injected corruptions beyond chance, or if clean but unusual digits are consistently ranked below corrupted typical digits, the central claim is false.","supporting_citations":[],"review_version":1}