{"id":"84871e8c-3026-49c9-abb1-4a11226cf3af","arxiv_id":"1908.07387","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using random complementary labels as negative examples, then selectively applying positive learning to high-confidence samples, gives state-of-the-art accuracy on image classification with noisy labels.","lead":"The paper trains image classifiers by teaching them what an image is not, using randomly chosen wrong labels as negative examples, then selectively applies standard positive training to samples the model is confident about. This method, SelNLPL, filters out mislabeled data and reaches the best published accuracy on several noisy-label benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on a fixed confidence threshold (gamma=0.5) separating clean from noisy data; this separation is only demonstrated on CIFAR10 symmetric noise and breaks under the CIFAR100 and 80%-noise extensions, so the no-tuning claim lacks support.","rationale":"The reader's weakest-assumption correctly identifies the fixed gamma=0.5 threshold and the reliance on clean/noisy confidence separation as the most load-bearing step. I agree with that location and therefore keep the reader's CONDITIONAL verdict unchanged: the experiments are extensive and the component ablations are informative, but the absence of code, the lack of variance estimates, and the acknowledged failure at high symmetric-exclusive noise and the need for a modified CIFAR100 extension all prevent unconditional acceptance. I add a partial disagreement in that the concern is not only about threshold brittleness: the paper's motivation that NL 'provides correct information' is not fully sound, because for noisy examples the true label can be selected as a complementary label and be suppressed, while the given wrong label is never suppressed and is thus reinforced over time. This makes the clean/noisy separation an empirical observation rather than a guaranteed property, which is why the threshold sweep and histogram check are needed. The proposed check would settle whether the central claim holds beyond the specific CIFAR10 symmetric-noise setting shown in the paper.","tokens_in":12695,"tokens_out":11847,"duration_ms":118407,"concrete_test":"Run the SelNLPL pipeline on CIFAR10 with 40% and 60% asymmetric noise and with 50% symmetric-exclusive noise, sweeping gamma over {0.3, 0.4, 0.5, 0.6, 0.7} with three seeds per setting, and plot the clean/noisy confidence histograms after the NL-SelNL stage. If final accuracy varies by more than about 2% across gamma, or if the histograms overlap substantially at gamma=0.5 for any of these settings, the fixed threshold is a sensitive hyperparameter rather than a principled separator, and the no-tuning SOTA claim is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that SelNLPL achieves state-of-the-art noisy-label accuracy depends on the threshold assumption in Section 3.3 and Algorithm 2: after NL/SelNL, a sample is treated as clean if its given-label confidence py exceeds gamma=0.5, and as noisy otherwise. This single fixed threshold, with no noise-rate estimate, is what produces the filtered clean set and the pseudo-labels used in Section 3.5; if the confidence separation is not clean, the whole pipeline inherits mislabeled clean/noisy partitions. The paper's evidence for separation is limited to CIFAR10 histograms under 30% symmetric-inclusive noise (Figure 3); no histograms are shown for asymmetric noise, for 80% symmetric-exclusive noise (where the method fails to converge), or for CIFAR100, where Section 6.1 admits the base method cannot converge and replaces it with an ad hoc 110-complementary-label extension. Moreover, the motivating intuition that NL is 'safe' is incomplete: for a noisy example the true label is not excluded from the complementary-label set, so selecting it as ybar actively drives the true-class probability down, while the given wrong label is never selected and is implicitly reinforced. The clean/noisy gap is therefore an empirical property, not a guaranteed one, and it is exactly the property needed for the paper's no-tuning SOTA claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Negative Learning (NL), in which a CNN is trained with randomly drawn complementary labels using the loss -log(1-p_ybar), as a noise-robust alternative to Positive Learning (PL). The authors then combine NL with two selective stages called SelNL (keeping only samples whose given-label confidence exceeds 1/c) and SelPL (keeping only samples whose confidence exceeds gamma=0.5), producing SelNLPL. After training with SelNLPL, the network is used to divide the training set into clean and noisy parts, noisy labels are discarded, and the network is re-trained by pseudo-labeling. The paper reports state-of-the-art accuracy on CIFAR10, CIFAR100, FashionMNIST, and MNIST under symmetric and asymmetric label noise, and claims that the method requires no tuning based on prior knowledge of the noise type or noise rate.","tokens_in":13043,"tokens_out":3032,"duration_ms":32307,"significance":"If the claims are sustained, the paper makes a useful practical contribution: the core NL idea is simple, intuitively motivated, and is tested across four datasets, three noise types, and multiple noise levels. The ablation study in Table 7, the filtering precision/recall analysis in Table 1 and Figure 6, and the gradient analysis in Eq. 3 are valuable explanatory elements beyond a bare benchmark comparison. The claim of state-of-the-art accuracy without noise-rate-dependent hyper-parameters is attractive for real-world use. The main weaknesses are that the no-tuning claim is contradicted by the CIFAR100 extension and the 80% symmetric-exclusive failure, and that the fixed threshold gamma=0.5 is only visualized for one noise setting. The paper also reports no variance or error bars, so the magnitude of the claimed improvements (often 1-5%) cannot be assessed statistically.","major_comments":[{"comment":"The paper states that the method 'only failed to converge when the symm-exc noise is 80%, which can be neglected because such a scenario is unrealistic.' Since the claimed contribution is a method that works without prior knowledge of the noise level, dismissing a tested operating point where the method diverges undermines the claim. A practitioner who does not know the noise rate will not know that 80% is 'unrealistic.' Please either explain why the method fails at 80% symmetric-exclusive noise, provide evidence that the failure is specific to this synthetic setting, or revise the no-tuning claim.","section":"Section 5.2, Table 3"},{"comment":"The base SelNLPL method is stated to fail to converge on CIFAR100, and the authors introduce an ad hoc extension that computes 110 complementary-label losses per image. This introduces a new hyper-parameter (the number of complementary labels) that depends on the class count, which contradicts the contribution in Section 1 that the method 'does not require any tuning of hyper-parameters that depend on prior knowledge.' The number of complementary labels is a dataset-dependent choice, not a fixed parameter. Please state clearly that this is a separate variant with its own hyper-parameter, and adjust the no-tuning claim accordingly.","section":"Section 6.1"},{"comment":"The clean/noisy separation after NL/SelNL and the fixed threshold gamma=0.5 are the load-bearing elements of the filtering step, yet the histograms in Figure 3 are only shown for CIFAR10 with 30% symmetric-inclusive noise. No histograms are provided for asymmetric noise, for 80% symmetric-exclusive noise (where the method fails), or for the CIFAR100 extension. If the confidence separation is not clean in these settings, the filtering step and the subsequent pseudo-labeling are unreliable. Please provide confidence-separation evidence for these settings or perform a sensitivity analysis of gamma across noise types and levels.","section":"Section 3.3 and Algorithm 2"},{"comment":"The motivation that NL 'decreases the risk of providing incorrect information' is incomplete. For a noisy example whose given label is wrong, the true label is a member of the complementary-label set, and if it is selected as ybar, Eq. 2 actively drives the probability of the true class toward zero. Thus the safety of NL is an empirical property of the training dynamics, not a guaranteed consequence of random complementary labels. The text should present the 'safe' property as an empirical observation, not as a logical implication of the sampling scheme.","section":"Section 1 and Section 3.1"}],"minor_comments":[{"comment":"The reported results are single runs without variance, error bars, or statistical significance tests. Given that many of the claimed improvements are 1-5%, please report multiple seeds or at least indicate the variability.","section":"Section 5.2, Tables 3-6"},{"comment":"The learning rates differ across stages (0.02 for NL and SelNL, 0.1 for SelPL) and across datasets (MNIST uses 0.1 for NL and SelNL). This is reasonable engineering, but it weakens the statement that the method requires no tuning; please clarify which parameters are fixed and which are dataset-dependent.","section":"Section 5.1"},{"comment":"The ablation shows that omitting SelPL or SelNL from SelNLPL yields small accuracy drops at 30% symmetric-inclusive noise (1.38 and 0.41 points), but large drops at 50% noise. Please discuss whether the significance of each step depends on the noise level, and whether the 30% row genuinely supports the claim that each step contributes.","section":"Section 6.2, Table 7"},{"comment":"In the fourth paragraph of Section 4, 'This resulted in state-of-the-accuracy' appears to be a typo for 'state-of-the-art accuracy.'","section":"Section 4"},{"comment":"Algorithm 1 uses the symbol y for both the original label and the complementary label; please rename the complementary label (e.g., ybar) consistently with Eq. 2.","section":"Algorithm 1"},{"comment":"The description of pseudo-labeling says 'noisy data's label is updated with the output of the CNN...' but does not specify whether the soft label is computed from the network trained in Figure 5(b) or from the final network in Figure 5(c). Please clarify the exact two-stage re-training protocol.","section":"Section 3.5"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a strong empirical core and the ablation study is well designed, but the headline claims of 'state-of-the-art without tuning' and 'no prior knowledge' are not supported by the admitted failures at 80% symmetric-exclusive noise and by the ad hoc CIFAR100 extension. I would recommend the editor ask for a revision that softens these claims, adds variance information, and provides confidence-separation evidence for settings beyond CIFAR10 30% symmetric-inclusive noise. No code was released in the arXiv version; release of the training and evaluation scripts would materially increase confidence in the reproducibility of the numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: it takes an old idea (complementary labels) and puts it to work in the noisy-label setting with a clever twist. Instead of training on the given (possibly wrong) label, you train on \"this image is not class Y\" where Y is randomly chosen from the labels that aren't the given label. The idea is that the true label is unlikely to be chosen as the complementary label, so you mostly give the network correct information. Then they selectively apply positive learning to samples whose confidence exceeds 0.5 after the negative phase. The combination, SelNLPL, is simple and beats strong baselines on CIFAR10/100, FashionMNIST, and MNIST under symmetric and asymmetric noise.\n\nWhat's genuinely good: the empirical work is extensive — four datasets, three noise types, multiple noise levels, and a clean ablation (Table 7) that shows each step earns its keep. They also show that the method can estimate the noise rate accurately, which is practically useful. The idea of using negative learning for filtering is a real contribution, and the paper is honest about the main limitation (failure at 80% symmetric-exclusive noise) even if they wave it off as unrealistic.\n\nWhere I'd push back: the whole pipeline rests on a fixed confidence threshold (gamma=0.5) separating clean from noisy. The paper only demonstrates that separation with histograms for CIFAR10 under 30% symmetric-inclusive noise. It's plausible the threshold holds across other settings — the strong results suggest it does — but they don't show the histograms, and the CIFAR100 extension (110 random complementary labels per image) feels ad hoc. The \"no tuning\" claim is a bit strong: gamma, the 1/c threshold, and the 720-epoch per stage are all fixed choices, just not tuned per noise type. No code, no error bars, and the gradient analysis (Eq. 3) is illustrative, not a proof. These are addressable issues, not fatal ones.\n\nThe bottom line: this is a solid empirical paper with a memorable idea and lots of evidence. It deserves a serious referee, but it needs strengthening — more threshold sensitivity analysis, a few histograms for other noise types, and ideally code. I'd cite it if I worked on noisy labels, and I'd bring it to a reading group as a good example of a simple method with strong empirical support.\n\nRecommendation: send it to peer review, with the expectation of major revision.","headline":"Simple and effective noisy-label method with strong empirical results, but the threshold assumption and lack of code/variance are real soft spots.","tokens_in":13505,"tokens_out":2819,"would_cite":true,"duration_ms":28839,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a network to know what an image is not lets it learn from noisy labels.","keywords":["negative learning","complementary labels","noisy label classification","selective positive learning","semi-supervised learning","pseudo-labeling","label noise filtering","CIFAR-10"],"falsifier":"Train SelNLPL on a noise process engineered so some wrong labels become highly confident, for example asymmetric flips between visually similar classes at high noise, then inspect the confidence histograms: if the clean and noisy peaks overlap around $\\gamma = 0.5$, the fixed-threshold filter mislabels a nontrivial fraction and the method would have to fall back on noise-rate knowledge.","tokens_in":12516,"feed_emoji":"🎯","tokens_out":6596,"duration_ms":67681,"temperature":0.7,"pith_summary":"This paper is trying to establish that a classifier can be trained on badly labeled images by, at first, refusing to say what each image is rather than asserting what it is. The method, negative learning (NL), picks a random complementary label and trains the network to assign that label probability zero, which is almost always a true statement about the image and therefore safe even when the original label is wrong. The authors further claim that once NL has separated confident clean samples from low-confidence noisy ones, selectively applying ordinary positive learning to the confident half of the data (SelNLPL) filters out the noise and lets a simple semi-supervised pseudo-labeling step achieve leading test accuracy. Why it matters: if true, noisy-label training needs no knowledge of the noise rate, no clean validation data, and no specialized loss functions.","feed_headline":"Teaching CNNs what images are not beats noisy-label training","feed_subtitle":"A complementary-label loss stops overfitting, filters noisy samples, then trains on confident ones.","key_machinery":"The machinery is a complementary-label cross-entropy that drives the probability of a randomly chosen 'not this' label to zero. Because the softmax output sums to one, suppressing one class redistributes probability over the remaining classes, including, most of the time, the true class, so the network learns genuine visual content without being told a possibly false positive label. Two confidence gates then steer training: SelNL keeps samples with $p_y > 1/c$, and SelPL keeps samples with $p_y > \\gamma = 0.5$ for ordinary positive cross-entropy. The paper also derives that NL's gradient on the true class scales as $1/(c(c-1))$, which is why CIFAR100 needed 110 complementary losses per image to converge in the same epoch budget.","core_discovery":"The paper's central claim is that a CNN trained with negative learning (NL), whose loss is $$L(f,\\bar y) = -\\sum_{k=1}^{c} \\bar y_k \\log(1-p_k),$$ where $\\bar y$ is a uniformly random label different from the given label $y$, does not overfit to noisy labels: the confidence of clean samples climbs while the confidence of noisy samples stays low. The authors then add Selective Negative Learning (SelNL), which keeps only samples with confidence above $1/c$, and Selective Positive Learning (SelPL), which trains with the usual cross-entropy only on samples whose confidence exceeds $\\gamma = 0.5$. They claim that this SelNLPL schedule separates clean from noisy training data with high precision and recall, and that a simple pseudo-label semi-supervised stage on the filtered data reaches state-of-the-art accuracy on noisy CIFAR10, CIFAR100, MNIST, and FashionMNIST without any prior knowledge of the noise type or ratio.","pith_inferences":["Editorial inference: The fixed threshold $\\gamma = 0.5$ is a tuning choice, and the CIFAR100 modification suggests the clean/noisy confidence gap itself shifts with class count, so per-class or annealed thresholds could make the filter more portable.","Editorial inference: Because the method only needs a 'not this label' signal, it could generalize to web-scraped or weakly supervised data where errors are not simple class flips, and even to settings where some labels are plain misinformation.","Editorial inference: If the confidence gap is the mechanism, then the natural stress test is asymmetric noise between visually similar classes at high noise; the paper's own failure to converge at 80% symmetric-exclusive noise hints that the gap closes under extreme corruption."],"forward_implications":["Training with NL before any PL gives a confidence gap that can be used to estimate the noise rate of a training set: the fraction of samples below the 0.5 threshold tracks the true noise fraction to within a few percent in the paper's CIFAR10 experiments.","Because the method never consults a validation set and does not tune hyper-parameters to known noise levels, it is portable to settings where label quality is unknown in advance.","The filtering step is strong enough that the subsequent semi-supervised stage can be a plain pseudo-labeling procedure rather than a specially designed noise-robust loss.","Large class counts slow NL convergence by a factor related to $c(c-1)$; the authors' multi-complementary-label extension restores convergence on CIFAR100 with the same epoch budget as CIFAR10."],"supporting_citations":[{"why":"Supplies the Generalized Cross Entropy baselines and the experimental protocol for Table 3, where SelNLPL reports its main accuracy gains.","marker":"[39]"},{"why":"Defines the asymmetric noise mappings and supplies the Forward/Backward loss-correction baselines compared throughout the paper.","marker":"[25]"},{"why":"Supplies the Joint optimization baselines and the CIFAR10 protocol in Table 4, which is also the main ablation setting.","marker":"[30]"},{"why":"Supplies the D2L dimensionality-driven baseline and the MNIST protocol in Table 5.","marker":"[22]"},{"why":"Supplies the two-stage semi-supervised noisy-label baseline and the dataset protocols in Table 6.","marker":"[2]"},{"why":"Provides the pseudo-labeling semi-supervised technique used after SelNLPL filtering.","marker":"[19]"},{"why":"Provides the bootstrap loss baselines and the soft-label update idea used in the pseudo-labeling stage.","marker":"[26]"},{"why":"Prior work on training with complementary labels, which the paper extends by generating complementary labels from noisy labels.","marker":"[12]"}],"fun_headline_variants":["Negative learning prevents noisy-label overfitting","Tell CNNs what an image isn't, not what it is","Complementary labels filter noise then train only on clean","Smart sampling with negative labels beats noisy training","No noise info needed: NL then PL on confident samples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one fixed confidence threshold cleanly separates clean from noisy samples after negative learning - a separation the paper demonstrates empirically rather than proves - so the method never needs to know the noise ratio or type.","fun_headline_variants_meta":{"raw":{"variants":["Negative learning prevents noisy-label overfitting","Tell CNNs what an image isn't, not what it is","Complementary labels filter noise then train only on clean","Smart sampling with negative labels beats noisy training","No noise info needed: NL then PL on confident samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1580,"prompt_tokens":965,"completion_tokens":615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":540}},"tokens_in":581,"tokens_out":615,"duration_ms":6574,"temperature":1.0,"reasoning_tokens":540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:39:13.070463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SelNLPL on a noise process engineered so some wrong labels become highly confident, for example asymmetric flips between visually similar classes at high noise, then inspect the confidence histograms: if the clean and noisy peaks overlap around $\\gamma = 0.5$, the fixed-threshold filter mislabels a nontrivial fraction and the method would have to fall back on noise-rate knowledge.","supporting_citations":[{"cited_title":"Making deep neural networks robust to label noise: A loss correction approach","cited_arxiv_id":null,"evidence_quote":"Defines the asymmetric noise mappings and supplies the Forward/Backward loss-correction baselines compared throughout the paper."},{"cited_title":"Dimensionality-Driven Learning with Noisy Labels","cited_arxiv_id":"1806.02612","evidence_quote":"Supplies the D2L dimensionality-driven baseline and the MNIST protocol in Table 5."},{"cited_title":"A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels","cited_arxiv_id":"1802.02679","evidence_quote":"Supplies the two-stage semi-supervised noisy-label baseline and the dataset protocols in Table 6."},{"cited_title":"Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the pseudo-labeling semi-supervised technique used after SelNLPL filtering."},{"cited_title":"Learning from complementary labels","cited_arxiv_id":null,"evidence_quote":"Prior work on training with complementary labels, which the paper extends by generating complementary labels from noisy labels."}],"review_version":1}