{"id":"df775c5c-6ec5-4c67-8624-b8f463cf3972","arxiv_id":"2506.11584","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A benchmark shows Self-Influence finds mislabeled training samples, while all tested influence signals fail to detect clustered anomalies and outliers under cumulative TracIn scoring.","lead":"This paper tests four existing influence-based methods for spotting bad training data, using the same underlying influence estimator across image and tabular datasets. It finds that Self-Influence detects mislabeled samples well, but none of the tested methods detect anomalous or outlier samples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The blanket claim that no existing influence signal detects anomalies rests on an underspecified 'signal' definition: Appendix D shows per-epoch Self-Influence detects Near-CA on multi-class tabular data, so the negative result may be an artifact of cumulative TracIn aggregation.","rationale":"The reader's weakest_assumption identified the most load-bearing concern, and I agree with it. To establish 'none of the existing signals can detect anomalies', the authors must show that the negative result is invariant to reasonable aggregations of TracIn scores over epochs; instead, Appendix D demonstrates that it is not invariant at all, because per-epoch Self-Influence detects Near-CA samples on multi-class tabular datasets. The positive result that SI detects uniform label noise is plausible and supported by multiple datasets, models, and five random seeds, though the paper lacks error bars. The issue is conceptual rather than statistical: the scope of the negative universal claim is narrower than stated. The reader's CONDITIONAL verdict remains appropriate because the paper can be revised by adding a formal definition of 'signal' and qualifying the headline to cumulative TracIn; there is no need to move to REJECT, since the benchmark of cumulative signals and the documented influence-cancellation effect are still useful contributions. A secondary concern about the last-layer-only TracIn approximation is real but less directly evidenced by the manuscript itself, so I focus on the Appendix D tension as the single most direct threat to the central claim.","tokens_in":11747,"tokens_out":6300,"duration_ms":63126,"concrete_test":"Run an Appendix D-style epoch-wise evaluation across all datasets and glitch types: for each of the four signals, compute F1 at each epoch from per-epoch TracIn scores (not cumulative) and record the maximum over epochs. If any signal achieves non-trivial F1 (say >0.5) at any epoch for Near-CA, Far-CA, or outliers, the abstract's 'none of the existing signals can detect anomalies' must be re-scoped to cumulative TracIn. Conversely, if no per-epoch signal exceeds baseline on Far-CA/outliers, the cumulative-aggregation caveat is sufficient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central negative conclusion—that 'none of the existing signals can detect anomalies' (Abstract; Section 6)—is tested only for one epoch-aggregation choice: cumulative TracIn scores. Section 3 defines TracIn as a sum over epochs, and Sections 5 and Appendix B use that cumulative score. The boundary between 'signal' and 'estimator' is never defined precisely enough to exclude per-epoch scores. Appendix D explicitly computes per-epoch Self-Influence without the cumulative TracIn aggregation and reports that on Jannis the first epoch spots all Near-CA samples, with substantially better performance on Forest Cover than the cumulative aggregation (F1 ~0.6 vs ~0). This is the same SI signal whose cumulative version was used to claim failure. So the headline statement is either false if per-epoch SI counts as an existing signal, or uninformatively narrow if 'signal' is stipulated to include the cumulative epoch sum. The paper must either formally define 'signal' so that epoch-wise application is excluded and justify that exclusion, or scope the claim to cumulative TracIn. As written, the negative universal claim is not supported by the paper's own evidence, and this is load-bearing because the paper's novelty is precisely that negative result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of four influence-based signals for data debugging—Self-Influence, Average Absolute Influence, Marginal Influence, and GD-class—all computed with the TracIn influence estimator on tabular and image datasets using both from-scratch and foundation models. The authors conclude that Self-Influence is effective for detecting uniform class noise, but that none of the existing signals can detect clustered anomalies (Near-CA, Far-CA) or outliers under cumulative TracIn aggregation. They also discuss influence cancellation and argue that new signals should exploit training dynamics. The paper includes an appendix showing that per-epoch Self-Influence can detect Near-CA samples in multiclass tabular datasets, which complicates the paper's blanket negative claim about anomaly detection.","tokens_in":12007,"tokens_out":4581,"duration_ms":45592,"significance":"If the negative result is scoped and stated carefully, the paper makes a useful contribution as a systematic comparative benchmark of influence signals for data debugging. The code is available, the contamination protocols are clearly described, and the evaluation spans multiple data modalities and model types, including foundation models. The paper also provides a concrete demonstration of the influence-cancellation effect and highlights the potential value of training dynamics, which is a plausible direction for future work. However, the central negative claim about anomaly detection needs to be reconciled with the paper's own per-epoch results, and several experimental-reporting gaps weaken the strength of the conclusions. The study is a reasonable initial benchmark rather than a definitive negative result, and the manuscript would benefit from a revised claim and fuller reporting of variability and missing conditions.","major_comments":[{"comment":"The Abstract and Section 6 state that 'none of the existing signals can detect anomalies,' but Appendix D (Figure 5) reports that Self-Influence computed per epoch, without the cumulative TracIn aggregation, spots all Near-CA samples on Jannis with FT-Transformer in the first epoch and achieves substantially higher F1 on Forest Cover than the cumulative result shown in Figure 4b. Since Self-Influence is one of the four existing signals evaluated in the paper, the universal negative claim is contradicted by the paper's own evidence unless 'signal' is defined narrowly to exclude per-epoch scores. The authors should either provide a formal definition of 'signal' that excludes per-epoch application and justify that exclusion, or restrict the conclusion explicitly to the cumulative TracIn aggregation.","section":"Abstract; Section 6; Appendix D"},{"comment":"The paper lists class-dependent noise as one of the glitch types studied and Section 4 describes how it is injected, but no figure or table reports detection results for class-dependent noise. Figures 1 and 4 cover uniform class noise and singular mislabeled classes, but not the class-dependent (asymmetric) noise defined in Section 2.1. The claimed evaluation across 'different glitch types' is therefore incomplete, and the paper should either present the missing results or explicitly state that class-dependent noise was not evaluated in the reported experiments.","section":"Section 4; Section 5; Figures 1 and 4"},{"comment":"The paper states that each experiment was repeated five times with different random seeds, but all figures report only the average F1-Score, with no standard deviations, confidence intervals, or significance tests. Given the small number of repetitions, the reliability of the observed differences among signals—and the strength of the negative conclusions—cannot be assessed. The authors should include measures of variability or per-seed results for at least the headline findings.","section":"Section 5; Figures 1, 2, and 4"}],"minor_comments":[{"comment":"The rows in Table 1 appear to have the Near-CA and Far-CA labels swapped: the image rows are labeled 'Near-CA' with accuracy 1, while the text and Section 5 indicate that image models classify Far-CA samples perfectly; the tabular rows are labeled 'Far-CA' with accuracy near 0, while the text says the tabular model fails on Near-CA. Please correct the table labels to match the described experiment.","section":"Appendix C, Table 1"},{"comment":"Appendix B ends with the incomplete sentence 'Finally, for the Far-CA samples,' immediately before Appendix C begins. This sentence should be completed or removed.","section":"Appendix B"},{"comment":"The evaluation uses the known glitch ratio to set the F1 threshold. This is a reasonable choice for a controlled benchmark, but it should be stated more prominently as a limitation because in real deployment the glitch ratio and the appropriate threshold are generally unknown.","section":"Section 4; Appendix A"},{"comment":"The per-epoch results in Figure 5 are reported only for FT-Transformer; the text says each model was trained for 10 epochs, but no per-epoch results are shown for ResNet or MLP. Adding them would strengthen the claim that the early-epoch effect is model-independent.","section":"Appendix D, Figure 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things you should know. First, the paper does something useful: it is the first benchmark comparing four influence signals (SI, AAI, MI, GD-class) under a single estimator (TracIn) across tabular and image data, including from-scratch and foundation models, and it tests glitch types that were not jointly covered before: class-dependent noise, Near-CA, Far-CA, and outliers. Code is public, and experiments are repeated five times, though only averages are reported.\n\nSecond, the main negative result is plausible but overstated. The abstract says \"none of the existing signals can detect anomalies.\" What the experiments actually show is that cumulative TracIn aggregations fail on Near-CA (multiclass), Far-CA, and outliers. Appendix D then shows that per-epoch Self-Influence spots all Near-CA samples in Jannis at epoch 1 and reaches around 0.6 F1 on Forest Cover, both well above the cumulative version. So the boundary between \"signal\" and \"estimator\" is doing a lot of work. If per-epoch SI counts as an existing signal, the claim is false; if the authors intended \"signal\" to include the epoch sum, they need to say so and justify it. As written, the universal claim is not supported by their own evidence.\n\nThird, the smaller issues are real but minor: no error bars or significance tests; the F1 threshold is set using the known glitch ratio, which is fine for a controlled benchmark but optimistic for real debugging; and Table 1 in Appendix C seems to swap Near-CA and Far-CA labels (image models get 1.0 on \"Near-CA\" while the text says the opposite).\n\nOverall, this is a solid empirical contribution that needs a revision of scope rather than new experiments. The per-epoch result is already inside the paper; the authors just need to reconcile it with the abstract. The discussion of training dynamics and sign-wise aggregation is a sensible pointer for future work.\n\nFor a reader, this is worth a look if you work on data-centric ML or influence functions. I would accept it for peer review; a good referee would push on the signal-definition issue and ask for confidence intervals. I'd cite it as evidence that cumulative TracIn is not the right aggregation for anomaly detection.","headline":"Useful benchmark with public code, but the headline claim that no signal detects anomalies is overscoped—the paper's own Appendix D shows per-epoch Self-Influence finds Near-CA on tabular data, so the negative result is really about cumulative TracIn aggregation.","tokens_in":12530,"tokens_out":2153,"would_cite":true,"duration_ms":20005,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-Influence catches mislabeled samples, but no influence signal catches anomalies.","keywords":["Data Debugging","Influence Functions","TracIn","Self-Influence","Label Noise","Anomaly Detection","Training Dynamics","Data Glitches"],"falsifier":"Re-run the Near-CA benchmark on Jannis and Forest Cover using Self-Influence computed from a single early epoch instead of summed across all epochs and report the F1-score; a high F1 would refute the claim that no existing influence signal can detect clustered anomalies.","tokens_in":11553,"feed_emoji":"🔍","tokens_out":11626,"duration_ms":102659,"temperature":0.7,"pith_summary":"To test whether influence-based signals can serve as a one-stop data debugger, the paper performs the first comparative evaluation of four signals—Self-Influence, Average Absolute Influence, Marginal Influence, and GD-class—under the same TracIn influence estimator, across tabular and image datasets and models trained from scratch or fine-tuned from foundation models. The central question is whether these signals can automatically flag both mislabeled samples and anomalies such as rare-class clusters, far-distribution clusters, and corrupted outliers. The answer the paper defends is partial: Self-Influence reliably detects uniform class noise, especially with foundation models, but none of the four signals, when scores are aggregated over all training epochs, reliably detects near-cluster anomalies in multi-class tabular data, far-cluster anomalies in images, or outliers. The paper attributes these failures to two mechanism-level causes: existing aggregations ignore training dynamics, and some signed aggregations cancel out and yield near-zero influence. A correct reading of the results, the paper argues, is that influence-based debugging should keep detecting label errors but needs a new class of signals built from per-epoch influence.","feed_headline":"Self-Influence finds bad labels, misses anomalies","feed_subtitle":"A four-signal benchmark shows which data glitches influence scores can catch and which need new signals.","key_machinery":"The machinery is TracIn, a gradient-tracing influence estimator that approximates how much one training sample changes another sample's loss by accumulating inner products of loss gradients across training epochs, sign included. Four aggregation signals turn TracIn scores into ranking scores: Self-Influence (a sample's influence on its own loss), Marginal Influence (signed sum of a sample's influence over a clean validation set), Average Absolute Influence (mean absolute influence over the validation set), and GD-class (influence aggregated only over same-class validation samples). The signals are what the benchmark evaluates; the cumulative versus per-epoch choice inside TracIn is the load-bearing design decision, because cumulative sums are what wash out anomaly signatures while per-epoch scores reveal them.","core_discovery":"On uniform class noise, Self-Influence—a sample's influence on its own loss—achieves the highest detection F1 of the four signals, and its performance grows as the noise ratio increases but drops when only a single class is mislabeled. Marginal Influence performs worst because its signed summation lets positive and negative contributions cancel toward zero; Average Absolute Influence avoids that cancellation but does not beat Self-Influence; GD-class, though it conditions on class information, is not enough. The paper's central negative result is that cumulative TracIn scores, aggregated over all training epochs, give every one of the four signals near-zero F1 on Near-Clustered Anomalies in multi-class tabular datasets, on Far-Clustered Anomalies in image datasets, and on corruption-based outliers. The paper then shows the failure is not baked into the influence estimates: in its per-epoch analysis, Self-Influence in the first training epochs detects Near-CA samples on the Jannis and Forest Cover datasets with much higher F1 than the cumulative aggregation, which the authors take as evidence that training dynamics are the missing ingredient in current signal definitions.","pith_inferences":["The paper's widely stated conclusion that no existing signal detects anomalies is scoped to cumulative TracIn aggregations; if per-epoch Self-Influence counts as a signal, the paper's own Appendix D already contradicts the blanket version.","A concrete next step implicit in the results is to define an explicit early-epoch Self-Influence signal and benchmark it against dedicated anomaly detectors on tabular and image data; the appendix suggests it would be competitive for Near-CA.","The cancellation diagnosis points to a design principle beyond the paper's tests: replacing signed cumulative sums with per-epoch maxima or absolute gradient products could reorder which samples rank highest, a prediction directly testable by re-running the authors' benchmark with modified aggregations.","If training dynamics carry the anomaly signal, influence-based debugging could be used as a training-time monitor that flags suspicious samples in the first epochs, rather than only after the model has converged."],"forward_implications":["Self-Influence is the strongest of the four current signals for spotting uniform label noise, so practitioners debugging label errors should prefer it over Marginal Influence, Average Absolute Influence, and GD-class.","Current influence signals cannot replace dedicated anomaly detectors for rare-class clusters, far-distribution clusters, or corrupted outliers when aggregated over full training.","New influence signals should be sign-aware in their aggregations, since signed summation produces cancellation and misleading zero scores.","Training dynamics belong in signal definitions: per-epoch or early-epoch influence scores catch near-cluster anomalies that cumulative TracIn scores miss.","Small noise ratios and single-class mislabeling weaken Self-Influence, so label-debugging pipelines should not rely on it alone in those regimes."],"supporting_citations":[{"why":"Supplies the TracIn estimator that all four influence signals in the benchmark are computed with.","marker":"(Pruthi et al., 2020)"},{"why":"Introduces influence functions and the Self-Influence signal for detecting mislabeled training samples.","marker":"(Koh and Liang, 2017)"},{"why":"Defines the Average Absolute Influence signal and its use for data cleansing.","marker":"(Hara et al., 2019)"},{"why":"Defines the Marginal Influence signal that aggregates a sample's influence over a validation set.","marker":"(Kong et al., 2021)"},{"why":"Defines the GD-class conditional influence signal used as one of the four compared signals.","marker":"(Nguyen-Duc et al., 2023)"},{"why":"Provides the tabular benchmarks and the FT-Transformer, ResNet, and MLP models evaluated from scratch.","marker":"(Gorishniy et al., 2021)"},{"why":"Supplies the MNIST-C corruptions used to inject outlier glitches into image training sets.","marker":"(Mu and Gilmer, 2019)"},{"why":"Supplies the CIFAR-10-C corruptions used to inject outlier glitches into image training sets.","marker":"(Hendrycks and Dietterich, 2018)"}],"fun_headline_variants":["Self-Influence catches bad labels, misses anomalies","Cumulative influence scores fail on data anomalies","Training dynamics key to spotting data anomalies","Influence signals need dynamics for anomaly detection","Anomalies invisible to existing influence signals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that no influence signal can detect anomalies depends on defining a signal as the cumulative TracIn score summed over all training epochs; if per-epoch Self-Influence is allowed, the paper's own appendix shows near-cluster anomalies being detected.","fun_headline_variants_meta":{"raw":{"variants":["Self-Influence catches bad labels, misses anomalies","Cumulative influence scores fail on data anomalies","Training dynamics key to spotting data anomalies","Influence signals need dynamics for anomaly detection","Anomalies invisible to existing influence signals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":2995,"prompt_tokens":974,"completion_tokens":2021,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1954}},"tokens_in":590,"tokens_out":2021,"duration_ms":13693,"temperature":1.0,"reasoning_tokens":1954,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:52.174846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Near-CA benchmark on Jannis and Forest Cover using Self-Influence computed from a single early epoch instead of summed across all epochs and report the F1-score; a high F1 would refute the claim that no existing influence signal can detect clustered anomalies.","supporting_citations":[],"review_version":1}