A Comparative Analysis of Influence Signals for Data Debugging

Ioannis Tsamardinos; Nikolaos Myrtakis; Vassilis Christophides

REVIEW 3 major objections 4 minor 14 references

A Comparative Analysis of Influence Signals for Data Debugging

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Self-Influence catches mislabeled samples, but no influence signal catches anomalies.

desk verdict Useful benchmark with public code, but the headline claim that no signal detects anomalies is overscoped—the paper's own Appendix D shows per-epoch Self-Influence finds Near-CA on tabular data, so the negative result is really about cumulative TracIn aggregation. read the letter →

arxiv 2506.11584 v1 pith:UZQJPMSF submitted 2025-06-13 cs.LG cs.AI

Nikolaos Myrtakis , Ioannis Tsamardinos , Vassilis Christophides This is my paper

classification cs.LGcs.AI

keywords DataDebuggingInfluenceFunctionsTracInSelf-InfluenceLabelNoiseAnomalyDetectionTrainingDynamicsGlitches

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

To test whether influence-based signals can serve as a one-stop data debugger, the paper performs the first comparative evaluation of four signals—Self-Influence, Average Absolute Influence, Marginal Influence, and GD-class—under the same TracIn influence estimator, across tabular and image datasets and models trained from scratch or fine-tuned from foundation models. The central question is whether these signals can automatically flag both mislabeled samples and anomalies such as rare-class clusters, far-distribution clusters, and corrupted outliers. The answer the paper defends is partial: Self-Influence reliably detects uniform class noise, especially with foundation models, but none of the four signals, when scores are aggregated over all training epochs, reliably detects near-cluster anomalies in multi-class tabular data, far-cluster anomalies in images, or outliers. The paper attributes these failures to two mechanism-level causes: existing aggregations ignore training dynamics, and some signed aggregations cancel out and yield near-zero influence. A correct reading of the results, the paper argues, is that influence-based debugging should keep detecting label errors but needs a new class of signals built from per-epoch influence.

What carries the argument

The machinery is TracIn, a gradient-tracing influence estimator that approximates how much one training sample changes another sample's loss by accumulating inner products of loss gradients across training epochs, sign included. Four aggregation signals turn TracIn scores into ranking scores: Self-Influence (a sample's influence on its own loss), Marginal Influence (signed sum of a sample's influence over a clean validation set), Average Absolute Influence (mean absolute influence over the validation set), and GD-class (influence aggregated only over same-class validation samples). The signals are what the benchmark evaluates; the cumulative versus per-epoch choice inside TracIn is the load-bearing design decision, because cumulative sums are what wash out anomaly signatures while per-epoch scores reveal them.

What would settle it

Re-run the Near-CA benchmark on Jannis and Forest Cover using Self-Influence computed from a single early epoch instead of summed across all epochs and report the F1-score; a high F1 would refute the claim that no existing influence signal can detect clustered anomalies.

Watch

Extended reading notes

Core claim

On uniform class noise, Self-Influence—a sample's influence on its own loss—achieves the highest detection F1 of the four signals, and its performance grows as the noise ratio increases but drops when only a single class is mislabeled. Marginal Influence performs worst because its signed summation lets positive and negative contributions cancel toward zero; Average Absolute Influence avoids that cancellation but does not beat Self-Influence; GD-class, though it conditions on class information, is not enough. The paper's central negative result is that cumulative TracIn scores, aggregated over all training epochs, give every one of the four signals near-zero F1 on Near-Clustered Anomalies in multi-class tabular datasets, on Far-Clustered Anomalies in image datasets, and on corruption-based outliers. The paper then shows the failure is not baked into the influence estimates: in its per-epoch analysis, Self-Influence in the first training epochs detects Near-CA samples on the Jannis and Forest Cover datasets with much higher F1 than the cumulative aggregation, which the authors take as evidence that training dynamics are the missing ingredient in current signal definitions.

Load-bearing premise

The conclusion that no influence signal can detect anomalies depends on defining a signal as the cumulative TracIn score summed over all training epochs; if per-epoch Self-Influence is allowed, the paper's own appendix shows near-cluster anomalies being detected.

Editorial extensions

If this is right

Self-Influence is the strongest of the four current signals for spotting uniform label noise, so practitioners debugging label errors should prefer it over Marginal Influence, Average Absolute Influence, and GD-class.
Current influence signals cannot replace dedicated anomaly detectors for rare-class clusters, far-distribution clusters, or corrupted outliers when aggregated over full training.
New influence signals should be sign-aware in their aggregations, since signed summation produces cancellation and misleading zero scores.
Training dynamics belong in signal definitions: per-epoch or early-epoch influence scores catch near-cluster anomalies that cumulative TracIn scores miss.
Small noise ratios and single-class mislabeling weaken Self-Influence, so label-debugging pipelines should not rely on it alone in those regimes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's widely stated conclusion that no existing signal detects anomalies is scoped to cumulative TracIn aggregations; if per-epoch Self-Influence counts as a signal, the paper's own Appendix D already contradicts the blanket version.
A concrete next step implicit in the results is to define an explicit early-epoch Self-Influence signal and benchmark it against dedicated anomaly detectors on tabular and image data; the appendix suggests it would be competitive for Near-CA.
The cancellation diagnosis points to a design principle beyond the paper's tests: replacing signed cumulative sums with per-epoch maxima or absolute gradient products could reorder which samples rank highest, a prediction directly testable by re-running the authors' benchmark with modified aggregations.
If training dynamics carry the anomaly signal, influence-based debugging could be used as a training-time monitor that flags suspicious samples in the first epochs, rather than only after the model has converged.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Three things you should know. First, the paper does something useful: it is the first benchmark comparing four influence signals (SI, AAI, MI, GD-class) under a single estimator (TracIn) across tabular and image data, including from-scratch and foundation models, and it tests glitch types that were not jointly covered before: class-dependent noise, Near-CA, Far-CA, and outliers. Code is public, and experiments are repeated five times, though only averages are reported.

Second, the main negative result is plausible but overstated. The abstract says "none of the existing signals can detect anomalies." What the experiments actually show is that cumulative TracIn aggregations fail on Near-CA (multiclass), Far-CA, and outliers. Appendix D then shows that per-epoch Self-Influence spots all Near-CA samples in Jannis at epoch 1 and reaches around 0.6 F1 on Forest Cover, both well above the cumulative version. So the boundary between "signal" and "estimator" is doing a lot of work. If per-epoch SI counts as an existing signal, the claim is false; if the authors intended "signal" to include the epoch sum, they need to say so and justify it. As written, the universal claim is not supported by their own evidence.

Third, the smaller issues are real but minor: no error bars or significance tests; the F1 threshold is set using the known glitch ratio, which is fine for a controlled benchmark but optimistic for real debugging; and Table 1 in Appendix C seems to swap Near-CA and Far-CA labels (image models get 1.0 on "Near-CA" while the text says the opposite).

Overall, this is a solid empirical contribution that needs a revision of scope rather than new experiments. The per-epoch result is already inside the paper; the authors just need to reconcile it with the abstract. The discussion of training dynamics and sign-wise aggregation is a sensible pointer for future work.

For a reader, this is worth a look if you work on data-centric ML or influence functions. I would accept it for peer review; a good referee would push on the signal-definition issue and ask for confidence intervals. I'd cite it as evidence that cumulative TracIn is not the right aggregation for anomaly detection.

Referee Report

3 major / 4 minor

Summary. The paper reports an empirical comparison of four influence-based signals for data debugging—Self-Influence, Average Absolute Influence, Marginal Influence, and GD-class—all computed with the TracIn influence estimator on tabular and image datasets using both from-scratch and foundation models. The authors conclude that Self-Influence is effective for detecting uniform class noise, but that none of the existing signals can detect clustered anomalies (Near-CA, Far-CA) or outliers under cumulative TracIn aggregation. They also discuss influence cancellation and argue that new signals should exploit training dynamics. The paper includes an appendix showing that per-epoch Self-Influence can detect Near-CA samples in multiclass tabular datasets, which complicates the paper's blanket negative claim about anomaly detection.

Significance. If the negative result is scoped and stated carefully, the paper makes a useful contribution as a systematic comparative benchmark of influence signals for data debugging. The code is available, the contamination protocols are clearly described, and the evaluation spans multiple data modalities and model types, including foundation models. The paper also provides a concrete demonstration of the influence-cancellation effect and highlights the potential value of training dynamics, which is a plausible direction for future work. However, the central negative claim about anomaly detection needs to be reconciled with the paper's own per-epoch results, and several experimental-reporting gaps weaken the strength of the conclusions. The study is a reasonable initial benchmark rather than a definitive negative result, and the manuscript would benefit from a revised claim and fuller reporting of variability and missing conditions.

major comments (3)

[Abstract; Section 6; Appendix D] The Abstract and Section 6 state that 'none of the existing signals can detect anomalies,' but Appendix D (Figure 5) reports that Self-Influence computed per epoch, without the cumulative TracIn aggregation, spots all Near-CA samples on Jannis with FT-Transformer in the first epoch and achieves substantially higher F1 on Forest Cover than the cumulative result shown in Figure 4b. Since Self-Influence is one of the four existing signals evaluated in the paper, the universal negative claim is contradicted by the paper's own evidence unless 'signal' is defined narrowly to exclude per-epoch scores. The authors should either provide a formal definition of 'signal' that excludes per-epoch application and justify that exclusion, or restrict the conclusion explicitly to the cumulative TracIn aggregation.
[Section 4; Section 5; Figures 1 and 4] The paper lists class-dependent noise as one of the glitch types studied and Section 4 describes how it is injected, but no figure or table reports detection results for class-dependent noise. Figures 1 and 4 cover uniform class noise and singular mislabeled classes, but not the class-dependent (asymmetric) noise defined in Section 2.1. The claimed evaluation across 'different glitch types' is therefore incomplete, and the paper should either present the missing results or explicitly state that class-dependent noise was not evaluated in the reported experiments.
[Section 5; Figures 1, 2, and 4] The paper states that each experiment was repeated five times with different random seeds, but all figures report only the average F1-Score, with no standard deviations, confidence intervals, or significance tests. Given the small number of repetitions, the reliability of the observed differences among signals—and the strength of the negative conclusions—cannot be assessed. The authors should include measures of variability or per-seed results for at least the headline findings.

minor comments (4)

[Appendix C, Table 1] The rows in Table 1 appear to have the Near-CA and Far-CA labels swapped: the image rows are labeled 'Near-CA' with accuracy 1, while the text and Section 5 indicate that image models classify Far-CA samples perfectly; the tabular rows are labeled 'Far-CA' with accuracy near 0, while the text says the tabular model fails on Near-CA. Please correct the table labels to match the described experiment.
[Appendix B] Appendix B ends with the incomplete sentence 'Finally, for the Far-CA samples,' immediately before Appendix C begins. This sentence should be completed or removed.
[Section 4; Appendix A] The evaluation uses the known glitch ratio to set the F1 threshold. This is a reasonable choice for a controlled benchmark, but it should be stated more prominently as a limitation because in real deployment the glitch ratio and the appropriate threshold are generally unknown.
[Appendix D, Figure 5] The per-epoch results in Figure 5 are reported only for FT-Transformer; the text says each model was trained for 10 epochs, but no per-epoch results are shown for ResNet or MLP. Adding them would strengthen the claim that the early-epoch effect is model-independent.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: empirical evaluation with independently defined prior-work signals; one minor self-citation is not load-bearing.

full rationale

This is an empirical benchmark study, not a derivation. The four influence signals (Self-Influence, Average Absolute Influence, Marginal Influence, GD-class) are defined in prior work and evaluated here under a common TracIn estimator; no parameter is fitted to a subset and then renamed as a prediction, and no claimed result reduces by construction to its own inputs. The only self-citation (Ntroumpogiannis et al., 2022) supports the tabular anomaly-injection convention of downsampling a class to create a rare-event distribution; this is a methodological detail and is not load-bearing for the paper's main findings. Appendix D's per-epoch Self-Influence results are explicitly presented as evidence about training dynamics, which the paper identifies as an open direction rather than as an existing signal, and the conclusion is correspondingly hedged ('when training dynamics are considered, SI can detect Near-CA samples even for multi-class datasets'). The broad phrasing that no existing signal detects anomalies may be overbroad relative to the cumulative-TracIn evaluation, but that is a scope and correctness concern, not circularity. The low score reflects only the presence of the minor, non-load-bearing self-citation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical comparison. No fitted parameters are used; the listed free parameters are experimental choices. The axioms are the modeling assumptions needed for TracIn and the anomaly protocol. No new entities are introduced.

free parameters (2)

Glitch ratio (epsilon) = 0.10 in main experiments, 0.01-0.30 in ablation
Chosen by hand to create controlled contamination; the paper's conclusions depend on this range and on the known ratio.
F1 threshold = set to known glitch ratio
The evaluation assumes the contamination rate is known, giving signals optimal thresholding; a real deployment would not have this information.

assumptions (4)

domain assumption The validation set is clean for joint signals (MI, AAI, GD-class).
Section 4 states joint signals assume clean validation samples, following prior benchmarking efforts.
domain assumption TracIn's assumptions hold: SGD without momentum, checkpoints saved per epoch, and influence computed only from the last layer weights.
Section 4 and the IF paragraph state these choices to match TracIn's requirements and prior work on last-layer approximation.
domain assumption Anomaly injection via downsampling a class (tabular) or splicing samples from another dataset with random labels (image) represents realistic anomalies.
Section 4; the negative results on anomalies are conditional on this injection protocol.
domain assumption Glitched samples receive higher signal scores, so ranking by descending score is the correct decision rule.
Section 3 and Appendix A; all signals assume higher scores indicate glitches.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Comparative Analysis of Influence Signals for Data Debugging." pith.science (2026). https://pith.science/paper/UZQJPMSF

@misc{pith2026250611584,
  author       = {Pith},
  title        = {Pith review of: A Comparative Analysis of Influence Signals for Data Debugging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZQJPMSF}},
  note         = {Machine review of arXiv:2506.11584}
}

read the original abstract

Improving the quality of training samples is crucial for improving the reliability and performance of ML models. In this paper, we conduct a comparative evaluation of influence-based signals for debugging training data. These signals can potentially identify both mislabeled and anomalous samples from a potentially noisy training set as we build the models and hence alleviate the need for dedicated glitch detectors. Although several influence-based signals (e.g., Self-Influence, Average Absolute Influence, Marginal Influence, GD-class) have been recently proposed in the literature, there are no experimental studies for assessing their power in detecting different glitch types (e.g., mislabeled and anomalous samples) under a common influence estimator (e.g., TraceIn) for different data modalities (image and tabular), and deep learning models (trained from scratch or foundation). Through extensive experiments, we show that signals like Self-Influence effectively detect mislabeled samples, but none of the existing signals can detect anomalies. Existing signals do not take into account the training dynamics, i.e., how the samples' influence on the model changes during training, while some signals fall into influence cancellation effects, i.e., influence score is zero due to unsigned scores accumulation, resulting in misleading influence attribution.

Figures

Figures reproduced from arXiv: 2506.11584 by the authors.

**Figure 2.** Detection performance of influence-based signals for mislabeled (mis.) samples, Far [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Evaluation Pipeline of Influence-based Signals [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Data glitch detection performance of influence signals w.r.t. increasing ratio of uni [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: Detection performance of influence-based signals for Near-CA reported per epoch when [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages

[1]

Abdelaal, C

M. Abdelaal, C. Hammacher, and H. Schoening. Rein: A comprehensive benchmark framework for data cleaning methods in ml pipelines. arXiv preprint arXiv:2302.04702,

arXiv
[8]

P. Li, X. Rao, J. Blase, Y . Zhang, X. Chu, and C. Zhang. Cleanml: A study for evaluating the impact of data cleaning on ml classification tasks. 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 13–24,

work page 2021
[9]

W. Li, L. Wang, W. Li, E. Agustsson, and L. Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862,

arXiv
[13]

Thimonier, F

H. Thimonier, F. Popineau, A. Rimmel, B.-L. Doan, and F. Daniel. Tracinad: measuring influence for anomaly detection. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1–6. IEEE,

work page 2022
[14]

S. E. Whang, Y . Roh, H. Song, and J.-G. Lee. Data collection and quality challenges in deep learning: A data-centric ai perspective. CoRR, 2112.06409,

arXiv
[1974]

X. Han, B. C. Wallace, and Y . Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint arXiv:2005.06676,

arXiv 2005
[2000]

Dosovitskiy, L

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021,

work page 2021
[2016]

Hendrycks and T

D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corrup- tions and surface variations. arXiv preprint arXiv:1807.01697,

arXiv

Show all 14 references

[2017]

J. Liu, Z. Shen, Y . He, X. Zhang, R. Xu, H. Yu, and P. Cui. Towards out-of-distribution generaliza- tion: A survey. arXiv preprint arXiv:2108.13624,

arXiv
[2018]

T. Dasu, T. Johnson, and E. Koutsofios. Hunting down glitches in massive time series data. In B. D. Klein and D. F. Rossin, editors, Fifth Conference on Information Quality (IQ 2000), pages 190–199. MIT,

2000
[2021]

Hammoudeh and D

Z. Hammoudeh and D. Lowd. Training data influence analysis and estimation: A survey. arXiv preprint arXiv:2212.04612,

arXiv
[2022]

Barshan, M.-E

E. Barshan, M.-E. Brunet, and G. K. Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pages 1899–1909. PMLR,

1909
[2023]

Ntroumpogiannis, M

A. Ntroumpogiannis, M. Giannoulis, N. Myrtakis, V . Christophides, E. Simon, and I. Tsamardinos. A meta-level analysis of online anomaly detectors. CoRR, 2209.05899,

arXiv
[2024]

Mu and J

10 A ComparativeAnalysis of Influence Signals for DataDebugging N. Mu and J. Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv:1906.02337,

1906 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Abdelaal, C

M. Abdelaal, C. Hammacher, and H. Schoening. Rein: A comprehensive benchmark framework for data cleaning methods in ml pipelines. arXiv preprint arXiv:2302.04702,

arXiv

[2] [8]

P. Li, X. Rao, J. Blase, Y . Zhang, X. Chu, and C. Zhang. Cleanml: A study for evaluating the impact of data cleaning on ml classification tasks. 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 13–24,

work page 2021

[3] [9]

W. Li, L. Wang, W. Li, E. Agustsson, and L. Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862,

arXiv

[4] [13]

Thimonier, F

H. Thimonier, F. Popineau, A. Rimmel, B.-L. Doan, and F. Daniel. Tracinad: measuring influence for anomaly detection. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1–6. IEEE,

work page 2022

[5] [14]

S. E. Whang, Y . Roh, H. Song, and J.-G. Lee. Data collection and quality challenges in deep learning: A data-centric ai perspective. CoRR, 2112.06409,

arXiv

[6] [1974]

X. Han, B. C. Wallace, and Y . Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint arXiv:2005.06676,

arXiv 2005

[7] [2000]

Dosovitskiy, L

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021,

work page 2021

[8] [2016]

Hendrycks and T

D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corrup- tions and surface variations. arXiv preprint arXiv:1807.01697,

arXiv