{"id":"22193ef6-e699-4dc7-a9e0-2beeea24ed86","arxiv_id":"2412.17170","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Influence-SSL defines influence as the sensitivity of a sample's representation to augmentation, and shows the resulting scores can identify duplicates, outliers, and fairness-relevant examples in SSL models.","lead":"This paper introduces Influence-SSL, a label-free method to score how much each training image shaped a self-supervised model's learned representations. It may help practitioners find duplicates, outliers, and biased samples in unlabeled datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof substitutes a single-example Hessian for the dataset Hessian required by Eq. 6, so the stated closed-form influence does not follow for the proposed method.","rationale":"The reader correctly identifies the deep-network gap, but the more damaging issue is internal to the linear proof. The theorem's Hessian is not the H in Eq. 6; the proof analyzes self-influence with a per-sample regularized Hessian. The computation above shows the discrepancy already at n=2, so this is not a question of nonlinear extrapolation. The empirical sections (stability, duplicates, fairness, pruning) are interesting and code is available, and the influence score may still be a useful heuristic. A conditional verdict remains appropriate, but the condition should be a corrected theorem or an explicit n=1 statement rather than only a clarification of the Hessian definition. This is why I retain the reader's conditional verdict while disagreeing partially with the stated weakest assumption.","tokens_in":19935,"tokens_out":12264,"duration_ms":114629,"concrete_test":"Recompute Theorem 1 with the full dataset Hessian. Minimal check: take d=1, n=2, ε₁=ε₂=ε, δ₁=δ₂=1, W=1; Eq. 6 gives -ε², Theorem 1 gives -2ε². For the general case, derive I_j = -4εⱼ⁴|Wδⱼ|² δⱼᵀ(2Σᵢεᵢ²δᵢδᵢᵀ + λI)⁻¹δⱼ and report whether it equals -2εⱼ²|Wδⱼ|² for n>1. Also inspect the released LoGra/LogIX code to confirm whether H is built from all training points; if it is, the appendix proof must be changed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation 6 defines Hθ as the Hessian of the model's loss over the dataset. In Appendix C, however, Theorem 1 is proved with H_ssl = 2ε²δδᵀ + λI, the regularized Hessian of one training example, and Sherman-Morrison is applied to that rank-1 matrix. If H is instead the dataset Hessian H = 2Σᵢ εᵢ²δᵢδᵢᵀ (+λI), the self-influence of point j is I_j = -4εⱼ⁴|Wδⱼ|² δⱼᵀH⁻¹δⱼ. The factor δⱼᵀH⁻¹δⱼ is not 1/(2εⱼ²) in general; it is a leverage score of δⱼ in the dataset. In the scalar case d=1, n=2, ε₁=ε₂=ε, δ₁=δ₂=1, Eq. 6 gives -ε², while Theorem 1 predicts -2ε². Thus the theorem's closed form only matches an n=1 Hessian. Since the implemented H (via LoGra/LogIX) is the full training-set Hessian, the theoretical characterization does not apply to the quantity computed in the experiments. The claimed bridge between Influence-SSL and augmentation consistency needs either a corrected derivation or an explicit statement that the result is for a single training point.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Influence-SSL, a label-free influence score for self-supervised learning defined in Eq. (6) as I(f,i) = -∇L(fθ(xi), fθ(x̂i))^T Hθ^{-1} ∇L(fθ(xi), fθ(x̂i)), where L is cosine distance and Hθ is described as the Hessian of the model's loss over the dataset. The authors provide a linear-network analysis (Theorem 1) claiming the score reduces to -2ε²|Wδ|², prove structural properties such as rotation invariance and scaling, and present experiments on CIFAR-10/100 and FairFace showing score stability across runs, detection of duplicates and uniform-background images, improved downstream accuracy after removing high-influence points, and uneven demographic representation among high-influence faces.","tokens_in":20292,"tokens_out":7738,"duration_ms":73782,"significance":"If the proposed score were a valid influence function, it would fill a real gap: supervised influence functions require labels and cannot be applied directly to SSL embeddings, and existing label-free attribution tools are mostly heuristic. The paper is commendable for releasing code, using three SSL frameworks (SimCLR, BYOL, Barlow Twins), reporting training configurations, and for the clean algebraic properties of the linearized score (Propositions 1-2, Appendix C). However, the theoretical bridge from Eq. (6) to the closed form in Theorem 1 is broken by the Hessian mismatch, and the simplified closed form is a scalar multiple of the per-sample SSL loss. The empirical removal results therefore currently read as augmentation-loss pruning rather than as downstream effects of causal example influence. These issues affect the central claim, so the paper in its present form is not ready for acceptance.","major_comments":[{"comment":"The proof applies Sherman-Morrison to H_ssl^λ = 2ε²δδᵀ + λI, the regularized Hessian of a single training example, but Eq. (6) and the classical influence definition require the Hessian of the full training objective over the dataset. For a dataset with Hessian H = 2Σ_i ε_i² δ_i δ_iᵀ + λI, the self-influence of point j is I_j = -4ε_j⁴ |Wδ_j|² δ_jᵀ H^{-1} δ_j, not -2ε²|Wδ_j|². The factor δ_jᵀH^{-1}δ_j is a leverage score and is not generally 1/(2ε_j²); even in the scalar case d=1, n=2, with ε_1=ε_2=ε and δ_1=δ_2=1, Eq. (6) gives -ε² while Theorem 1 predicts -2ε². Since the experiments use the full training-set Hessian via LoGra/LogIX, the theorem does not characterize the quantity actually computed.","section":"§3.1 / Appendix C.1, Theorem 1"},{"comment":"In the simplified setting, the derived influence I_ssl(x) = -2ε²|Wδ|² is exactly -2 times the per-sample SSL loss L_ssl(x)=ε²|Wδ|². Thus the proposed 'influence' reduces, by the paper's own equations, to a deterministic monotone transform of the loss of one augmented pair. The score therefore measures augmentation sensitivity of the current representation, not the effect of the example on the learned parameters; a point can be 'high-influence' without causing any change in θ. This is a conceptual issue, not just a presentational one: it means the theoretical argument does not establish that Eq. (6) is an influence function in the Koh-Liang sense.","section":"§3.1, Theorem 1 and Lemma 3"},{"comment":"The main empirical claim is that removing high-Influence-SSL points improves downstream accuracy, but the experiments do not compare against random removal or against removal by the per-sample SSL/consistency loss. Given Major Comment 2, the observed improvement may be exactly the effect of removing the largest per-sample augmentation losses; such hard-example pruning is a known phenomenon and would not validate the proposed attribution method. Please add baselines (random, highest raw SSL loss, highest gradient norm) and report the rank correlation between Influence-SSL and raw SSL loss on the actual ResNet18 models.","section":"§4.3, Figure 6"},{"comment":"The theory is stated for a two-layer linear network with small perturbations δ satisfying δᵀx = 0 and Euclidean loss, yet the experiments use ResNet18, Gaussian noise with σ=0.2, cosine distance, and nonlinear SSL objectives. No argument or diagnostic is given that the Hessian inverse in Eq. (6) is approximately constant across examples or that the linear closed form is predictive of deep-network scores. The paper's own caveat that influence functions can deviate from leave-one-out effects in neural networks (Section 3.1, citing [5,2]) makes this gap load-bearing for the claim that the experiments validate the theory. Please provide, at minimum, a rank-correlation check between Eq. (6) with full Hessian and the linearized score, or explicitly reframe the theory as intuition only.","section":"§4 and Theorem 1"}],"minor_comments":[{"comment":"Eq. (6) defines I(f,i) as a negative quadratic form, so scores are non-positive, but the paper reports positive 'log influence scores' and ranks 'top influential' without saying whether it uses |I| or -I; please clarify this for all figures and tables.","section":"Eq. (6) and experimental ranking"},{"comment":"Duplicate identification is based on visual inspection; provide a quantitative duplicate-detection measure (e.g., fraction of nearest-neighbor pairs among lowest-influence images) and a threshold for declaring duplicates.","section":"Appendix B.4, Figure 12"},{"comment":"Appendix A lists seeds and says error bars are used, but the figure appears without visible error bars; add them or state explicitly that they are omitted.","section":"Figure 6"},{"comment":"The racial imbalance claim needs a statistical test against the FairFace base rates and a control using the lowest-influence examples; currently it is a histogram without error bars or a significance statement.","section":"§4.5, Figure 8"},{"comment":"There are several typos, including 'validation ses' in Appendix A, 'eac' in Appendix C.2, and 'Inluence' in the Figure 9 caption; also, reference [21] is the authors' own workshop paper and its relation to this submission should be clarified.","section":"Appendix A and C.2"}],"recommendation":"major_revision","confidential_remarks":"The referee report identifies a mismatch between Theorem 1's proof and Eq. (6). If the authors can supply a corrected theorem for the full-dataset Hessian and the baseline comparisons, the paper may become a useful study of augmentation-sensitivity-based data scoring; without those, the central claim of an 'influence function for SSL' is not supported. I therefore recommend major revision rather than outright rejection, but the requested changes are substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the paper offers a genuinely useful idea: define self-influence in SSL by the stability of a sample's representation under augmentation, which needs no labels. The empirical observations that high-influence SSL examples tend to have uniform backgrounds, and that removing them can improve downstream accuracy, are interesting and worth following up. Second, the theoretical support is weaker than it looks. Theorem 1's proof substitutes a one-example Hessian for the dataset Hessian required by Equation 6, and in that linear setting the 'influence' is just a fixed multiple of the per-sample loss.\n\nOn the theory: the proof in Appendix C derives the influence using H_ssl = 2ε²δδᵀ + λI, the regularized Hessian of a single training example. Equation 6 defines Hθ as the Hessian of the model's loss over the dataset. These are different objects. For n>1, the dataset Hessian is a sum of rank-one matrices, and the closed-form I_ssl = -2ε²|Wδ|² does not follow. A simple scalar example with two identical points gives a factor-of-two discrepancy. This is not a cosmetic issue; the theorem cannot be cited as a characterization of the influence score being computed in the experiments. The paper should either derive the dataset-Hessian case or explicitly state that the result is for a single training point. Also, even in that one-example setting, the influence is exactly -2 times the per-sample SSL loss, which makes the theory close to tautological.\n\nOn the experiments: the removal study (Figure 6) lacks a random-removal baseline, and the error bars are not visible in the figure. Without a random baseline I cannot tell whether the performance change is specific to the influence ranking or just an effect of removing any subset of that size. The fairness analysis is a histogram over the top 100 points with no statistical comparison to the dataset's race distribution. And the definition of Hθ in Equation 6 is ambiguous: is it the Hessian of the actual SSL pretraining loss (e.g., InfoNCE) or of the cosine consistency loss? If the latter, then H is not the Hessian of the objective whose minimizer is being perturbed, and the classical influence interpretation breaks down. The paper should specify which loss H is computed from.\n\nWhat the paper does well: the idea of a label-free, augmentation-based influence score is novel for SSL, and the qualitative findings on duplicates and background bias are thought-provoking. The writing is clear, the related work is honestly situated, and the code is public. The authors also acknowledge that the theoretical analysis is limited to linear networks and that extending it to nonlinear architectures remains open.\n\nWho is this for: researchers working on SSL interpretability, data attribution, and dataset auditing. They will find the empirical patterns useful even if they cannot trust the theoretical derivation. I would send this to peer review, but with major revisions expected. A good referee should ask for a corrected theorem or a narrowed claim, clarification of which Hessian is used, and a random-removal baseline. The core idea is promising; it is just not the finished article.","headline":"Useful label-free influence score for SSL, but Theorem 1 is proven for a single-example Hessian, not the dataset Hessian the method actually uses, so the theory needs rework before the empirical claims are supported.","tokens_in":20789,"tokens_out":5205,"would_cite":true,"duration_ms":48960,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Influence-SSL, a label-free influence score for self-supervised visual models that measures each training image's contribution to its own augmented-view consistency, and reports that removing the highest-scoring…","keywords":["influence functions","self-supervised learning","data attribution","representation learning","augmentation invariance","outlier detection","fairness analysis","duplicate detection"],"falsifier":"Retrain a small SSL model after removing each of the top-1000 Influence-SSL points and compare the actual leave-one-out change in downstream accuracy with the score's predictions; if removing high-influence points does not consistently improve accuracy across seed runs, the central removal claim fails. A second check is to compute Equation 6 in the exact two-layer linear network of Theorem 1 with $\\varepsilon\\ll1$ and verify that the score equals $-2\\varepsilon^2|W\\delta|^2$ within numerical precision.","tokens_in":19762,"feed_emoji":"🔎","tokens_out":7889,"duration_ms":65443,"temperature":0.7,"pith_summary":"Self-supervised models learn representations without labels, so classical influence functions—which trace a training point's contribution to a labeled prediction—do not directly apply. This paper defines a label-free influence score for SSL that asks how much a training image stabilizes its own representation when the image is augmented; the score is the negative quadratic form of the augmentation-consistency gradient with the inverse Hessian. The authors prove that in a two-layer linear network with small additive perturbations this influence is exactly a scaled version of the per-sample SSL loss, and they show empirically that the scores are stable across training runs, flag duplicates and uniform-background outliers, reveal an uneven racial profile in a face dataset, and—counterintuitively—that removing the highest-influence training images improves downstream classification accuracy in three SSL methods.","feed_headline":"Removing the most influential images improves self-supervised accuracy","feed_subtitle":"A label-free influence score shows SSL models key on background-heavy outliers; deleting them sharpens representations.","key_machinery":"The load-bearing object is the SSL self-influence score $I(f,i) = -\\nabla_\\theta L(f_\\theta(x_i), f_\\theta(\\hat{x}_i))^\\top H_\\theta^{-1} \\nabla_\\theta L(f_\\theta(x_i), f_\\theta(\\hat{x}_i))$, where $L$ is the cosine distance between a training image and an augmented view and $H_\\theta$ is the Hessian of the model's loss over the dataset. The paper's theoretical engine is the reduction of this score in a two-layer linear network with $x_{\\mathrm{aug}} = x + \\varepsilon\\delta(x)$, $|\\delta|=1$, $\\varepsilon\\ll1$: there the score collapses to $-2\\varepsilon^2|W\\delta|^2$, making influence exactly proportional to the per-sample SSL loss. In practice the Hessian inverse is approximated with a low-rank inverse-Hessian-vector-product method, and the augmentation is implemented as Gaussian noise, which the appendix shows is the perturbation choice that best preserves score rankings.","core_discovery":"The central claim is that a training example's influence in self-supervised learning can be measured without any labels by looking at how much the learned representation changes when that example is augmented, and that this score is informative about what the model learned. In the simplified two-layer linear setting the score has the closed form $I_{\\mathrm{ssl}}(x) = -2\\varepsilon^2 |W\\delta(x)|^2$, so influence is proportional to the per-sample augmentation loss and measures how far the model's representation moves under that input's perturbation. Empirically, high-influence CIFAR images are dominated by examples with uniform backgrounds, low-influence images include near-duplicates, and deleting the highest-influence examples improves downstream accuracy for SimCLR, BYOL, and Barlow Twins on CIFAR-10 and CIFAR-100, the opposite of the supervised baseline. On FairFace, the top-100 influential faces over-represent Black and Middle Eastern groups, which the paper presents as evidence that the score can surface biases hidden from conventional metrics.","pith_inferences":["A testable corollary of the linear theory is that in deep networks the score's ranking may be dominated by the per-sample augmentation loss; comparing Influence-SSL rankings against that loss alone would show whether the inverse-Hessian term adds signal.","The uniform-background result suggests SSL pretraining can encode background as a shortcut; influence scores could be used as a pre-deployment audit to find images whose backgrounds, not objects, drive their representations.","Because low-influence examples are near-duplicates, Influence-SSL could double as a label-free deduplication filter before pretraining; one extension would test whether dropping low-influence images reduces pretraining cost without hurting downstream accuracy.","The paper deliberately leaves cross-point influence (a training point's effect on a held-out query's representation) for future work; defining query-conditional influence would let practitioners ask which training images best explain a specific test prediction."],"forward_implications":["Removing the highest-influence training points improves downstream top-1 accuracy for SimCLR, BYOL, and Barlow Twins on CIFAR-10 and CIFAR-100, while the same removal degrades a supervised baseline.","Influence-SSL scores are reproducible: Pearson rank correlations exceed 0.96 across independent training runs for all three SSL frameworks.","Low-influence examples tend to be semantic duplicates and high-influence examples tend to have uniform backgrounds, so the score can serve as a label-free signal for deduplication and outlier detection.","On FairFace, the top-100 influential faces are disproportionately Black and Middle Eastern, indicating the score can reveal demographic skew in SSL representation learning.","In the linear model, influence decomposes into a perturbation scale factor and a representation-sensitivity term, so the score directly measures how strongly augmentations move a point in representation space."],"supporting_citations":[{"why":"Gives the gradient–Hessian influence-function formalism that Influence-SSL adapts to unlabeled data.","marker":"[25]"},{"why":"Supplies the inverse-Hessian-vector-product approximation used to compute the scores at the scale of ResNet18.","marker":"[9]"},{"why":"Defines SimCLR, one of the three SSL methods whose influence scores and removal curves are measured.","marker":"[8]"},{"why":"Defines BYOL, the SSL method used for the influence, duplicate, and FairFace experiments.","marker":"[16]"},{"why":"Defines Barlow Twins, the third SSL method used across the main experiments.","marker":"[34]"},{"why":"Documents the fragility of influence functions in deep networks, which the paper cites to justify its exact theorem being restricted to the linear setting.","marker":"[5]"},{"why":"Provides the FairFace dataset whose top-influence demographic distribution is analyzed for fairness.","marker":"[24]"},{"why":"Establishes the conventional expectation that removing influential examples degrades performance, the claim the SSL removal experiment overturns.","marker":"[12]"}],"fun_headline_variants":["Label-free influence finds SSL's background-heavy outliers","Deleting top influence images boosts self-supervised accuracy","Influence-SSL: Tracing what self-supervised models learn","Unmasking SSL's influential data without any labels","Why background-heavy images drive self-supervised models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument assumes that what is true for a two-layer linear network with tiny perturbations is still true for deep nonlinear SSL models; if the inverse-Hessian term stops behaving like the constant it becomes in the linear proof, the theoretical foundation no longer covers the experiments.","fun_headline_variants_meta":{"raw":{"variants":["Label-free influence finds SSL's background-heavy outliers","Deleting top influence images boosts self-supervised accuracy","Influence-SSL: Tracing what self-supervised models learn","Unmasking SSL's influential data without any labels","Why background-heavy images drive self-supervised models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2229,"prompt_tokens":932,"completion_tokens":1297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1220}},"tokens_in":548,"tokens_out":1297,"duration_ms":9775,"temperature":1.0,"reasoning_tokens":1220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:43:42.314678+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain a small SSL model after removing each of the top-1000 Influence-SSL points and compare the actual leave-one-out change in downstream accuracy with the score's predictions; if removing high-influence points does not consistently improve accuracy across seed runs, the central removal claim fails. A second check is to compute Equation 6 in the exact two-layer linear network of Theorem 1 with $\\varepsilon\\ll1$ and verify that the score equals $-2\\varepsilon^2|W\\delta|^2$ within numerical precision.","supporting_citations":[{"cited_title":"Understanding black-box predictions via influence functions","cited_arxiv_id":null,"evidence_quote":"Gives the gradient–Hessian influence-function formalism that Influence-SSL adapts to unlabeled data."},{"cited_title":"What is your data worth to gpt? llm-scale data valuation with influ- ence functions, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the inverse-Hessian-vector-product approximation used to compute the scores at the scale of ResNet18."},{"cited_title":"Influence functions in deep learning are fragile","cited_arxiv_id":null,"evidence_quote":"Documents the fragility of influence functions in deep networks, which the paper cites to justify its exact theorem being restricted to the linear setting."},{"cited_title":"Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation","cited_arxiv_id":null,"evidence_quote":"Provides the FairFace dataset whose top-influence demographic distribution is analyzed for fairness."},{"cited_title":"Does learning require memorization? a short tale about a long tail","cited_arxiv_id":null,"evidence_quote":"Establishes the conventional expectation that removing influential examples degrades performance, the claim the SSL removal experiment overturns."}],"review_version":1}