{"id":"a3ec70ad-37da-46bd-b30c-b0cb824eeb65","arxiv_id":"2412.01335","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"VIF approximates leave-one-out retraining influence for non-decomposable losses (contrastive, ranking, Cox) using a finite-difference of the loss evaluated via auto-differentiation.","lead":"This paper introduces Versatile Influence Function (VIF), a way to estimate how much each training data point affects a model's predictions even when the loss function cannot be split into per-point terms, such as contrastive or ranking losses. This makes influence-based data attribution practical for many modern machine learning models without case-specific math, and it is up to 1000 times faster than retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VIF is not actually derived from Eq. (7): Definition 3.3 defines L̃ only on uniform subset distributions, so the ε→0 limit is undefined and the ε=1 endpoint is not a finite-difference approximation for non-decomposable losses.","rationale":"The reader's weakest assumption was convexity and the lack of guarantees for non-convex losses. I agree that is a real limitation, but I find a more fundamental problem that precedes convexity: the paper never supplies a well-defined path through which VIF approximates the IF in Eq. (7). Definition 3.3 restricts L̃ to uniform distributions on subsets, so the perturbed measures (1−ε)P_n + εQ used in Eq. (7) are outside the domain for all ε ∈ (0,1). The ε = 1 finite difference used in Proposition 3.1 is therefore not a finite-difference approximation of the derivative; it is an endpoint difference of a function that is not even defined between the endpoints. This does not mean VIF is useless: the Cox regression analysis provides a genuine O_p(1/n) error bound for that specific loss, and the learning-to-rank experiments show strong correlations with brute-force LOO. But the paper's headline claim, that VIF extends the classical influence function to any non-decomposable loss via auto-differentiation, is not supported by the derivation. The empirical evidence is strong enough to keep the paper in CONDITIONAL status, and I do not change the reader's verdict, but the authors should either extend Definition 3.3 to weighted distributions and re-derive the connection to Eq. (7), or explicitly reposition VIF as a heuristic with only case-specific guarantees such as the Cox result.","tokens_in":22213,"tokens_out":10813,"duration_ms":100615,"concrete_test":"Take a small strictly convex non-decomposable loss that has a natural weighted extension, e.g., the Cox partial likelihood or a softmax/ListMLE loss with case weights w_i, and compare three quantities at n = 50, 100, 200: (i) the exact Gateaux derivative of the weighted loss at P_n toward δ_i (obtained analytically or by automatic differentiation through ε), (ii) VIF from Eq. (10), and (iii) brute-force leave-one-out retraining. If VIF differs from (i) by more than the constant factor n and the gap does not shrink as O(1/n), the claim that VIF is a finite-difference approximation of Eq. (7) fails. As a minimal check, attempt to evaluate Eq. (7) under Definition 3.3: for ε ∈ (0,1), the mixture (1−ε)P_n + εδ_i is not in P(n), so the limit is undefined; this alone demonstrates the derivation is incomplete.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central derivation has a domain gap. Definition 3.3 defines L̃(θ,P) only for P in P(n), the set of uniform distributions on subsets of the n training points. The IF in Eq. (7) requires evaluating L̃ along the mixture (1−ε)P_n + εQ for ε → 0; for any ε ∈ (0,1) this mixture is not uniform on a subset, so L̃ is undefined and the limit in Eq. (7) does not exist. The paper avoids this by using the endpoint ε = 1, where (1−ε)P_n + εQ = Q(−i) is in P(n). But a one-step secant between endpoints is a finite-difference approximation of a derivative only if the function is defined and sufficiently regular on the whole interval. For a general non-decomposable loss, L(θ,b) is not linear in b, and no error bound is provided. In fact, if one naively extends the support-indicator definition of L̃ to the mixture, the gradient would be constant for ε ∈ (0,1) and the Gateaux derivative in Eq. (7) would be zero, while VIF in Eq. (10) is nonzero. Thus VIF is not the finite-difference IF of Eq. (8) for generic non-decomposable losses. The Cox Op(1/n) bound in Theorem 3.2 is a special case and does not establish the general claim that VIF approximates the classical IF. This gap is independent of convexity: even for strictly convex non-decomposable losses, the connection from Eq. (7) to Eq. (10) is missing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Versatile Influence Function (VIF), defined in Eq. (10) as a Hessian-inverse times the parameter gradient of the difference between the full loss and the leave-one-out loss evaluated at the trained parameters. The intended contribution is to extend influence-function-based data attribution from decomposable M-estimator losses to general non-decomposable losses such as Cox partial likelihood, contrastive losses, and listwise ranking losses. The paper proves that VIF reduces exactly to the classical influence function for M-estimators, proves an O_p(1/n) approximation error relative to the Reid--Crepeau influence function for Cox regression (Theorem 3.2 / Theorem A.1), and reports experiments on survival analysis, node embedding, and learning-to-rank showing high correlations with brute-force leave-one-out retraining and large speedups.","tokens_in":22498,"tokens_out":7952,"duration_ms":74379,"significance":"If the claimed generality held, VIF would be a valuable practical tool: it would let practitioners compute influence-style attributions for losses that are not sums of per-point terms, using only automatic differentiation. The M-estimator exactness result and the Cox regression rate are real mathematical contributions, and the empirical protocol is strengthened by reporting a brute-force retraining noise ceiling rather than only comparing against one retraining run. The Cox proof in Appendix A.5 is nontrivial and appears carefully executed. However, the central derivation from the robust-statistics influence function to VIF has a domain gap for generic non-decomposable losses, so the broader ``arbitrary non-decomposable loss'' claim is currently supported only heuristically and by case-specific experiments.","major_comments":[{"comment":"The derivation of VIF from the influence-function formulation in Eq. (7) is not valid for generic non-decomposable losses. Eq. (7) is a Gateaux derivative that requires evaluating L~(θ, (1−ε)P_n + εQ) for ε in a neighborhood of 0. However, Definition 3.3 defines L~(θ, ·) only on P(n), the set of uniform distributions supported on subsets of the n training points. For any ε ∈ (0,1), the mixture (1−ε)P_n + εδ_z_i is not uniform on a subset, so L~ is undefined and the limit in Eq. (7) does not exist. The paper instead uses the endpoint ε = 1, where the perturbed measure Q(−i) does lie in P(n); but a one-point secant between P_n and Q(−i) is a finite-difference approximation of a derivative only if the relevant function is defined and sufficiently regular on the whole interval. For a general non-decomposable loss, L(θ,b) is not linear in b, so no such regularity is available. Proposition 3.1 establishes an algebraic identity for the endpoint, not a connection to Eq. (7). This gap is independent of convexity: even for strictly convex non-decomposable losses, the link from Eq. (7) to Eq. (10) is missing. The paper should either define a continuous extension of L(θ,b) to mixtures and prove an approximation bound, or explicitly reframe VIF as a direct leave-one-out approximation heuristic and revise the claims that it ``serves as an approximation of the general formulation of IF''.","section":"§3.3--3.4, Definition 3.3, Eq. (8), Proposition 3.1"},{"comment":"The theoretical support for the general non-decomposable claim is limited to two cases: exact recovery for M-estimators and an O_p(1/n) bound for Cox regression. Theorem 3.2 is specifically about Cox partial likelihood and does not extend to the other two showcased settings, contrastive node embedding and listwise ranking. For those settings, the paper provides empirical correlations against brute-force leave-one-out retraining (Table 1), but leave-one-out retraining is not the influence function of Eq. (7), so these experiments do not fill the derivation gap. The abstract and introduction state that VIF ``can be straightforwardly applied to machine learning models trained with any non-decomposable loss''; as written, that generality is not established. At minimum, the paper should clearly state which losses are covered by formal guarantees and which are supported only by empirical evidence.","section":"§3.4--3.5, Theorem 3.2, Table 1"}],"minor_comments":[{"comment":"There are several typos and grammatical slips, including ``mathetical'' (Section 1), ``Tabel 2'' (Section 4.3), ``Senario'' (Table 2), ``Actural'' (Figure 1), and ``close-form'' (Section 3.5). These should be corrected in a revision.","section":"Throughout"},{"comment":"The heatmap appendix already notes a color-scale misalignment between VIF and LOO values. It would be helpful to state explicitly in the main text that VIF is designed for ranking/correlation agreement rather than for exact absolute-value agreement, since the Pearson correlations in Table 1 measure the former.","section":"Appendix D, Figure 2"},{"comment":"For the neural-network Cox model, VIF correlations are around 0.2 for all methods, including the brute-force baseline. The paper should clarify whether this is a noise ceiling of the retraining procedure and, if so, report the correlation between two brute-force runs for the neural network model as was done in Table 1; otherwise readers may infer that the method degrades sharply on non-convex models.","section":"Appendix C, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The core method is promising and the empirical work is honest in reporting noise ceilings, but the paper currently overclaims the theoretical connection to the robust-statistics influence function. The domain gap in Definition 3.3 is fixable by reframing VIF as a heuristic leave-one-out approximation with exact or asymptotic justification in special cases; I would encourage the authors to make that revision rather than to remove the theoretical sections. The Cox proof is a genuine strength and should be retained."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: VIF is a worthwhile contribution and deserves peer review, but the derivation connecting it to the classical influence function has a real gap. The finite-difference step only works for M-estimators; for generic non-decomposable losses the object in Eq. (7) is not defined.\n\nWhat's new and good: VIF replaces case-by-case analytical IF derivations with a single autodiff-friendly secant between the full loss and the leave-one-out loss. That is a practical advance. For M-estimators it recovers the classical IF exactly, and the Cox error bound (Theorem A.1) is a serious piece of work with a detailed proof. The empirical results are mostly strong: high correlations with brute-force LOO on Cox and listwise ranking, and the node embedding comparison to a noise ceiling is honest.\n\nThe soft spot is not cosmetic. Definition 3.3 defines L̃ only on uniform distributions over subsets of the training points. The mixture (1−ε)Pn + εQ in Eq. (7) is not such a distribution for any ε in (0,1), so the limit that defines the IF does not exist under their own setup. The paper jumps to ε=1, which lands on Q(−i), but a secant between two discrete points is not a derivative unless the function is defined and sufficiently regular on the whole interval. For a generic non-decomposable loss there is no such extension. The Cox result is a special case, not a general license. This does not invalidate VIF as a heuristic—the empirical evidence supports it—but the paper should either prove an approximation bound for a reasonable class of losses (e.g., after a smooth relaxation of the binary indicator) or explicitly present VIF as a heuristic with only case-specific theoretical backing.\n\nMinor issues: no code or data released, no error bars in Table 1, and the node embedding correlations around 0.4 are modest even if the noise-ceiling comparison is appropriate. The paper's own limitation paragraph acknowledges the convexity assumption, which is fine, but it does not acknowledge the domain gap in the derivation.\n\nBottom line: I'd send this to review. The idea is novel, the experiments are useful, and the theoretical gap is fixable. The authors should be asked to either close the gap or soften the claim that VIF approximates the general IF.","headline":"A clever and useful method, but the paper's central claim that VIF approximates the general influence function is not actually established by the derivation.","tokens_in":23060,"tokens_out":3680,"would_cite":true,"duration_ms":32484,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the classic influence function can be extended to any loss that depends jointly on multiple training points, by replacing the per-point weight perturbation with a finite difference between the full dataset and the…","keywords":["influence function","data attribution","non-decomposable loss","leave-one-out retraining","Cox regression","contrastive loss","learning-to-rank","auto-differentiation"],"falsifier":"Compute VIF and the exact influence function on a Cox regression at several finite sample sizes $n$ and check that the difference actually scales as $1/n$; if the difference does not shrink (or fails to match brute-force leave-one-out in absolute value, not just rank correlation), the central approximation claim is falsified. The same test can be run on any convex non-decomposable loss whose exact influence function is known.","tokens_in":21973,"feed_emoji":"🎯","tokens_out":5039,"duration_ms":37691,"temperature":0.7,"pith_summary":"This paper claims that data attribution by influence functions—quantifying how removing one training object changes a model—can be extended from the usual per-point loss sums (M-estimators) to any loss that depends jointly on many training objects. The proposed Versatile Influence Function (VIF) approximates the leave-one-out parameter change using only quantities at the original optimum: the inverse Hessian of the total loss and the gradient of the difference between the full loss and the loss with object $i$ removed. For decomposable losses VIF reduces exactly to the classical influence function, and for Cox regression its deviation from the exact influence function is $O_p(1/n)$. If the paper is right, influence-based attribution becomes available for contrastive embeddings, ranking losses, and survival models without case-by-case derivations, computed purely through auto-differentiation.","feed_headline":"One formula brings influence functions to contrastive and ranking losses","feed_subtitle":"VIF replaces per-point derivations with a finite difference, matching leave-one-out retraining at up to 1000x speed.","key_machinery":"The central object is the finite-difference influence function $\\widehat{\\mathrm{IF}}_\\varepsilon(\\tilde{\\theta}(P);Q)$, which replaces the limit in the Gateaux derivative of the minimum of the loss with a finite difference at $\\varepsilon$; choosing $P$ as the empirical distribution on the full data and $Q$ as the empirical distribution with object $i$ removed converts that derivative into a comparison of $L(\\theta,1)$ and $L(\\theta,1_{-i})$. The identity that carries the argument is the linearized optimality condition $$\\nabla_\\$\\theta$ \\tilde{L}(\\tilde{\\$\\theta$}(P),P)=0$$ together with the finite-difference form, which yields the inverse-Hessian-times-gradient formula. Because the formula only requires gradients and Hessians of the total loss at the original optimum, it can be implemented with auto-differentiation instead of per-loss analytic derivations.","core_discovery":"The paper's central claim is that the leave-one-out effect $\\hat{\\theta}(1_{-i})-\\hat{\\theta}(1)$ for any non-decomposable loss can be approximated by $$\\mathrm{VIF}(\\hat{\\$\\theta$}(1);i) := -\\left[\\frac{1}{n}\\$nabla^{2}$_\\$\\theta$ L(\\hat{\\$\\theta$}(1),1)\\right]^{-1}\\nabla_\\$\\theta$\\left[L(\\hat{\\$\\theta$}(1),1)-L(\\hat{\\$\\theta$}(1),1_{-i})\\right],$$ where $L(\\theta,b)$ is the loss as a function of the presence vector $b$. The argument starts from the robust-statistics influence function as a functional derivative, replaces the limit by a finite difference between the full empirical distribution and the distribution with object $i$ removed, and then writes that difference directly on the non-decomposable loss's presence vector. The paper shows that under the M-estimator specification the finite-difference influence function coincides exactly with the classical influence function (up to a constant), and that under Cox regression the VIF differs from the exact influence function of Reid and Crepeau by $O_p(1/n)$. Empirically, VIF matches brute-force leave-one-out retraining correlations on survival analysis, node embedding, and listwise learning-to-rank while running orders of magnitude faster.","pith_inferences":["VIF's finite-difference view suggests a natural test: for any convex non-decomposable loss where leave-one-out can be computed exactly on small problems, the absolute (not just rank) error of VIF should shrink with $n$; verifying this on new losses would show the Cox guarantee is not special.","The same construction could apply to discrete or combinatorial objects beyond data points, such as removing labels, features, or edges in a graph, as long as the loss can be evaluated with the object's presence toggled.","For non-convex models the paper uses heuristics with no guarantee; a plausible extension is to check VIF against trajectory-based attribution on small neural networks and see where the convex approximation breaks."],"forward_implications":["Influence-function attribution becomes applicable to contrastive losses, ranking losses, and survival partial likelihoods without new derivations.","For decomposable losses, VIF coincides with the classical influence function, so existing methods are recovered as a special case.","VIF computation costs no model retraining; experiments report speed-ups of up to about $10^3\\times$ over brute-force leave-one-out retraining.","The same inverse-Hessian acceleration tricks (conjugate gradient, LiSSA) carry over, letting VIF scale to larger models.","Cox regression's approximation error of $O_p(1/n)$ suggests accuracy improves as the training set grows."],"supporting_citations":[{"why":"The classical influence-function derivation for decomposable losses that VIF generalizes.","marker":"[Koh & Liang, 2017]"},{"why":"The general statistical influence-function formulation that VIF approximates.","marker":"[Huber & Ronchetti, 2009]"},{"why":"The exact Cox-regression influence function used as the comparison target in Theorem 3.2.","marker":"[Reid & Crepeau, 1985]"},{"why":"Defines the Cox partial likelihood, one of the three example losses.","marker":"[Cox, 1972]"},{"why":"Defines ListMLE, the listwise learning-to-rank loss used in experiments.","marker":"[Xia et al., 2008]"},{"why":"DeepWalk node embeddings, the contrastive-loss example.","marker":"[Perozzi et al., 2014]"},{"why":"LiSSA, used for the inverse-Hessian acceleration experiments.","marker":"[Agarwal et al., 2017]"}],"fun_headline_variants":["Influence functions extend to contrastive and ranking losses","VIF brings influence functions to non-decomposable losses","Finite-difference influence matches leave-one-out at 1000x speed","Auto-diff influence for any loss, 1000x faster than retraining","From M-estimators to any loss: influence via finite differences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the loss has a unique minimizer with an invertible Hessian for every subset of training objects, so the leave-one-out parameter $\\hat{\\theta}(1_{-i})$ is well defined and the first-order linearization in the finite-difference step is valid; the paper states this explicitly and falls back on heuristics for non-convex neural networks.","fun_headline_variants_meta":{"raw":{"variants":["Influence functions extend to contrastive and ranking losses","VIF brings influence functions to non-decomposable losses","Finite-difference influence matches leave-one-out at 1000x speed","Auto-diff influence for any loss, 1000x faster than retraining","From M-estimators to any loss: influence via finite differences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000715,"raw_usage":{"total_tokens":3310,"prompt_tokens":1138,"completion_tokens":2172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":754,"completion_tokens_details":{"reasoning_tokens":2081}},"tokens_in":754,"tokens_out":2172,"duration_ms":13873,"temperature":1.0,"reasoning_tokens":2081,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:26:54.715767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute VIF and the exact influence function on a Cox regression at several finite sample sizes $n$ and check that the difference actually scales as $1/n$; if the difference does not shrink (or fails to match brute-force leave-one-out in absolute value, not just rank correlation), the central approximation claim is falsified. The same test can be run on any convex non-decomposable loss whose exact influence function is known.","supporting_citations":[],"review_version":1}