{"id":"068ff5ca-b1b6-4a17-9135-8f61b883cc6c","arxiv_id":"2507.04059","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SAM-HIF and SAM-GIF are proposed as data attribution scores for SAM-trained models, but SAM-GIF is TracIn with SAM gradients and SAM-HIF's derivation contains a load-bearing error.","lead":"The paper proposes two ways to score how much each training image influenced a model trained with Sharpness-Aware Minimization (SAM): one uses the Hessian at the final weights, the other uses gradients saved during training. The trajectory-based method is essentially TracIn applied to SAM checkpoints, while the Hessian method adds a perturbation correction that rests on a flawed stationarity condition.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 4.2 drops the λω* term and asserts a false equivalence; every Taylor expansion in SAM-IF and SAM-HIF inherits this error, so the central derivation does not follow from the stated SAM objective.","rationale":"The reader's weakest_assumption identifies the same defect in Lemma 4.2, and the stress-test confirms it is the most load-bearing issue. The paper's headline theoretical contribution, SAM-HIF, is built on a stationarity condition that drops the λω* term and then asserts an equivalence that is mathematically false; the proof of Theorem 4.3 in Appendix A.1 explicitly uses this incorrect condition as the anchor for its Taylor expansion. If the true stationarity is used, the influence formulas acquire additional terms and the derivation no longer yields the closed forms claimed in Theorems 4.3 and 4.5. The separate conflict between Eq. (2) and Danskin's theorem compounds the problem, but the dropped λ term alone is sufficient to invalidate the derivation. The empirical sections do not repair this: there is no code release, no formal verification, and experiments compare against retraining without controlling for the test-label leakage ambiguity noted by the reader. Because the central claim depends on a false lemma, the preprint's theoretical contribution is unsupported, and a rejection or major revision is warranted.","tokens_in":17937,"tokens_out":3845,"duration_ms":38121,"concrete_test":"Use a scalar quadratic example LS(ω+ϵ)=0.5(ω+ϵ−a)² with constraint |ϵ|≤ρ and λ>0. Compute ˆϵ(ω) explicitly, solve min_ω [LS(ω+ˆϵ(ω)) + (λ/2)ω²], and verify whether ∇LS(ω*+ˆϵ(ω*))=0. It will not; the true condition is ∇LS(ω*+ˆϵ(ω*))=−λω*. Then re-derive dωδ/dδ at δ=0 from the full stationarity including λωδ and compare the result with Theorem 4.5's SAM-HIF formula. If the expressions differ, Theorem 4.5 does not follow from Eq. (1), settling whether the dropped λ term is load-bearing.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's SAM objective (Eq. 1) is argmin_ω [LS(ω+ˆϵ(ω)) + (λ/2)||ω||²], so the stationarity condition at ω* must contain λω*. Lemma 4.2 first states the condition ∇LS(ω*+ˆϵ(ω*)) + (dˆϵ(ω*)/dω)·∇LS(ω*+ˆϵ(ω*)) + λω* = 0, then asserts it is 'equivalent to' ∇LS(ω*+ˆϵ(ω*)) = 0. That equivalence is false for λ>0: the displayed condition with λω* ≠ 0 cannot be reduced to ∇LS = 0 unless ω* = 0. The proof of Theorem 4.3 in Appendix A.1 uses ∇LS(ω*+ˆϵ(ω*)) = 0 as the base point of a Taylor expansion, and the expansion (Eq. 11) omits the λ(ωδ−ω*) term that would arise from expanding λωδ. Theorem 4.5's SAM-HIF is derived from the same incorrect stationarity. Separately, Eq. (2) includes the chain-rule term dˆϵ(ω)/dω · ∇LS(ω+ˆϵ(ω)) in the gradient of the SAM objective, which conflicts with Danskin's theorem for a max over a compact ball; Danskin gives ∇LS(ω+ˆϵ(ω)) with no dϵ/dω term. Thus both the objective gradient and the stationarity reduction are questionable. Because every Taylor expansion in the Hessian-based formulas starts from this condition, the central closed-form contribution of the paper is not actually derived from the SAM objective as stated. This is the most load-bearing concern: if Lemma 4.2 is wrong, SAM-IF and SAM-HIF are approximations of a different optimization problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes two data-attribution methods for Sharpness-Aware Minimization (SAM): a Hessian-based influence function (SAM-HIF) and a gradient-trajectory influence function (SAM-GIF). The authors derive SAM-HIF as a closed-form approximation of the leave-one-out parameter change for SAM-trained models, and SAM-GIF as a checkpoint-based estimator that avoids Hessian inversion. They validate the methods on CIFAR-10/100, MNIST, MiniImageNet, HAM10000, CUB, and FOOD-101, reporting large runtime reductions relative to retraining and demonstrating applications in mislabeled-data detection, model editing, and interpretability. The central theoretical contribution is the claim that SAM-HIF and SAM-GIF accurately approximate LOO retraining for SAM without retraining.","tokens_in":18349,"tokens_out":9098,"duration_ms":95593,"significance":"If the derivations were sound, the paper would address a real and timely gap: existing influence-function methods do not directly handle SAM's bilevel structure. The proposed SAM-GIF in particular, being Hessian-free and trajectory-based, could be practically useful and scalable, and the experiments show order-of-magnitude speedups (e.g., CIFAR-10 retraining 3516 s vs. SAM-GIF 4.9 s). However, the paper's theoretical foundation has multiple load-bearing errors: a false stationarity equivalence, an incorrect gradient expression, an unquantified dropped term in the SAM-HIF derivation, and a sign inconsistency in SAM-GIF. These issues prevent the main closed-form claims from being accepted as derived from the stated SAM objective.","major_comments":[{"comment":"Lemma 4.2 states that the SAM stationarity condition is 'equivalent to' ∇LS(ω*+ε^(ω*))=0 after displaying a condition that contains λω* and a dε^(ω*)/dω term. This equivalence is false for λ>0 unless ω*=0 and the dε/dω term also vanishes. Equation (10) in Appendix A.1 then uses ∇LS(ω*+ε^(ω*))=0 as the base point for every Taylor expansion in Theorems 4.3 and 4.5, so the influence formulas are not actually derived from the optimization problem in Eq. (1). Additionally, Eq. (2) includes dε^(ω)/dω·∇LS(ω+ε^(ω)) in the gradient of the SAM objective, which conflicts with Danskin's theorem for a max over a compact ball; the correct Danskin gradient is ∇LS(ω+ε^(ω)) with no dε/dω term. These are not stylistic issues: the claimed equivalence and the gradient expression are the foundation of the subsequent derivations.","section":"Section 4.1, Lemma 4.2 and Appendix A.1, Eq. (10)"},{"comment":"The Taylor expansion of the stationarity condition omits the λ(ωδ−ω*) term that must appear when the objective contains (λ/2)||ω||². The text defines Hω = ∇²LS(ω*+ε^(ω*)) + λI, so the λI contribution is introduced in the final formula, but it is not present in the displayed expansion that is supposed to justify the formula. As written, the derivation is internally inconsistent: either the expansion should be performed on the regularized gradient, in which case λ(ωδ−ω*) appears, or the Hω definition should not include λI. This indeterminacy affects the central closed-form claim of Theorem 4.3.","section":"Appendix A.1, Eq. (11)"},{"comment":"The derivation of SAM-HIF drops the term −(Hω + Hω dε^(ω*)/dω)^{-1} Hω · dεδ(ω*)/dδ|_{δ=0} from dωδ/dδ, with the justification 'to enhance the computation efficiency.' However, this term exactly captures how the worst-case perturbation changes when sample k is up-weighted, which is the perturbation influence that SAM-HIF is claimed to model. No argument is given that this term is negligible, and in general it is of the same order as the retained term δ∇Lk. Consequently, SAM-HIF is not a comprehensive estimator of data influence as advertised; it is a further approximation whose error is unquantified.","section":"Appendix A.1, Eqs. (13)–(15), Theorem 4.5"},{"comment":"There is a sign inconsistency in the gradient-trajectory influence function. The derivation gives dω_{T,δ}/dδ|_{δ=0} = −Σ_t η_t ∇Lk(ωt+ε^(ωt)) (up to the discarded Hessian term), which implies that for δ=−1 (removal) one obtains ωk ≈ ω* + Σ_t η_t ∇Lk(ωt+ε^(ωt)). The theorem, however, defines SAM-GIFSGD = Σ_t η_t B_{k,t}∇Lk(ωt+ε^(ωt)) with a positive sign and then states ωk ≈ ω* − SAM-GIFSGD. This is the opposite sign from what the derivation implies. Earlier in Section 4.2 the quantity SAM-GIFGD is defined with a minus sign, so the text is internally inconsistent about the sign convention, and the parameter-update formula in Theorem 4.6 does not follow from the displayed derivation.","section":"Section 4.2, Theorem 4.6"},{"comment":"The interpretation of the influence score IS is incorrect. The theorem defines IS ≈ Σ_{(x,y)∈Dval} ℓ(x,y;ω*) − ℓ(x,y;ω*_{−k}); that is, IS > 0 means the loss before removal is higher than the loss after removal, so removal improves performance on the validation set. The sentence 'A positive IS indicates that removing the data point will deteriorate the model's performance' is therefore backwards. This sign error also propagates to the downstream use in Section 5.4, where low IS is used to identify harmful data; under the stated definition, harmful data should correspond to positive IS.","section":"Section 4.3.2, Theorem 4.8"}],"minor_comments":[{"comment":"The intermediate quantity εδ(ω*) is defined as an arg min, but it should be an arg max to match the perturbation objective.","section":"Appendix A.2, Lemma A.2 proof"},{"comment":"The text refers to 'Proposition 4.3' when the result is Theorem 4.3; the label is incorrect.","section":"Section 4.3.1"},{"comment":"The baseline is called 'TARK' in the text and tables, but reference [46] is the TRAK paper by Park et al.; the name should be corrected consistently.","section":"Section 5.2, Tables 2 and 3"},{"comment":"CUB and FOOD-101 are used in Table 3 but are not described in the dataset list in Section 5.1; please add their descriptions and the random-seed or train/test split details.","section":"Section 5.1 and Table 3"},{"comment":"The runtime column for retraining on MINI-ImageNet is 682.56 s while the accuracy is 0.6835; it would be helpful to state the number of epochs and architecture for that dataset, as the numbers appear to be on a different scale from CIFAR-10/100.","section":"Section 5.2, Table 1"}],"recommendation":"reject","confidential_remarks":"The paper's claimed novelty as the first to apply influence functions to SAM is plausible, but the theoretical derivation as presented is not sound. The false equivalence in Lemma 4.2, the missing λ term in the key expansion, and the sign inconsistency in SAM-GIF are load-bearing and would require a substantial reworking of the theory and likely re-running of the experiments. The empirical results are promising, but they cannot be interpreted as validating the proposed formulas until the derivation and sign conventions are corrected. There is also a possible related-work overlap with recent trajectory-based influence methods that is not discussed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: the paper's strongest contribution is SAM-GIF, which is essentially TracIn applied to SAM checkpoints, and the experiments there look reasonable. But the central theory—SAM-HIF—does not hold up. The derivation is built on a stationarity condition that is simply wrong.\n\nWhat is new: the idea of using influence functions for SAM-trained models is worth pursuing. SAM-GIF is a direct adaptation of Pruthi et al.'s gradient-tracing estimator to the SAM update rule, which uses gradients at perturbed weights. That's a small but legitimate extension, and the experiments show it works well at identifying noisy labels. SAM-HIF adds a perturbation-derivative term to the Hessian; that's not in the cited literature, so the formula is new. But the derivation does not support it.\n\nThe load-bearing problem is Lemma 4.2. The SAM objective includes λ/2 ||ω||², so the stationarity condition must contain λω*. The lemma writes the full condition, then asserts it is \"equivalent to\" ∇LS(ω*+ε(ω*))=0. That equivalence is false for λ>0. Every Taylor expansion in Theorems 4.3 and 4.5 starts from that base point, so the resulting SAM-IF and SAM-HIF formulas are not derived from the objective the paper states. There's also the issue that Eq. (2) includes a dε/dω term in the gradient, which conflicts with Danskin's theorem for the max over a compact ball; Danskin gives ∇LS(ω+ε*) without the chain-rule term. So the paper's own gradient formula is suspect.\n\nThe derivation of SAM-HIF is also not \"comprehensive\" as advertised: in Eq. (14), the authors drop a non-negligible term involving dε_δ(ω*)/dδ to get the closed form. They mention dropping it for \"computation efficiency,\" but that undercuts the claim of a closed-form comprehensive estimate. It's an approximation, and the paper should say so.\n\nThe experiments are more solid than the theory. They compare against retraining and show large speedups with similar accuracy, and the harmful-data detection results look credible. But the paper doesn't report code, and it's unclear whether the same test labels are used for scoring and evaluation in the mislabel-detection experiments. Those are fixable issues. The theory is not.\n\nBottom line: this is a useful empirical trick in search of a theory. A serious referee could push the authors to fix the stationarity condition and reframe SAM-HIF as a heuristic approximation rather than a derived closed form. As it stands, the central claim doesn't follow.\n\nRecommendation: send it to peer review, but expect major revision and a rewrite of Section 4.1.","headline":"SAM-GIF is a sensible reuse of TracIn for SAM and the experiments support it as a heuristic, but the paper's central SAM-HIF derivation relies on a false stationarity condition and needs major rework.","tokens_in":18884,"tokens_out":2591,"would_cite":false,"duration_ms":25778,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper derives two influence-function estimators for Sharpness-Aware Minimization, SAM-HIF and SAM-GIF, that approximate the leave-one-out retrained model and support mislabeled-data detection and model editing without retraining.","keywords":["data attribution","influence functions","sharpness-aware minimization","leave-one-out approximation","gradient trajectory","mislabeled data detection","model editing","Neumann series iHVP"],"falsifier":"Compute the gradient norm $\\|\\nabla L_S(\\omega^*+\\hat{\\epsilon}(\\omega^*))\\|$ at the end of a standard SAM run; if it is far from zero, the stationary-point premise fails. In parallel, retrain several SAM models with a single point deleted and correlate the measured $\\omega_k-\\omega^*$ with the SAM-HIF and SAM-GIF predictions; a correlation near zero would refute the central approximation.","tokens_in":1557,"feed_emoji":"🔍","tokens_out":1890,"duration_ms":78839,"temperature":0.7,"pith_summary":"The paper aims to make data attribution work for models trained with Sharpness-Aware Minimization, where removing one training point changes both the final weights and the loss-maximizing perturbation computed in the inner loop. It derives two estimators of what the model would become if a point were removed. The Hessian-based estimator uses only the trained weights and the perturbation at the optimum; the gradient-trajectory estimator sums the stored per-step gradients. The paper reports that both match retraining closely, identify mislabeled data with high precision, and update models in seconds instead of hours.","feed_headline":"SAM data attribution without retraining, two ways","feed_subtitle":"Hessian- and trajectory-based scores estimate leave-one-out effects for SAM models in seconds.","key_machinery":"The machinery is the leave-one-out (LOO) approximation built on Taylor expansion of the SAM optimality condition. Lemma 4.2 reduces the SAM stationary condition to $\\nabla L_S(\\omega^*+\\hat{\\epsilon}(\\omega^*))=0$; Theorem 4.5 then solves the expanded equation to obtain $\\mathrm{SAM\\text{-}HIF}$, and Theorem 4.6 sums the checkpoint gradients to obtain $\\mathrm{SAM\\text{-}GIF}_{\\mathrm{SGD}}$. The two formulas are the central objects: they convert the question 'what would retraining without this point produce?' into linear-algebra operations on quantities already available from training.","core_discovery":"On the paper's own terms, the central discovery is that the leave-one-out retrained SAM model $\\omega_k$ is accurately approximated by $\\omega^*$ minus one of two closed-form influence scores. $\\mathrm{SAM\\text{-}HIF}$ extends the classical influence function by including the derivative of the inner perturbation in the Hessian, giving $\\mathrm{SAM\\text{-}HIF}(x_k,y_k) = -(H_\\omega + H_\\omega\\, d\\hat{\\epsilon}(\\omega^*)/d\\omega)^{-1} \\nabla L_k^S(\\omega^*+\\hat{\\epsilon}(\\omega^*))$. $\\mathrm{SAM\\text{-}GIF}$ instead accumulates learning-rate-weighted gradients at the perturbed points along the stored training trajectory, giving $\\mathrm{SAM\\text{-}GIF}_{\\mathrm{SGD}}(x_k,y_k)=\\sum_{t=0}^{T-1}\\eta_t B_{k,t}\\nabla L_k^S(\\omega_t+\\hat{\\epsilon}(\\omega_t))$. These are presented as the first influence-based attribution tools for SAM, with experiments on CIFAR-10/100, MNIST, Mini-ImageNet, and HAM10000 showing that removing points flagged as harmful recovers accuracy from label noise, removing points flagged as valuable degrades accuracy like retraining, and the estimated edited parameters stay close to truly retrained parameters.","pith_inferences":["The same leave-one-out argument should extend to SAM variants that compute perturbations differently, because only the Jacobian $d\\hat{\\epsilon}(\\omega^*)/d\\omega$ changes in the Hessian-based formula.","If the reported accuracy holds at scale, these estimators offer a cheap deletion audit for privacy: a deployed SAM model could be adjusted for a removed user's data by subtraction rather than retraining.","The trajectory-based estimator's accuracy depends on how many checkpoints are stored; the paper's ablation suggests performance approaches retraining as checkpoints grow, implying a direct cost-accuracy trade-off in deployment.","A natural next test is whether the scores remain calibrated for large deletion fractions or shifted validation distributions, where the local Taylor assumptions are most strained."],"forward_implications":["Removing the points with the lowest influence scores improves test accuracy under label noise, and the paper reports detecting over 90% of noisy labels once roughly 40% of samples are removed.","Model editing becomes feasible without retraining: the edited parameters $\\omega^* - \\mathrm{IF}(x_k,y_k)$ approximate the retrained model, with accuracy close to a full retrain but runtime reduced from thousands of seconds to a few seconds.","SAM-HIF works when no training trajectory is stored, and the Neumann-series approximation of the inverse Hessian-vector product avoids storing the full Hessian matrix.","When checkpoints are available, SAM-GIF matches retraining more closely than the Hessian-based variant on CIFAR-10 and runs faster, since it avoids Hessian computation.","The same influence scores support interpretability by tracing misclassified test predictions back to the most helpful and most harmful training examples."],"supporting_citations":[{"why":"Supplies the SAM objective and the concrete training algorithm whose perturbation estimate and gradient update the attribution formulas target.","marker":"[16]"},{"why":"Supplies the classical influence-function approximation that the paper adapts to the SAM bilevel structure.","marker":"[34]"},{"why":"Supplies the gradient-trajectory influence method that SAM-GIF follows and extends to perturbed-point gradients.","marker":"[47]"},{"why":"Supplies the M-estimator foundation for influence functions, which motivates the leave-one-out parameter-change derivation.","marker":"[30]"},{"why":"Supplies the max-min optimality theorem used to simplify the SAM stationary condition in Lemma 4.2.","marker":"[12]"}],"fun_headline_variants":[],"cache_read_input_tokens":20864,"weakest_assumption_plain":"The load-bearing premise is Lemma 4.2 in Section 4.1: that a SAM optimum satisfies $\\nabla L_S(\\omega^*+\\hat{\\epsilon}(\\omega^*))=0$, which lets every later Taylor expansion start from a stationary point; if actual SAM training stops at points where this residual is large, the SAM-HIF and SAM-GIF formulas inherit that error.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3879,"prompt_tokens":1029,"completion_tokens":2850,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":2807}},"tokens_in":645,"tokens_out":2850,"duration_ms":29471,"temperature":1.0,"reasoning_tokens":2807,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:56:52.230156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the gradient norm $\\|\\nabla L_S(\\omega^*+\\hat{\\epsilon}(\\omega^*))\\|$ at the end of a standard SAM run; if it is far from zero, the stationary-point premise fails. In parallel, retrain several SAM models with a single point deleted and correlate the measured $\\omega_k-\\omega^*$ with the SAM-HIF and SAM-GIF predictions; a correlation near zero would refute the central approximation.","supporting_citations":[{"cited_title":"Understanding black-box predictions via influence functions","cited_arxiv_id":null,"evidence_quote":"Supplies the classical influence-function approximation that the paper adapts to the SAM bilevel structure."},{"cited_title":"Robust statistics","cited_arxiv_id":null,"evidence_quote":"Supplies the M-estimator foundation for influence functions, which motivates the leave-one-out parameter-change derivation."},{"cited_title":"The theory of max-min and its application to weapons allocation problems, volume 5","cited_arxiv_id":null,"evidence_quote":"Supplies the max-min optimality theorem used to simplify the SAM stationary condition in Lemma 4.2."}],"review_version":1}