REVIEW 4 major objections 4 minor 1 cited by
Delayed Feedback Modeling with Influence Functions
T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Influence functions can update a deployed conversion-rate model in about 15 seconds, absorbing delayed conversions and new behavior data without retraining.
desk verdict A plausible, clearly derived application of influence functions to delayed-feedback CVR, with a correct-looking quadratic reformulation, but the load-bearing finite-epsilon approximation and the Hessian assumptions are not validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the influence function of the vanilla-trained model, evaluated at the empirical minimizer $\hat\theta$, whose parameter-change formula $\Delta\theta \approx -\epsilon H_{\hat\theta}^{-1}\nabla_\theta L(z,\hat\theta)$ is the standard result from the influence-function literature. The paper's specific machinery is the reformulation of the required inverse-Hessian-vector product as the solution of the finite-sum quadratic program (Eqs. 18-20), where each summand uses the per-sample Hessian $\nabla_\theta^2 L_{\text{BCE}}(z_i,\hat\theta)$ and a shared vector $b$ built from gradients of label-reversed and newly added samples. Because $\hat\theta$ is a minimizer of the vanilla loss, the Hessian is positive semidefinite, making the quadratic program convex and solvable by stochastic optimizers such as Adam, with auto-differentiation and Hessian-vector products supplying each $f_i$. This replaces direct Hessian inversion with a scalable optimization problem, which is what makes the delayed-feedback update practical.
What would settle it
Retrain a backbone (for example, MLP) on the Criteo data with corrected labels for a fixed delay window, then compare the retrained model's test AUC and parameter vector with those of the IF-DFM-updated vanilla model; if they are not close (the AUC gap is not roughly 85 percent of the vanilla-retrain gap, or the parameter vectors differ greatly), the first-order influence approximation is not the mechanism behind the reported gains.
Extended reading notes
Core claim
IF-DFM's central claim is that the effect of delayed feedback on a trained CVR model can be captured, to a good approximation, by a first-order influence function acting on the vanilla loss. A label reversal from 0 to 1 for a sample $z_i$ is represented as a perturbation with $\delta=1$, and a newly arrived sample $z$ is integrated with effective weight $\epsilon=1/(n+1)$; both are combined into a single parameter displacement $\Delta\theta_{\text{total}} = \Delta\theta_{\text{delay}} + \Delta\theta_{\text{add}}$ given by the inverse Hessian of the vanilla loss times a sum of gradient differences (Eq. 14). The paper then converts this inverse-Hessian-vector product into the solution of the convex quadratic program $\min_{\Delta} F(\Delta) = \frac{1}{2}\Delta^\top \nabla_\theta^2 L_V(\hat\theta)\Delta - \langle b, \Delta\rangle$, whose objective has a finite-sum structure, enabling scalable solution by Adam. On the Criteo and Taobao datasets, with MLP, DeepFM, AutoInt, and DCNV2 backbones, the updated model is reported to outperform thirteen baselines across AUC, PRAUC, and log loss, close about 85 percent of the vanilla-to-retrain AUC gap, and do so in about 14.8 seconds of additional computation.
Load-bearing premise
The method assumes that a first-order influence approximation, which is exact only for tiny data changes, stays accurate when applied to full label corrections and new samples at realistic sample sizes, so the computed parameter update actually matches what retraining would produce.
Editorial extensions
If this is right
- A deployed CVR model can be refreshed within seconds after each delay window by a direct parameter update, rather than waiting for a full retraining cycle.
- The approach removes the need for auxiliary delay-distribution models, label-correction networks, or duplicated fake-negative samples, simplifying the training pipeline.
- The same update mechanism works in offline and online settings and across MLP, DeepFM, AutoInt, and DCNV2 backbones, so it can be applied to existing deployed models.
- Because the update closes roughly 85 percent of the vanilla-to-retrain metric gap, models can track evolving user interest with less label staleness between retraining periods.
Reading between the lines
- The paper does not directly compare the influence-updated parameters to the parameters produced by actual retraining; measuring that distance (for example, cosine similarity or prediction alignment) would test whether the reported gains come from the approximation or from the small update step itself.
- The first-order influence formula is exact only for infinitesimal reweighting, and the paper itself cites work showing influence functions can be fragile in deep learning; applying it at $\epsilon=1/n$ for full label flips deserves an error bound or a perturbation test that the paper does not provide.
- The finite-sum quadratic reformulation is essentially a scalable iterative solver for inverse-Hessian-vector products, so the same trick could be reused in other influence-function applications, such as data debugging, unlearning, or explainability, where full Hessian inversion is currently a bottleneck.
- The reported 14.8-second update time and the approximation quality likely depend on the number of label-reversed and newly arrived samples and on the Hessian's condition number; a reader should expect these to trade off when the method is scaled to much larger or more non-convex models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes IF-DFM, a delayed-feedback modeling method for CVR prediction. Instead of retraining when delayed conversions arrive, IF-DFM estimates the parameter change induced by (i) flipping fake-negative labels to positive and (ii) incorporating newly arrived behavioral data, using a first-order influence-function approximation. The inverse Hessian-vector product is reformulated as a finite-sum quadratic optimization problem and solved with ADAM. Experiments on Criteo and Taobao with four backbones compare IF-DFM against 13 baselines and report that IF-DFM significantly outperforms baselines and closes 85.16% of the AUC gap between Vanilla and Retrain.
Significance. If the central approximation is valid, this is a practically valuable contribution: it offers a way to update a deployed CVR model in about 15 seconds without retraining or auxiliary models. The paper provides experiments on two large-scale datasets across multiple backbones and baselines, and the algebraic reduction of the inverse Hessian-vector product to a finite-sum quadratic problem is a useful computational idea. However, the validity of the first-order influence approximation for finite label flips is not established, and the quadratic reformulation has a well-posedness gap when the Hessian is singular. The paper itself cites Basu et al. (2021) on the fragility of influence functions in deep learning but does not address that conflict. These issues are load-bearing because every reported gain is attributed to the approximate parameter update Δθ_total.
major comments (4)
- [§Method, Eqs. (7)–(14)] The core update rule is a first-order influence approximation evaluated at ε=1/n for finite label reversals and new-sample additions. The paper cites Basu et al. (2021) showing that influence functions in deep learning can be fragile, but it does not explain why the present setting is immune, nor does it provide an error bound or an empirical check. In particular, there is no comparison between Δθ_IF and the actual retrained parameter difference Δθ_retrain, and no sensitivity study in ε. Since all experimental gains are attributed to Δθ_total, this missing validation is load-bearing for the claim that IF-DFM 'approaches the performance of Retrain'.
- [§An Efficient and Scalable Method, Eq. (18)] The assertion that positive semidefiniteness of the Hessian implies that Δθ_total is the solution of a convex quadratic program is not sufficient. If b ∉ range(H), the objective F(Δ) is unbounded below along directions in null(H), so Algorithm 1 has no well-defined target. The earlier influence-function derivation requires H to be invertible; replacing H^{-1} by a PSD quadratic solve does not remove the singularity issue. The authors should verify b ∈ range(H), add damping or regularization, solve a least-squares/min-norm formulation, or report relevant spectral information about the Hessian at the practical solution.
- [§Experiment, Tables 2–3 and Figures 3–4] The evaluation reports only point estimates. The double asterisks indicate a t-test with p≤0.05, but the number of independent runs, standard deviations, and confidence intervals are not reported, making the significance claim unverifiable. Please report these statistics for the main comparisons, including the relative-improvement numbers that support the central claim that IF-DFM closes most of the gap to Retrain.
- [§Method, Eq. (14) and online setting] The new-data term in Eq. (14) includes newly arrived samples z_k with their observed labels as if those labels were ground truth. However, a click observed in [T,T′] without a conversion by T′ is itself a potential fake negative because conversions may be delayed beyond T′. The paper does not model or correct this, and the definition of the new-data perturbation should be stated with the same label-reversal logic used for old samples, or the assumption should be justified explicitly.
minor comments (4)
- [Throughout] There are multiple typos and formatting issues, including 'Auxiliay Model' in Figure 1, 'i nfluence f unction' in the introduction, 'approximate∆θtotal' missing a space, and 'to some extend' in the related work.
- [§Efficiency of Calculating Parameter Changes] The text says runtime results are shown in Figure 4, but the actual runtime table is Table 4. Also clarify what the 'Vanilla + Ours' time includes (e.g., whether the 14.8 seconds includes data preprocessing and constructing b).
- [References and Appendix] The manuscript repeatedly refers to an Appendix for baseline descriptions, implementation details, additional results, and the derivation of Eq. (11), but no appendix is present in the submitted version. Either include the appendix or remove the references.
- [References] The Nocedal and Wright reference is incomplete: the year and publisher are listed as '????'.
Circularity Check
No significant circularity: IF-DFM's update consumes genuinely new labels from [T,T'] and is evaluated on a future window; the only self-citations are non-load-bearing contextual references.
full rationale
The central derivation (Eqs. 7-14) is self-contained: it takes the trained Vanilla model, computes gradients of label-reversed samples (J) and newly arrived samples (K), and solves HΔ=b. All quantities come from the interval [T,T'] and the evaluation is on the disjoint future interval [T',T'+dtest], so the test metric is external to the update construction. No parameter is fitted to the test metric; the approximation has no free constants beyond the fixed ε=1/n choice dictated by the reweighting. The RI metric (Eq. 21) normalizes by the Retrain oracle, and the paper explicitly designs IF-DFM as a first-order approximation of retraining; therefore a high RI is a consistency check of that approximation, not a prediction forced by definition. The self-citations (e.g., Wu et al. 2023 for influence-based unlearning) are contextual and non-load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The acknowledged limitations (reliance on gradient information, full training set) and the mathematical concerns about finite-ε accuracy and the PSD quadratic program (Eq. 18) are correctness risks, not circularity.
Assumptions & free parameters
free parameters (2)
- ADAM optimizer hyperparameters for solving Eq (19) =
not reported
- Temporal split parameters c and dtest =
c in {5, 10, 14}; dtest not stated explicitly
assumptions (5)
- standard math Influence function formula (6) and its derivation
- ad hoc to paper First-order influence approximation is accurate at epsilon = 1/n for deep CVR models
- domain assumption Vanilla model theta_hat is a local minimum with positive semidefinite Hessian
- domain assumption Label reversal is representable as unit perturbation delta = 1 of the observed label
- domain assumption The sets J and K are known at update time
Cite this review
Pith. "Pith review of Delayed Feedback Modeling with Influence Functions." pith.science (2026). https://pith.science/paper/6NUFN3EP
@misc{pith2026250201669,
author = {Pith},
title = {Pith review of: Delayed Feedback Modeling with Influence Functions},
year = {2026},
howpublished = {\url{https://pith.science/paper/6NUFN3EP}},
note = {Machine review of arXiv:2502.01669}
}
read the original abstract
In online advertising under the cost-per-conversion (CPA) model, accurate conversion rate (CVR) prediction is crucial. A major challenge is delayed feedback, where conversions may occur long after user interactions, leading to incomplete recent data and biased model training. Existing solutions partially mitigate this issue but often rely on auxiliary models, making them computationally inefficient and less adaptive to user interest shifts. We propose IF-DFM, an \underline{I}nfluence \underline{F}unction-empowered for \underline{D}elayed \underline{F}eedback \underline{M}odeling which estimates the impact of newly arrived and delayed conversions on model parameters, enabling efficient updates without full retraining. By reformulating the inverse Hessian-vector product as an optimization problem, IF-DFM achieves a favorable trade-off between scalability and effectiveness. Experiments on benchmark datasets show that IF-DFM outperforms prior methods in both accuracy and adaptability.
Figures
Forward citations
Cited by 1 Pith paper
-
Unified Parameter-Efficient Unlearning for LLMs
An influence-function-based parameter editing framework performs instance removal, query modification, and response correction on PEFT adapters without retraining.
Reference graph
Works this paper leans on
-
[4]
A Practical Framework of Conversion Rate Prediction for Online Display Advertising. In ADKDD@KDD, 9:1– 9:9. Ma, X.; Zhao, L.; Huang, G.; Wang, Z.; Hu, Z.; Zhu, X.; and Gai, K. 2018. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. In SIGIR, 1137–1140. Nocedal, J.; and Wright, S. J. ???? Numerical optimization...
work page 2018
-
[5]
GIF: A General Graph Unlearning Strategy via Influ- ence Function. In WWW, 651–661. Yang, J.; Li, X.; Han, S.; Zhuang, T.; Zhan, D.; Zeng, X.; and Tong, B. 2021. Capturing Delayed Feedback in Conver- sion Rate Prediction via Elapsed-Time Sampling. In AAAI, 4582–4589. Yasui, S.; and Kato, M. 2022. Learning Classifiers under Delayed Feedback with a Time Win...
arXiv 2021
-
[2017]
In Proceedings of the 26th international conference on world wide web companion , 689–698
Model ensemble for click prediction in bing search ads. In Proceedings of the 26th international conference on world wide web companion , 689–698. Liu, J.; Xue, M.; Lou, J.; Zhang, X.; Xiong, L.; and Qin, Z
-
[2023]
MUter: Machine Unlearning on Adversarially Trained Models. In ICCV, 4869–4879. Lu, Q.; Pan, S.; Wang, L.; Pan, J.; Wan, F.; and Yang, H
-
[2024]
Complex & Intelligent Systems, 1–13
A counterfactual explanation method based on modi- fied group influence function for recommendation. Complex & Intelligent Systems, 1–13. Guo, Y .; Li, H.; Ao, X.; Lu, M.; Liu, D.; Xiao, L.; Jiang, J.; and He, Q. 2022. Calibrated Conversion Rate Prediction via Knowledge Distillation under Delayed Feedback in Online Advertising. In CIKM, 3983–3987. Hampel,...
work page 2022
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.