REVIEW 4 major objections 6 minor 38 references
Entire-Space Variational Information Exploitation for Post-Click Conversion Rate Prediction
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read EVI uses a click-propensity-conditioned teacher to generate unbiased pseudo labels for unclicked samples, adds variational information exploitation and logit distillation, and reports the best CVR AUC on all six datasets.
desk verdict Useful empirical recipe with an overstated theoretical wrapper; worth reviewing if the authors fix the circular theorem and the undefined loss. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the conditional entire-space CVR teacher, a CVR network whose input representations are multiplied by a learned projection of the predicted click probability $\hat{o}_{u,i}$; this click-conditioned representation is what converts unclicked samples from biased guesswork into propensity-adjusted pseudo labels. The second piece is variational information exploitation (VIE): for each transferred layer pair $(t_k, s_k)$ of teacher and student, the student predicts the teacher activation with a Gaussian variational distribution, and the loss maximizes the variational lower bound on the mutual information $I(t_k; s_k)$ between the two networks. Logit distillation then carries the pseudo labels themselves. Together the teacher, the layer-wise information bound, and the propensity-weighted student loss are the machinery that the unbiasedness theorems and the reported gains rest on.
What would settle it
Train EVI with a deliberately corrupted CTR model, for example by shuffling or adding noise to the predicted click probabilities while keeping all other components unchanged, and compare the teacher's pseudo-label log-loss and the student's AUC on unclicked test samples against the accurate-CTR run; the theory predicts clear degradation, so a flat or reversed result would falsify the propensity-conditioning mechanism.
Extended reading notes
Core claim
The paper's central claim is that existing entire-space CVR methods transfer selection bias to the student because their pseudo-label generators discard or weaken click information, which is where the confounders of the click-conversion relationship live. EVI's conditional entire-space CVR teacher conditions on the predicted click propensity $\hat{o}_{u,i}$ by projecting it through a linear layer and taking the outer product with the teacher's representations, so the teacher models $P(r_{u,i}=1 \mid \hat{O}_{u,i}=\hat{o}_{u,i})$ rather than the marginal $P(r_{u,i}=1)$. Under an accurate CTR model this is exactly the propensity-score conditioning that makes the unobserved conversion outcome independent of click status, and Theorem 2 states that the teacher's pseudo labels are then unbiased. The student is trained over the whole impression space with inverse click propensity and inverse non-click propensity weighting, pseudo-label logit distillation, and a variational information exploitation loss that maximizes a lower bound on mutual information between teacher and student layer activations; Theorem 3 states that with accurate propensity and accurate pseudo labels the EVI loss equals the ideal full-space loss in expectation. The experimental section reports EVI as the top CVR method on all six datasets, with the best teacher log-loss on the non-click space and the lowest student mean bias in the three datasets tested.
Load-bearing premise
The load-bearing premise is that the model's predicted click probability captures every shared cause of clicking and converting, so that after conditioning on it, an unclicked sample's missing conversion outcome is statistically equivalent to a randomly missing one.
Editorial extensions
If this is right
- Unclicked impressions become usable training signal rather than discarded data, so EVI attacks data sparsity without reintroducing selection bias.
- If the propensity model is accurate, the full-space student loss is the ideal loss in expectation, meaning the method's training objective is aligned with its evaluation metric.
- The reported gains are systematic across datasets, best AUC on all six and best NLL on five, rather than an isolated win on one benchmark.
- The recipe is a drop-in multi-task architecture with shared embeddings, CTR and CTCVR heads, a teacher and a student, and five tunable loss weights, so production recommender systems can adopt it incrementally.
- The teacher's pseudo labels also improve calibration in the non-click space, lowering log-loss relative to the UKD and DDPO teachers, which strengthens downstream ranking quality.
Reading between the lines
- A natural stress test the paper does not run is to corrupt the CTR model, for example by adding noise to or shuffling the predicted click probabilities, and measure EVI's AUC degradation; the theory predicts a direct hit, and the paper's own conclusion says EVI relies on accurate CTR estimation.
- The Gaussian variational family is a modeling choice rather than a consequence of the theorems, so replacing it with a richer density could close more of the residual teacher-student information gap on sparse conversion events.
- Because the paper flags sensitivity to noisy clicked samples, combining EVI with label-noise-robust losses or sample reweighting on the click space is the obvious next step and would test whether the bias reduction survives real-world click noise.
- The conditioning mechanism uses only the scalar click probability, so conditioning on a richer click-side context embedding might weaken the unconfoundedness assumption in practice while preserving the spirit of the proof.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes EVI, an entire-space CVR estimation framework that combines a click-propensity-conditioned teacher (CECT), variational information exploitation (VIE) at the feature level, and logit distillation with pseudo labels for unclicked samples. The authors claim that the conditional teacher produces unbiased pseudo labels, that the EVI estimator is unbiased under the stated conditions, and that EVI achieves the best CVR AUC and NLL on six datasets, with an average AUC improvement of 2.25% over the best baselines. The paper includes theoretical statements (Theorems 1-3), ablations, parameter sensitivity studies, and a public code repository.
Significance. If the empirical claims hold, EVI offers a practically useful combination of techniques for a well-known problem: entire-space CVR estimation with sample selection bias and data sparsity. The paper provides a comparatively broad evaluation across five public and one industrial dataset, includes ablations isolating the two proposed components, and releases code to support reproducibility. However, the theoretical support for the central 'unbiased pseudo-label' claim is much weaker than presented: the key theorem relies on an assumption that the pseudo labels equal the true labels, and the teacher loss in Eq. (14) is underspecified for unclicked samples. The headline average improvement of 2.25% is also not consistently supported by the numbers in Table 2. These issues are load-bearing because the paper's novelty is framed largely in terms of theoretical unbiasedness.
major comments (4)
- [Section 4.4, Eq. (14)] The CVR teacher loss in Eq. (14) is written as a sum over the entire space D, using the true conversion label r_{u,i}. For unclicked samples (u,i) in N, r_{u,i} is unobserved by definition. The manuscript does not specify whether these terms are masked, assigned r=0, or replaced by pre-existing pseudo-labels. As written, the objective cannot be computed, and if the unclicked terms are silently set to zero or assigned r=0, the 'entire-space' teacher may reintroduce the very selection bias that Section 3 motivates. This must be clarified and the actual training procedure stated precisely.
- [Appendix B.3, Theorem 3] The proof of Theorem 3 assumes r*_{u,i} = r_{u,i} for unclicked samples and ED[o_{u,i}] = ED[\hat{o}_{u,i}] to reach the conclusion. In EVI, r* is the teacher's predicted probability, a continuous value in [0,1], so r* = r holds only in degenerate cases. The theorem therefore assumes the exact property that the paper claims to establish: that the teacher's pseudo-labels are accurate for unclicked samples. This makes the unbiasedness claim circular. The theorem should be restated as a conditional result (e.g., 'if the teacher is well-calibrated and the propensity model is correctly specified, then ...'), or the proof needs a substantive argument that the EVI teacher achieves r* = r in the relevant limit.
- [Appendix B.2, Theorem 2] Theorem 2 relies on Rosenbaum and Rubin's propensity-score result, which requires conditioning on the true propensity score. The paper instead conditions on the predicted click propensity \hat{o}_{u,i} and states that unbiasedness holds 'provided that \hat{o}_{u,i} is accurate.' The accuracy of an estimated pCTR is exactly the unconfoundedness assumption that is unverifiable from observational data. The paper's own conclusion acknowledges that EVI 'relies on accurate CTR estimation,' but no experiment or sensitivity analysis varies CTR quality or otherwise probes the accuracy of the propensity model. The theoretical claim should be weakened to a conditional statement, and a sensitivity analysis should be added if the authors wish to substantiate it empirically.
- [Section 5.2, Table 2] The average AUC improvement of 2.25% reported in the abstract, the introduction, and Section 5.2 does not match the numbers in Table 2. For example, the AE-US improvement over the best baseline (0.8057 vs. 0.7889) is approximately 2.13%, not 3.13%; the AE-NL improvement (0.7741 vs. 0.7685) is approximately 0.73%, not 4.84%. Recomputing the per-dataset relative improvements from Table 2 gives an average of roughly 1.4%, not 2.25%. The same check applies to the claimed NLL reductions. The text and the table need to be reconciled, and the per-dataset improvement numbers in the bullet list should be corrected or the computation should be explained.
minor comments (6)
- [Section 5.2] The text refers to 'AR-ES' when the table and Section A.1 use 'AE-ES'; this typo should be fixed.
- [Section 3.2, Eqs. (8) and (15)] There is a parenthesis mismatch in Eq. (8) and Eq. (15): the term for unclicked samples is written as (1 - o_{u,i})\delta(...), but the opening bracket only encloses part of the expression. The formulas should be rewritten cleanly.
- [Section 4.3] The phrase 'Kullback-Leiber divergence' should be 'Kullback-Leibler divergence'.
- [Section 5.3] The text mentions 'EVT-teacher' in the discussion of Figure 2, but the model is called EVI; this typo should be corrected.
- [Appendix A.2] The list of comparison methods says '7 representative baseline models' but lists eight methods (ESMM, ESCM2-IPW, ESCM2-DR, MRDR-GPL, DR-V2, UKD, DCMT, DDPO); the count should be updated.
- [General] The definition of the variational distribution q(t,s) in Section 4.3 says 'we employ Gaussian distribution with mean µ and ρ', but ρ is the standard deviation in Eq. (11); the notation should be clarified so that it is clear that ρ enters through the variance term.
Circularity Check
Unbiasedness proof assumes pseudo-labels equal true labels, making the theoretical support for EVI's central claim circular; empirical results remain independent.
-
other
[Appendix B.3, Proof of Theorem 3 (used by Section 4.2 and Eq. 15)]
"where (1) holds given that the click propensity ˆou,i and the pseudo conversion labels of unclicked samples r∗u,i are accurate (Theorem 2), i.e., ED[ou,i] =ED[ˆou,i], r∗u,i = ru,i."
The proof's key reduction is to replace δ(r*_{u,i}, \hat r_{u,i}) with δ(r_{u,i}, \hat r_{u,i}) for unclicked samples, i.e. to assume r*_{u,i} = r_{u,i}. But r* is the CVR teacher's predicted probability; pointwise equality to the unobserved binary label is exactly the 'accurate pseudo labels' property that the paper claims EVI's conditional teacher provides (Abstract, Section 4.2). Theorem 2 establishes at most an expectation-level propensity-score independence under an accurate-CTR assumption, not pointwise equality of predictions to unobserved labels. Therefore the theorem derives EVI's unbiasedness by assuming the very property the method is supposed to deliver; the theoretical support is conditional on its own conclusion.
full rationale
The empirical core of the paper is self-contained: EVI is evaluated against eight baselines on six datasets, and the ablation study (Table 1) shows that both the conditional teacher and the variational loss contribute to the reported AUC gains. These comparisons do not reduce to the paper's assumptions and provide independent support for the method's practical value. The circularity is confined to the theoretical unbiasedness claim. Theorem 3's proof substitutes the assumption r*_{u,i}=r_{u,i} for the unclicked-sample pseudo-labels, which is exactly the accuracy property the CVR teacher is claimed to produce; Theorem 2 only supplies conditional independence given an accurate propensity model and does not yield pointwise equality to unobserved binary labels. The teacher's own objective (Eq. 14) is also underspecified because true post-click labels r_{u,i} do not exist for unclicked rows; the paper does not state whether those rows are masked, zero-labeled, or filled with the same pseudo-labels the teacher is meant to generate. This prevents Eq. 14 from serving as an external derivation of the teacher's accuracy. No self-citation chain is load-bearing, and no empirical result is forced by construction. Overall, the central theoretical derivation is partially circular, while the empirical contribution remains independent, giving a score of 6 on the circularity scale.
Assumptions & free parameters
free parameters (4)
- loss trade-off weights lambda_c, lambda_t, lambda_r, lambda_i, lambda_g =
Ali-CCP: (1, 0.2, 2, 0.2, 0.1); other datasets: (0.2, 0.2, 2, 0.2, 0.2)
- number of transfer layers for variational information transfer =
3
- variational distribution parameters mu and rho =
learned per dimension, passed through softplus
- embedding size, expert dimension, number of experts, MLP hidden sizes =
5, 256, 8, [128, 64, 32]
assumptions (4)
- domain assumption Unconfoundedness given the propensity score: {r(1), r(0)} independent of click status conditional on \hat{o}.
- domain assumption The CTR model is accurate, i.e., \hat{o}_{u,i} equals the true click propensity.
- domain assumption Pseudo conversion labels for unclicked samples are accurate, r*_{u,i} = r_{u,i}.
- standard math The variational lower bound for mutual information and the KL decomposition are valid (standard information theory, Barber and Agakov 2004; Ahn et al. 2019).
Cite this review
Pith. "Pith review of Entire-Space Variational Information Exploitation for Post-Click Conversion Rate Prediction." pith.science (2026). https://pith.science/paper/DL2CRRMI
@misc{pith2026250215687,
author = {Pith},
title = {Pith review of: Entire-Space Variational Information Exploitation for Post-Click Conversion Rate Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/DL2CRRMI}},
note = {Machine review of arXiv:2502.15687}
}
read the original abstract
In recommender systems, post-click conversion rate (CVR) estimation is an essential task to model user preferences for items and estimate the value of recommendations. Sample selection bias (SSB) and data sparsity (DS) are two persistent challenges for post-click conversion rate (CVR) estimation. Currently, entire-space approaches that exploit unclicked samples through knowledge distillation are promising to mitigate SSB and DS simultaneously. Existing methods use non-conversion, conversion, or adaptive conversion predictors to generate pseudo labels for unclicked samples. However, they fail to consider the unbiasedness and information limitations of these pseudo labels. Motivated by such analysis, we propose an entire-space variational information exploitation framework (EVI) for CVR prediction. First, EVI uses a conditional entire-space CVR teacher to generate unbiased pseudo labels. Then, it applies variational information exploitation and logit distillation to transfer non-click space information to the target CVR estimator. We conduct extensive offline experiments on six large-scale datasets. EVI demonstrated a 2.25\% average improvement compared to the state-of-the-art baselines.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Ahn, S.; Hu, S. X.; Damianou, A.; Lawrence, N. D.; and Dai, Z. 2019. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9163--9171
work page 2019
-
[4]
Barber, D.; and Agakov, F. 2004. The im algorithm: a variational approach to information maximization. Advances in neural information processing systems, 16(320): 201
work page 2004
-
[5]
Dai, Q.; Li, H.; Wu, P.; Dong, Z.; Zhou, X.-H.; Zhang, R.; Zhang, R.; and Sun, J. 2022. A generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 252--262
work page 2022
-
[6]
Fawcett, T. 2006. An introduction to ROC analysis. Pattern recognition letters, 27(8): 861--874
2006
-
[7]
Feller, W. 1991. An introduction to probability theory and its applications, Volume 2, volume 81. John Wiley & Sons
work page 1991
-
[8]
Ganin, Y.; Ustinova, E.; Ajakan, H.; Germain, P.; Larochelle, H.; Laviolette, F.; March, M.; and Lempitsky, V. 2016. Domain-adversarial training of neural networks. Journal of machine learning research, 17(59): 1--35
2016
Show all 38 references
-
[9]
Goldberger; and Greenspan. 2003. An efficient image similarity measure based on approximations of KL-divergence between two Gaussian mixtures. In Proceedings Ninth IEEE International conference on computer vision, 487--493. IEEE
2003
-
[10]
Guo, S.; Zou, L.; Liu, Y.; Ye, W.; Cheng, S.; Wang, S.; Chen, H.; Yin, D.; and Chang, Y. 2021. Enhanced Doubly Robust Learning for Debiasing Post - Click Conversion Rate Estimation . In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in I...
2021
-
[11]
A.; Jordan, M
Jacobs, R. A.; Jordan, M. I.; Nowlan, S. J.; and Hinton, G. E. 1991. Adaptive mixtures of local experts. Neural computation, 3(1): 79--87
1991
-
[12]
P.; and Ba, J
Kingma, D. P.; and Ba, J. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980
2017 arXiv
-
[13]
Li, H.; Xiao, Y.; Zheng, C.; and Wu, P. 2023 a . Balancing unobserved confounding with a few unbiased ratings in debiased recommendations. In Proceedings of the ACM Web Conference 2023, 1305--1313
2023
-
[14]
Li, H.; Xiao, Y.; Zheng, C.; Wu, P.; and Cui, P. 2023 b . Propensity matters: Measuring and enhancing balancing for recommendation. In International Conference on Machine Learning, 20182--20194. PMLR
2023
-
[15]
Luo, H.; Zhuang, F.; Xie, R.; Zhu, H.; Wang, D.; An, Z.; and Xu, Y. 2024. A survey on causal inference for recommendation. The Innovation
2024
-
[16]
Ma, J.; Zhao, Z.; Yi, X.; Chen, J.; Hong, L.; and Chi, E. H. 2018 a . Modeling Task Relationships in Multi -task Learning with Multi -gate Mixture -of- Experts . In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 1930--1939. L...
2018
-
[17]
Ma, X.; Zhao, L.; Huang, G.; Wang, Z.; Hu, Z.; Zhu, X.; and Gai, K. 2018 b . Entire space multi-task model: An effective approach for estimating post-click conversion rate. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , 1137--1140
2018
-
[18]
Mao, A.; Mohri, M.; and Zhong, Y. 2023. Cross-entropy loss functions: Theoretical analysis and applications. In International conference on Machine learning, 23803--23828. PMLR
2023
-
[19]
M.; and Zemel, R
Marlin, B. M.; and Zemel, R. S. 2009. Collaborative prediction and ranking with non-random missing data. In Proceedings of the third ACM conference on Recommender systems, 5--12
2009
-
[20]
Pearl, J. 2009. Causal inference in statistics: An overview. Statistics Surveys
2009
-
[21]
pengcheng Li; Li, R.; Da, Q.; Zeng, A.-X.; and Zhang, L. 2020. Improving Multi-Scenario Learning to Rank in E-commerce by Exploiting Task Relationships in the Label Space. In proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 202...
2020
-
[22]
R.; and Rubin, D
Rosenbaum, P. R.; and Rubin, D. B. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1): 41--55
1983
-
[23]
Rubin, D. B. 2005. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469): 322--331
2005
-
[24]
Saito, Y. 2020. Doubly robust estimator for ranking metrics with post-click conversions. In Proceedings of the 14th ACM Conference on Recommender Systems, 92--100
2020
-
[25]
Saito, Y.; Yaginuma, S.; Nishino, Y.; Sakata, H.; and Nakata, K. 2020. Unbiased recommender learning from missing-not-at-random implicit feedback. In Proceedings of the 13th International Conference on Web Search and Data Mining, 501--509
2020
-
[26]
Schnabel, T.; Swaminathan, A.; Singh, A.; Chandak, N.; and Joachims, T. 2016. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning, 1670--1679. PMLR
2016
-
[27]
Su, H.; Meng, L.; Zhu, L.; Lu, K.; and Li, J. 2024. DDPO: Direct Dual Propensity Optimization for Post-Click Conversion Rate Estimation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1179--1188
2024
-
[28]
Wang, H.; Chang, T.-W.; Liu, T.; Huang, J.; Chen, Z.; Yu, C.; Li, R.; and Chu, W. 2022 a . Escm2: Entire space counterfactual multi-task model for post-click conversion rate estimation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development i...
2022
-
[29]
Wang, Z.; Shen, S.; Wang, Z.; Chen, B.; Chen, X.; and Wen, J.-R. 2022 b . Unbiased sequential recommendation with latent confounders. In Proceedings of the ACM Web Conference 2022, 2195--2204
2022
-
[30]
Wen, H.; Zhang, J.; Lv, F.; Bao, W.; Wang, T.; and Chen, Z. 2021. Hierarchically modeling micro and macro behaviors via multi-task learning for conversion rate prediction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information ...
2021
-
[31]
Wen, H.; Zhang, J.; Wang, Y.; Lv, F.; Bao, W.; Lin, Q.; and Yang, K. 2020. Entire space multi-task modeling via post-click behavior decomposition for conversion rate prediction. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Inform...
2020
-
[32]
Xi, D.; Chen, Z.; Yan, P.; Zhang, Y.; Zhu, Y.; Zhuang, F.; and Chen, Y. 2021. Modeling the sequential dependence among audience multi-step conversions with multi-task learning in targeted display advertising. In Proceedings of the 27th ACM SIGKDD , 3745--3755
2021
-
[33]
Xu, Z.; Wei, P.; Zhang, W.; Liu, S.; Wang, L.; and Zheng, B. 2022. Ukd: Debiasing conversion rate estimation via uncertainty-regularized knowledge distillation. In Proceedings of the ACM Web Conference 2022, 2078--2087
2022
-
[34]
Zhang, D.; Wu, H.; Zeng, G.; Yang, Y.; Qiu, W.; Chen, Y.; and Hu, H. 2022. CTnoCVR: A Novelty Auxiliary Task Making the Lower-CTR-Higher-CVR Upper. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2272--2276
2022
-
[35]
Zhang, W.; Bao, W.; Liu, X.-Y.; Yang, K.; Lin, Q.; Wen, H.; and Ramezani, R. 2020. Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning. In Proceedings of The Web Conference 2020 , 2775--2781
2020
-
[36]
Zheng, H.; Yang, Z.; Liu, W.; Liang, J.; and Li, Y. 2015. Improving deep neural networks using softplus units. In 2015 International joint conference on neural networks (IJCNN), 1--4. IEEE
2015
-
[37]
Zhou, Y.; Feng, T.; Liu, M.; and Zhu, Z. 2023. A Generalized Propensity Learning Framework for Unbiased Post-Click Conversion Rate Estimation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 3554--3563
2023
-
[38]
Zhu, F.; Zhong, M.; Yang, X.; Li, L.; Yu, L.; Zhang, T.; Zhou, J.; Chen, C.; Wu, F.; Liu, G.; et al. 2023. DCMT: A Direct Entire-Space Causal Multi-Task Framework for Post-Click Conversion Estimation. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), 3113-...
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.