{"id":"4b86926c-002b-4cb2-ade4-2dbf19a9ec3b","arxiv_id":"2502.15687","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A click-propensity-conditioned teacher plus variational mutual information and logit distillation yields the best reported CVR AUC on all six datasets tested.","lead":"This paper presents EVI, a framework for predicting post-click conversion rates that trains on all impressions by using a click-propensity-conditioned teacher and a variational information transfer loss. It reports AUC gains over existing entire-space and causal methods on six large-scale datasets, which matters because conversion prediction is central to ad and recommendation economics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unbiasedness theorem is vacuous: it assumes pseudo-labels equal true labels (r*=r), so it assumes the very property to be proved, and Eq. 14's loss over unclicked samples is undefined because true post-click labels are missing.","rationale":"The reader's conditional verdict is the right one, but the sharpest problem is not merely the unconfoundedness assumption; it is that the theoretical argument is vacuous at its hinge. Theorem 3's condition 'r* accurate' is defined as r*=r, which a soft-label teacher cannot satisfy. This is not a matter of empirical approximation: the theorem's proof requires exact equality. Also Eq. 14 is undefined on unclicked rows, so the method's core component is underspecified. These are internal correctness risks, not disagreements with consensus. The empirical tables do show EVI with the highest AUC on all six datasets and ablations supporting both components, so the method may still work as a practical recipe; the theory, however, overclaims. A synthetic test with ground truth and instrumented code would settle whether the implementation actually trains an entire-space teacher and whether pseudo-labels are biased. I keep the reader's CONDITIONAL verdict because the empirical contribution is plausible and the theoretical flaws are fixable in revision, but the paper should not be accepted without addressing them.","tokens_in":14607,"tokens_out":12343,"duration_ms":109467,"concrete_test":"Run the released EVI code (https://github.com/q1179897215/EVI) on a synthetic dataset with known ground-truth P(r=1|o=1,x) and instrument the CVR-T loss (Eq. 14) to log, for every unclicked sample, the label actually used and whether the sample contributes a gradient. Then compare the teacher's pseudo-labels on unclicked test samples to the ground-truth conversion probability. If unclicked samples are labeled 0 or excluded, the teacher is not an entire-space CVR model; if the pseudo-labels differ from ground truth for any non-degenerate prediction, the r*=r assumption in Theorem 3 is never satisfied and the unbiasedness proof fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim that EVI generates unbiased pseudo labels and an unbiased CVR estimator is not actually established. Theorem 3 states that the EVI estimator is unbiased when 'the click propensity \\hat o and pseudo conversion labels of unclicked samples r* are accurate,' and the proof's only non-trivial step uses r*_{u,i} = r_{u,i}. In EVI, r* is the teacher's predicted probability, so r* = r holds only for degenerate predictions; the theorem assumes the exact property the paper claims to prove. Theorem 2 cannot fill this gap: Rosenbaum and Rubin's theorem guarantees conditional independence of potential outcomes given the true propensity score, not the equality ED[P(r=1|\\hat O=\\hat o)] = ED[P(r=1|o=1)] asserted in the proof, and the accuracy of an estimated pCTR is precisely the unverifiable unconfoundedness assumption the paper acknowledges in its conclusion. In addition, Eq. 14 defines the CVR-T loss as a sum over the entire space D, but true post-click labels r are missing for unclicked samples; the paper never specifies whether unclicked rows are masked, set to 0, or assigned pre-existing pseudo-labels, so the teacher's training objective is underspecified and may itself reintroduce the selection bias the method is designed to remove.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EVI, an entire-space CVR estimation framework that combines a click-propensity-conditioned teacher (CECT), variational information exploitation (VIE) at the feature level, and logit distillation with pseudo labels for unclicked samples. The authors claim that the conditional teacher produces unbiased pseudo labels, that the EVI estimator is unbiased under the stated conditions, and that EVI achieves the best CVR AUC and NLL on six datasets, with an average AUC improvement of 2.25% over the best baselines. The paper includes theoretical statements (Theorems 1-3), ablations, parameter sensitivity studies, and a public code repository.","tokens_in":14918,"tokens_out":3141,"duration_ms":28115,"significance":"If the empirical claims hold, EVI offers a practically useful combination of techniques for a well-known problem: entire-space CVR estimation with sample selection bias and data sparsity. The paper provides a comparatively broad evaluation across five public and one industrial dataset, includes ablations isolating the two proposed components, and releases code to support reproducibility. However, the theoretical support for the central 'unbiased pseudo-label' claim is much weaker than presented: the key theorem relies on an assumption that the pseudo labels equal the true labels, and the teacher loss in Eq. (14) is underspecified for unclicked samples. The headline average improvement of 2.25% is also not consistently supported by the numbers in Table 2. These issues are load-bearing because the paper's novelty is framed largely in terms of theoretical unbiasedness.","major_comments":[{"comment":"The CVR teacher loss in Eq. (14) is written as a sum over the entire space D, using the true conversion label r_{u,i}. For unclicked samples (u,i) in N, r_{u,i} is unobserved by definition. The manuscript does not specify whether these terms are masked, assigned r=0, or replaced by pre-existing pseudo-labels. As written, the objective cannot be computed, and if the unclicked terms are silently set to zero or assigned r=0, the 'entire-space' teacher may reintroduce the very selection bias that Section 3 motivates. This must be clarified and the actual training procedure stated precisely.","section":"Section 4.4, Eq. (14)"},{"comment":"The proof of Theorem 3 assumes r*_{u,i} = r_{u,i} for unclicked samples and ED[o_{u,i}] = ED[\\hat{o}_{u,i}] to reach the conclusion. In EVI, r* is the teacher's predicted probability, a continuous value in [0,1], so r* = r holds only in degenerate cases. The theorem therefore assumes the exact property that the paper claims to establish: that the teacher's pseudo-labels are accurate for unclicked samples. This makes the unbiasedness claim circular. The theorem should be restated as a conditional result (e.g., 'if the teacher is well-calibrated and the propensity model is correctly specified, then ...'), or the proof needs a substantive argument that the EVI teacher achieves r* = r in the relevant limit.","section":"Appendix B.3, Theorem 3"},{"comment":"Theorem 2 relies on Rosenbaum and Rubin's propensity-score result, which requires conditioning on the true propensity score. The paper instead conditions on the predicted click propensity \\hat{o}_{u,i} and states that unbiasedness holds 'provided that \\hat{o}_{u,i} is accurate.' The accuracy of an estimated pCTR is exactly the unconfoundedness assumption that is unverifiable from observational data. The paper's own conclusion acknowledges that EVI 'relies on accurate CTR estimation,' but no experiment or sensitivity analysis varies CTR quality or otherwise probes the accuracy of the propensity model. The theoretical claim should be weakened to a conditional statement, and a sensitivity analysis should be added if the authors wish to substantiate it empirically.","section":"Appendix B.2, Theorem 2"},{"comment":"The average AUC improvement of 2.25% reported in the abstract, the introduction, and Section 5.2 does not match the numbers in Table 2. For example, the AE-US improvement over the best baseline (0.8057 vs. 0.7889) is approximately 2.13%, not 3.13%; the AE-NL improvement (0.7741 vs. 0.7685) is approximately 0.73%, not 4.84%. Recomputing the per-dataset relative improvements from Table 2 gives an average of roughly 1.4%, not 2.25%. The same check applies to the claimed NLL reductions. The text and the table need to be reconciled, and the per-dataset improvement numbers in the bullet list should be corrected or the computation should be explained.","section":"Section 5.2, Table 2"}],"minor_comments":[{"comment":"The text refers to 'AR-ES' when the table and Section A.1 use 'AE-ES'; this typo should be fixed.","section":"Section 5.2"},{"comment":"There is a parenthesis mismatch in Eq. (8) and Eq. (15): the term for unclicked samples is written as (1 - o_{u,i})\\delta(...), but the opening bracket only encloses part of the expression. The formulas should be rewritten cleanly.","section":"Section 3.2, Eqs. (8) and (15)"},{"comment":"The phrase 'Kullback-Leiber divergence' should be 'Kullback-Leibler divergence'.","section":"Section 4.3"},{"comment":"The text mentions 'EVT-teacher' in the discussion of Figure 2, but the model is called EVI; this typo should be corrected.","section":"Section 5.3"},{"comment":"The list of comparison methods says '7 representative baseline models' but lists eight methods (ESMM, ESCM2-IPW, ESCM2-DR, MRDR-GPL, DR-V2, UKD, DCMT, DDPO); the count should be updated.","section":"Appendix A.2"},{"comment":"The definition of the variational distribution q(t,s) in Section 4.3 says 'we employ Gaussian distribution with mean µ and ρ', but ρ is the standard deviation in Eq. (11); the notation should be clarified so that it is clear that ρ enters through the variance term.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The empirical evaluation is the paper's strongest asset: EVI is ahead of all baselines on all six datasets in Table 2, and the ablation and sensitivity studies are useful. However, the theoretical framing as 'unbiased pseudo-label generation' is not supported by the proofs as written, and the headline improvement numbers are inconsistent with the reported table. The authors should be asked to either strengthen the theory to a genuinely conditional statement (e.g., under correct specification and calibrated teacher), add a sensitivity analysis for CTR quality, and fix the numerical reporting, or substantially soften the theoretical claims. These are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the EVI paper. The practical idea is more interesting than the theory suggests: a CVR teacher conditioned on predicted click propensity via an outer-product injection, then transferring to the student via both logit distillation and a variational mutual-information term. The combination is new in this space, as far as the cited literature goes. The experiments are real: six datasets, code released, ablations, sensitivity analyses, and EVI leads on every dataset in Table 2. The gains are consistent but incremental — recomputing from the table gives roughly 1.4% average relative AUC, not the 2.25% in the abstract. That's a reporting error, not a fabrication.\n\nThe soft spots are where the paper claims theoretical backing. Theorem 3 proves unbiasedness only by assuming r* = r, the exact property the teacher is supposed to produce. That's a conditional statement, not a demonstration. Theorem 2 doesn't bridge the gap; Rosenbaum-Rubin gives conditional independence given the true propensity score, not the specific equality asserted, and 'accurate pCTR' is precisely an unverifiable assumption. Also, Eq. 14 defines the teacher loss over D with r labels that don't exist for unclicked rows; the paper never says whether they mask, zero-fill, or use pseudo labels there. That's a real underspecification, not a nit. The conclusion honestly admits reliance on accurate CTR, but no sensitivity analysis to pCTR quality is provided, which would have been more convincing than the theorems.\n\nNone of this kills the empirical method. The architecture and losses are concrete enough to reproduce, and the consistent win across six datasets is a useful data point for people working on entire-space CVR. The theory should be reframed as a sensitivity/consistency result under an accuracy assumption, and the undefined loss needs a line in the text.\n\nWho it's for: researchers in recommender systems CVR debiasing. A serious referee should see it — the flaws are clear and fixable, and the experiments deserve checking. I'd engage with it, but I'd ask for a revised version that fixes the theorem statements, clarifies Eq. 14, and recomputes the average improvement.","headline":"Useful empirical recipe with an overstated theoretical wrapper; worth reviewing if the authors fix the circular theorem and the undefined loss.","tokens_in":15455,"tokens_out":4947,"would_cite":true,"duration_ms":39041,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EVI uses a click-propensity-conditioned teacher to generate unbiased pseudo labels for unclicked samples, adds variational information exploitation and logit distillation, and reports the best CVR AUC on all six datasets.","keywords":["post-click conversion rate","sample selection bias","data sparsity","entire-space modeling","pseudo labels","knowledge distillation","variational information exploitation","propensity score"],"falsifier":"Train EVI with a deliberately corrupted CTR model, for example by shuffling or adding noise to the predicted click probabilities while keeping all other components unchanged, and compare the teacher's pseudo-label log-loss and the student's AUC on unclicked test samples against the accurate-CTR run; the theory predicts clear degradation, so a flat or reversed result would falsify the propensity-conditioning mechanism.","tokens_in":14401,"feed_emoji":"🎯","tokens_out":10347,"duration_ms":83731,"temperature":0.7,"pith_summary":"Post-click conversion rate (CVR) prediction is trained on clicked impressions but must predict for all impressions, so the missing conversion labels on unclicked samples introduce selection bias and leave little data to learn from. This paper proposes EVI, an entire-space framework whose central move is to condition the CVR 'teacher' on the predicted click probability before it generates pseudo labels for unclicked samples. The conditioning is meant to make the missing conversion outcome statistically independent of click status, yielding unbiased pseudo labels that are then transferred to the production CVR model by logit distillation plus a variational information term. On five public datasets and one industrial dataset, EVI reports the best CVR AUC on all six and the best log-loss on five, averaging a 2.25% AUC gain and a 2.78% NLL reduction over the strongest baseline. If the claim holds, unclicked impressions cease to be wasted or dangerous training signal, which would mitigate both sample selection bias and data sparsity at once.","feed_headline":"EVI converts unclicked impressions into unbiased CVR training data","feed_subtitle":"Predicted click probability makes unclicked samples usable for unbiased CVR training.","key_machinery":"The load-bearing object is the conditional entire-space CVR teacher, a CVR network whose input representations are multiplied by a learned projection of the predicted click probability $\\hat{o}_{u,i}$; this click-conditioned representation is what converts unclicked samples from biased guesswork into propensity-adjusted pseudo labels. The second piece is variational information exploitation (VIE): for each transferred layer pair $(t_k, s_k)$ of teacher and student, the student predicts the teacher activation with a Gaussian variational distribution, and the loss maximizes the variational lower bound on the mutual information $I(t_k; s_k)$ between the two networks. Logit distillation then carries the pseudo labels themselves. Together the teacher, the layer-wise information bound, and the propensity-weighted student loss are the machinery that the unbiasedness theorems and the reported gains rest on.","core_discovery":"The paper's central claim is that existing entire-space CVR methods transfer selection bias to the student because their pseudo-label generators discard or weaken click information, which is where the confounders of the click-conversion relationship live. EVI's conditional entire-space CVR teacher conditions on the predicted click propensity $\\hat{o}_{u,i}$ by projecting it through a linear layer and taking the outer product with the teacher's representations, so the teacher models $P(r_{u,i}=1 \\mid \\hat{O}_{u,i}=\\hat{o}_{u,i})$ rather than the marginal $P(r_{u,i}=1)$. Under an accurate CTR model this is exactly the propensity-score conditioning that makes the unobserved conversion outcome independent of click status, and Theorem 2 states that the teacher's pseudo labels are then unbiased. The student is trained over the whole impression space with inverse click propensity and inverse non-click propensity weighting, pseudo-label logit distillation, and a variational information exploitation loss that maximizes a lower bound on mutual information between teacher and student layer activations; Theorem 3 states that with accurate propensity and accurate pseudo labels the EVI loss equals the ideal full-space loss in expectation. The experimental section reports EVI as the top CVR method on all six datasets, with the best teacher log-loss on the non-click space and the lowest student mean bias in the three datasets tested.","pith_inferences":["A natural stress test the paper does not run is to corrupt the CTR model, for example by adding noise to or shuffling the predicted click probabilities, and measure EVI's AUC degradation; the theory predicts a direct hit, and the paper's own conclusion says EVI relies on accurate CTR estimation.","The Gaussian variational family is a modeling choice rather than a consequence of the theorems, so replacing it with a richer density could close more of the residual teacher-student information gap on sparse conversion events.","Because the paper flags sensitivity to noisy clicked samples, combining EVI with label-noise-robust losses or sample reweighting on the click space is the obvious next step and would test whether the bias reduction survives real-world click noise.","The conditioning mechanism uses only the scalar click probability, so conditioning on a richer click-side context embedding might weaken the unconfoundedness assumption in practice while preserving the spirit of the proof."],"forward_implications":["Unclicked impressions become usable training signal rather than discarded data, so EVI attacks data sparsity without reintroducing selection bias.","If the propensity model is accurate, the full-space student loss is the ideal loss in expectation, meaning the method's training objective is aligned with its evaluation metric.","The reported gains are systematic across datasets, best AUC on all six and best NLL on five, rather than an isolated win on one benchmark.","The recipe is a drop-in multi-task architecture with shared embeddings, CTR and CTCVR heads, a teacher and a student, and five tunable loss weights, so production recommender systems can adopt it incrementally.","The teacher's pseudo labels also improve calibration in the non-click space, lowering log-loss relative to the UKD and DDPO teachers, which strengthens downstream ranking quality."],"supporting_citations":[{"why":"Introduces the entire-space CTCVR modeling paradigm that EVI extends with a teacher-student split.","marker":"Ma et al. 2018b"},{"why":"Supplies the IPS and DR entire-space debiasing losses and the do-operator perspective that the EVI student loss and teacher formulation build on.","marker":"Wang et al. 2022a"},{"why":"Defines the UKD distillation teacher baseline whose pseudo-label bias EVI targets and against which EVI is compared.","marker":"Xu et al. 2022"},{"why":"Defines DDPO, the dual-propensity pseudo-label baseline that EVI improves upon and compares teacher log-loss against.","marker":"Su et al. 2024"},{"why":"Provides the propensity-score theorem used to prove Theorem 2, the unbiasedness of the conditional teacher.","marker":"Rosenbaum and Rubin 1983"},{"why":"Supplies the variational information distillation objective that the VIE loss adapts for layer-wise teacher-student transfer.","marker":"Ahn et al. 2019"},{"why":"Provides the variational lower bound on mutual information that the VIE loss maximizes.","marker":"Barber and Agakov 2004"},{"why":"Defines the DCMT non-conversion estimator baseline used in the comparison and motivation.","marker":"Zhu et al. 2023"}],"fun_headline_variants":["EVI turns unclicked data into unbiased CVR labels","Variational info exploitation boosts CVR prediction","Unbiased pseudo labels from whole impression space","EVI: entire-space CVR with unbiased pseudo labels","CVR prediction improved via variational info exploitation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's predicted click probability captures every shared cause of clicking and converting, so that after conditioning on it, an unclicked sample's missing conversion outcome is statistically equivalent to a randomly missing one.","fun_headline_variants_meta":{"raw":{"variants":["EVI turns unclicked data into unbiased CVR labels","Variational info exploitation boosts CVR prediction","Unbiased pseudo labels from whole impression space","EVI: entire-space CVR with unbiased pseudo labels","CVR prediction improved via variational info exploitation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1305,"prompt_tokens":1008,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":224}},"tokens_in":624,"tokens_out":297,"duration_ms":3044,"temperature":1.0,"reasoning_tokens":224,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:48:34.782830+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train EVI with a deliberately corrupted CTR model, for example by shuffling or adding noise to the predicted click probabilities while keeping all other components unchanged, and compare the teacher's pseudo-label log-loss and the student's AUC on unclicked test samples against the accurate-CTR run; the theory predicts clear degradation, so a flat or reversed result would falsify the propensity-conditioning mechanism.","supporting_citations":[{"cited_title":"X.; Damianou, A.; Lawrence, N","cited_arxiv_id":null,"evidence_quote":"Supplies the variational information distillation objective that the VIE loss adapts for layer-wise teacher-student transfer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the variational lower bound on mutual information that the VIE loss maximizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DCMT non-conversion estimator baseline used in the comparison and motivation."}],"review_version":1}