{"id":"f2c67c99-863f-4065-af84-ba8756e3dd1b","arxiv_id":"2412.06852","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"EGEAN combines exposure-guided embedding alignment with a parameter-varying doubly robust loss and reports higher CVR and GMV in offline and online advertising experiments.","lead":"This paper proposes EGEAN, a neural model that aligns clicked and unclicked advertising data with an exposure-prediction task and a metric-learning loss, plus a new doubly robust loss for conversion estimation. The authors report offline AUC gains on Taobao and Meituan data and an online A/B test with 5.94% higher CVR and 6.29% higher GMV.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PVDR (Eq. 12) contains no imputation term, so it cannot be doubly robust; the steady-state condition (Eq. 11) tunes λ with the imputation model but does not insert it into the estimator.","rationale":"The reader's verdict is REJECT, and my check supports that. The reader identified Eq. 11 as an asserted identity used to select λ; I agree that this is circular/unsupported. But the sharper problem is Eq. 12: even if the identity is accepted, the estimator lacks the imputation component that defines double robustness, so the claim 'doubly robust' cannot hold in the usual sense. This is internal to the manuscript, not a disagreement with community consensus. The empirical part of the paper—EGEAN architecture, offline AUC improvements, and the online A/B numbers—is directionally plausible, and the public code link is a positive artifact. However, the theoretical claim is load-bearing because the paper motivates PVDR as the debiasing mechanism and claims unbiased CVR estimation; without it, the method reduces to a heuristically stabilized IPS with a data-dependent λ and no demonstrated unbiasedness. My recommendation is unchanged: reject or require major revision, because the central estimator needs a correct formulation and proof. I am not accusing the authors of dishonesty; this is a technical gap in the paper's argument.","tokens_in":7470,"tokens_out":7593,"duration_ms":81210,"concrete_test":"Simulate a small MNAR dataset: draw x_i, true loss e_i = δ(r_i,\\hat r_i), propensity p_i = P(o=1|x_i), and observe o_i. Fit a deliberately misspecified \\hat p_i (e.g., ignore x_2) and a perfectly correct imputation model \\hat e_i = e_i. Solve Eq. (13) for λ on a training fold, then compute L_PVDR on a test fold. Repeat 1000 times and report mean bias relative to L_ideal=(1/N)Σ e_i. If the mean bias is not zero (or not substantially smaller than the misspecified-IPS bias), the double-robustness claim is refuted: a true DR estimator with correct imputation and misspecified propensity would be unbiased. The same simulation with correct \\hat p and misspecified \\hat e can separate the two hinges.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2.2 asserts that 'when the steady-state condition is met, PVDR is a doubly robust estimator, and the CVR estimation is unbiased.' This is the theoretical basis for the main improvement and is not supported by the estimator as written. Equation (12) is L_PVDR = Σ o e / p / [λ|D| + (1−λ) Σ o/p]. A doubly robust loss must combine the inverse-propensity term o e/p with an imputation term (1−o/p)\\hat e, so that bias from a misspecified propensity can be offset by a correct imputation model. In Eq. (12), \\hat e appears nowhere in the estimator: the imputation model enters only as a scalar through the steady-state equation (11)/(13), which is used to choose λ. Even if Eq. (11) can be solved, choosing λ with \\hat e does not add the missing imputation correction to the numerator; under a misspecified \\hat p, E[o e/\\hat p] ≠ E[e] and no term in Eq. (12) can compensate. In addition, λ is data-dependent (solved from Eq. 13), so unbiasedness would require an expectation over λ not provided; the paper gives no derivation at all. Thus the central double-robustness/unbiasedness claim is at best unproven and, on the face of the equation, false in the standard sense.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EGEAN, a multi-task network for post-click conversion estimation, with three components: an exposure probability prediction task, a task-personalized network (LoRA, EPNet, PPNet), and an MMD-based metric learning module. It also introduces PVDR, a 'Parameter Varying Doubly Robust Estimator' with a steady-state control condition intended to reduce variance under small propensities. The paper reports offline AUC results on Ali-CCP and Meituan, an ablation study, and a seven-day online A/B test showing 5.94% CVR and 6.29% GMV gains. The central theoretical claim is in Section 2.2.2: 'When the steady-state condition is met, PVDR is a doubly robust estimator, and the CVR estimation is unbiased.' The manuscript provides no derivation of this claim, and the estimator as written does not match the standard doubly robust form.","tokens_in":7724,"tokens_out":5479,"duration_ms":57399,"significance":"If the architecture results are reproducible, the exposure-guided alignment idea and the large-scale online deployment are notable strengths of the paper; the released code is also a positive element. However, the paper's main contribution is the PVDR estimator and its claimed unbiasedness/double-robustness property. That property is not merely under-derived; the estimator in Eq. (12) contains no imputation term, so the claim cannot hold in the standard doubly robust sense. Because the title, abstract, and contribution list all rest on this theoretical assertion, the significance of the paper as submitted is substantially undermined.","major_comments":[{"comment":"The central claim that PVDR is doubly robust and unbiased is not supported by the estimator as written. The loss in Eq. (12), L_PVDR = sum o e / p / [lambda |D| + (1-lambda) sum o/p], contains no imputation term; it is a self-normalized IPS estimator. A doubly robust loss must combine the inverse-propensity term o e/p with an imputation term such as (1 - o/p) e_hat, so that a correct imputation model can compensate for a misspecified propensity model. Here e_hat appears only as a scalar through the steady-state equation (11)/(13) used to choose lambda. Choosing lambda with e_hat does not add the missing correction to the numerator. Under a misspecified propensity model, E[o e/p] is not E[e], and no term in Eq. (12) can compensate unless e_hat is exactly the true error e, which is not argued. Thus the double-robustness/unbiasedness claim is at best unproven and, in the standard sense, false.","section":"Section 2.2.2, Eq. (12)"},{"comment":"The steady-state condition lambda + (1-lambda) A = B is asserted without derivation. No proof is given that this condition is sufficient for unbiasedness, and no assumptions are stated under which it would be. Moreover, lambda is data-dependent because it is solved from Eq. (13) using the fitted imputation model and propensity estimates; an unbiasedness proof would require taking expectations over this data-dependent lambda. If Eq. (11) can always be satisfied by choosing lambda, then the condition is a normalization identity enforced by construction, not a substantive modeling assumption, and invoking it as a precondition for unbiasedness is circular. This is load-bearing because the contribution list and abstract claim the PVDR estimator's theoretical property.","section":"Section 2.2.2, Eqs. (11) and (13)"},{"comment":"The imputation model is not actually defined as a supervised model. Eq. (10) writes Lhat = (1/|D|) sum e_hat, which is just an average of predicted values; the text does not specify the target labels or the training objective used to fit phi_theta to the 'CVR prediction error' e_u,i. Without a concrete training objective, the role of e_hat in the steady-state condition is not implementable or reproducible from the manuscript. This matters because e_hat is the only mechanism by which the imputation model influences the estimator.","section":"Section 2.2.2, Eq. (10)"},{"comment":"The offline results report AUC values without standard deviations, numbers of repeated runs, or significance tests. The text states that EGEAN 'significantly outperforms' the baselines, but the table alone does not support a claim of statistical significance. This is secondary to the theoretical problems but affects the empirical credibility of the paper, especially because the gains over EGEAN+DR are small in absolute AUC terms.","section":"Section 3.1, Table 2"}],"minor_comments":[{"comment":"The text refers to the 'TSN module' but the described module is the task personalized network, abbreviated TPN elsewhere; the terminology should be made consistent.","section":"Section 3.3"},{"comment":"The online A/B section says 'As shown in Table 3' but Table 3 is the ablation table; no table reports the online A/B results, traffic allocation, or confidence intervals.","section":"Section 3.4"},{"comment":"References [7] and [8] are the same StableDR paper, and reference [15] is a duplicate of reference [14]; these should be merged.","section":"References"},{"comment":"The ACM template metadata still contains placeholder conference information ('Conference acronym ’XX', 2018, Woodstock, NY, placeholder DOI) and needs to be updated for submission.","section":"General"},{"comment":"Eq. (13) simply restates Eq. (11) with A and B expanded; consider defining A and B once and numbering a single condition to avoid redundancy.","section":"Section 2.2.2, Eq. (13)"},{"comment":"The paper reports '5.94% CVR and 6.29% GMV improvement' but does not state the baseline value, the number of users in each arm, or the significance of the differences; these details are needed for an industrial validation claim.","section":"Section 3.4"}],"recommendation":"reject","confidential_remarks":"This is a close call between major revision and rejection. I recommend rejection because the central theoretical contribution, the PVDR estimator's doubly robust/unbiased property, is not merely missing a proof but is false as written: Eq. (12) contains no imputation term. Repairing this would require redefining the estimator, re-deriving the steady-state condition, retraining the models, and rerunning the experiments. The architectural and empirical parts alone, while potentially useful, are not enough to justify acceptance without the theoretical claim. There is no indication of misconduct; the issue is a load-bearing technical error in the manuscript's core method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a competent engineering paper with a real production A/B result, but the central theoretical claim doesn't survive contact with the estimator as written. The PVDR loss (Eq. 12) is a self-normalized IPS sum; it has no imputation term, so \"doubly robust\" is not the right label.\n\nWhat's actually new and good: the exposure-guided embedding alignment is a sensible alternative to adversarial domain adaptation. Combining an exposure prediction task, PEPNet-style personalization, LoRA, and MMD metric learning is a plausible architecture, and the public-dataset gains are large (CVR AUC 0.6604 on Ali-CCP vs 0.6401 for the best baseline, and 0.7093 vs 0.6802 on Meituan). The seven-day online A/B reports 5.94% CVR and 6.29% GMV lifts, and the code is linked. For an applied ad-systems paper, that is real evidence.\n\nSoft spots, in order of importance. The stress-test note is accurate: Eq. 12 contains no (1 - o/p)\\hat e correction. The imputation model only selects lambda via the steady-state identity (Eq. 11/13). Choosing a scalar with a fitted model is not double robustness, and because lambda is data-dependent, unbiasedness would need an extra argument that isn't supplied. The paper simply asserts the property. Second, the offline results have no error bars or significance tests, and the A/B is a single one-week run without a reported confidence interval. Third, the statement that PVDR \"obviously\" has better variance than StableDR is hand-wavy, and there are minor citation/reference duplications (StableDR appears twice).\n\nWho should read this: practitioners building CVR models in large-scale ad systems, who can borrow the architecture. It is not a source for theoretical guarantees. The central argument for PVDR's unbiasedness should be removed or fixed. That's a load-bearing flaw because the title and framing lean on it.\n\nRecommendation: send it to peer review, but expect major revision. The empirical contribution is valuable enough to warrant referee time. Ask the authors to either prove the claim properly or drop the \"doubly robust/unbiased\" language, add error bars, and report the A/B test design more fully.","headline":"Solid engineering with a broken theoretical claim: the PVDR estimator as written is not doubly robust, but the paper's empirical contribution deserves a referee.","tokens_in":8269,"tokens_out":3561,"would_cite":false,"duration_ms":34029,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that aligning embeddings to the exposure space, together with a parameter-varying doubly robust estimator, removes covariate-shift bias in post-click conversion prediction and improves real-world advertising revenue.","keywords":["post-click conversion rate","covariate shift","sample selection bias","doubly robust estimator","embedding alignment","metric learning","multi-task learning","online advertising"],"falsifier":"Recompute PVDR's expectation under the missing-not-at-random model and check whether the steady-state condition forces the first-order bias to zero; a two-point or synthetic dataset with known true CVRs and very small propensities would expose any residual bias directly. Simpler: run EGEAN+PVDR with λ fixed at a value violating Eq. (11) and observe whether AUC or calibration degrades in the predicted way.","tokens_in":7232,"feed_emoji":"📈","tokens_out":4818,"duration_ms":42707,"temperature":0.7,"pith_summary":"This paper tries to establish that the covariate shift between clicked and unclicked ad impressions—not just the missing labels that cause sample selection bias—is a correctable source of error in post-click conversion rate (CVR) estimation. Its proposed network, EGEAN, aligns user-item embeddings toward the exposure space using an exposure prediction task, task-personalized modules, and an MMD metric loss, so that knowledge from non-clicked impressions transfers to clicked ones. The paper also proposes PVDR, a parameter-varying doubly robust estimator with a steady-state control condition, which it claims is unbiased when that condition holds and which is designed to handle small propensity scores better than the StableDR estimator it extends. On the Ali-CCP public dataset and a three-month Meituan industrial dataset, EGEAN+PVDR reports the highest CVR and CTCVR AUC, and a seven-day online A/B test reports 5.94% higher CVR and 6.29% higher GMV against the deployed DDPO baseline.","feed_headline":"Exposure-guided CVR model lifts conversion rate 5.94% in live ads","feed_subtitle":"A new doubly robust estimator and exposure-guided embeddings raise CVR and GMV in production A/B tests.","key_machinery":"The load-bearing objects are the EGEAN architecture and the PVDR estimator. The architecture's exposure prediction task defines positive samples as exposed impressions and uses in-batch negative sampling, giving the shared embedding a target that exists in both click and non-click spaces; the task-personalized network (LoRA adaptors plus EPNet and PPNet gates) then lets the CTR and CVR towers modify those embeddings and parameters without forgetting the shared structure, while the metric-learning loss, measured with MMD in an RKHS, pulls CVR-task embeddings back toward the shared embedding. The PVDR estimator is the ratio of a propensity-weighted prediction-error sum to λ|D|+(1−λ) times the sum of inverse propensities, with the steady-state condition λ+(1−λ)A=B asserting unbiasedness. Together these mechanisms are what the paper claims remove both sample selection bias and covariate shift.","core_discovery":"The paper's central claim is that an exposure-guided alignment mechanism can correct the covariate shift between click and non-click spaces, and that this correction, combined with a parameter-varying doubly robust estimator, yields unbiased CVR estimation over the full impression space. Concretely, EGEAN trains an exposure probability prediction task on exposed samples, fine-tunes shared embeddings with LoRA, personalizes both embeddings and MLP layers through task-specific gates, and minimizes the Maximum Mean Discrepancy between CVR-task embeddings and shared embeddings. The PVDR estimator generalizes StableDR by introducing a parameter λ that interpolates between IPS (λ=0) and StableDR (λ=1); when the steady-state condition λ+(1−λ)A=B is met, the paper asserts the estimator is doubly robust and unbiased. Empirically, EGEAN+PVDR reaches 0.6604 CVR AUC on Ali-CCP and 0.7093 on Meituan, outperforming all listed baselines, and the deployed system reportedly serves main Meituan traffic after the A/B test.","pith_inferences":["The unbiasedness claim of PVDR is only as strong as Eq. (11), which the paper states without proof; a natural next step is to derive the condition from the expectation of Eq. (12) and test it on synthetic data with known ground truth.","Exposure-guided alignment should transfer to other label-sparse prediction tasks that share an intermediate event between two spaces, such as post-view conversion or subscription prediction.","The paper reports point AUC values without confidence intervals; comparing EGEAN+PVDR against EGEAN+DR on repeated runs would quantify how much of the gain comes from the architecture versus the estimator."],"forward_implications":["If PVDR is indeed unbiased under the steady-state condition, CVR models can be trained and evaluated directly on the entire impression space without relying on the click space alone.","The exposure-guided alignment gives a way to transfer information from non-clicked impressions to clicked ones without adversarial training, which the paper argues is unstable on small datasets.","Because λ interpolates between IPS and StableDR, PVDR offers a tunable bias-variance trade-off for small propensity scores.","The reported online gains (5.94% CVR, 6.29% GMV) imply that correcting covariate shift can produce revenue-relevant improvements in a large-scale advertising system.","Ablations show that each of the three components—exposure network, task personalization, and metric learning—contributes to the final AUC, meaning the gains are not from a single trick."],"supporting_citations":[{"why":"Defines the covariate-shift problem between click and unclick spaces that EGEAN targets and serves as the adversarial AECM baseline.","marker":"[16]"},{"why":"StableDR is the estimator that PVDR extends with the parameter λ and the steady-state control.","marker":"[7]"},{"why":"ESMM introduces the entire-space multi-task setup and provides the Ali-CCP dataset used for evaluation.","marker":"[10]"},{"why":"DDPO is the strongest baseline in the comparison and the pre-deployment model used as the online A/B control.","marker":"[12]"},{"why":"LoRA supplies the low-rank adaptation mechanism used to fine-tune the pre-trained embeddings.","marker":"[4]"},{"why":"PEPNet provides the EPNet and PPNet designs for task-personalized embedding and parameter networks.","marker":"[1]"},{"why":"Supplies the MMD metric used to measure and minimize distribution distance between CVR and shared embeddings.","marker":"[2]"},{"why":"DCMT is another entire-space causal baseline that the paper compares against.","marker":"[18]"}],"fun_headline_variants":["Exposure-guided alignment lifts CVR 5.94% in live ads","Doubly robust estimator + exposure alignment beat baselines on CVR","Aligning click and non-click spaces boosts CVR by 5.94% in production","Exposure-guided network raises CVR and GMV in Meituan A/B tests","CVR model with exposure-guided embeddings gains 5.94% in live test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The estimator's unbiasedness collapses if the steady-state condition λ+(1−λ)A=B (Eq. 11) is not a genuine sufficient condition for unbiasedness, since the paper asserts this identity without proof and chooses λ so that it holds.","fun_headline_variants_meta":{"raw":{"variants":["Exposure-guided alignment lifts CVR 5.94% in live ads","Doubly robust estimator + exposure alignment beat baselines on CVR","Aligning click and non-click spaces boosts CVR by 5.94% in production","Exposure-guided network raises CVR and GMV in Meituan A/B tests","CVR model with exposure-guided embeddings gains 5.94% in live test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1630,"prompt_tokens":903,"completion_tokens":727,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":519,"tokens_out":727,"duration_ms":6406,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:14:31.503766+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute PVDR's expectation under the missing-not-at-random model and check whether the steady-state condition forces the first-order bias to zero; a two-point or synthetic dataset with known true CVRs and very small propensities would expose any residual bias directly. Simpler: run EGEAN+PVDR with λ fixed at a value violating Eq. (11) and observe whether AUC or calibration degrades in the predicted way.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the covariate-shift problem between click and unclick spaces that EGEAN targets and serves as the adversarial AECM baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MMD metric used to measure and minimize distribution distance between CVR and shared embeddings."}],"review_version":1}