{"id":"0259ea46-9558-4716-b5ad-3db1963110b5","arxiv_id":"2505.19755","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"EGA-V1 unifies advertising ranking and auction into a single non-autoregressive generative model with cluster attention, and is reported to beat multi-stage cascades on Meituan's ad traffic.","lead":"Meituan researchers built EGA-V1, one end-to-end model that replaces the usual multi-stage ad ranking pipeline and directly generates the whole ad list from a large local candidate set. The system reports higher click-through rate and revenue per thousand impressions in offline replay and a one-week online A/B test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Offline eCTR/eRPM gains are computed with EGA-V1's own permutation-aware evaluator (Eq. 18) -- the same model used as the RLAF reward -- so the headline 8.3%/11.4% lifts may be self-referential rather than real.","rationale":"The reader's weakest assumption correctly identifies the self-referential offline evaluator as the load-bearing risk. The abstract and Section 5.6 advance a strong architectural claim, but the only large-scale quantitative support (Table 1) is produced by an evaluation loop in which EGA-V1's generator is trained against the same score function used to measure it. This is not an internal inconsistency in the method, but a correctness risk: if the evaluator has any systematic preference for EGA-V1's output distribution -- plausible because it was trained on EGA-V1 representations and never calibrated on baseline sequences -- the eCTR and eRPM lifts are not trustworthy. The proposed check (an independent, log-trained evaluator plus calibration on held-out exposures) would settle this directly. Until then, CONDITIONAL acceptance is the right stance; the concern does not change the reader's verdict.","tokens_in":17337,"tokens_out":3530,"duration_ms":36587,"concrete_test":"Retrain the Eq. 18 evaluator as a standalone pCTR model using only logged impression/click data from the training period, with no RecFormer hidden states and no AucFormer/RLAF training. On the held-out 14-day test set, first check calibration: compare the evaluator's predicted q_ctr with observed clicks on actually-exposed sequences. Then replay the MCA, FS-LTR, and EGA-V1 output sequences through this independent evaluator and recompute eCTR/eRPM with standard errors. If EGA-V1's lifts over FS-LTR fall below significance or reverse, the offline claim is an artifact of evaluator bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim hinges on offline replay results in Table 1: eCTR +8.3% and eRPM +11.4% for EGA-V1 over FS-LTR. These metrics (Section 5.1.2) use q_ctr from the permutation-aware evaluator of Eq. 18. That evaluator is not an independent judge: it is a component of EGA-V1, trained with Eq. 21 on EGA-V1's RecFormer representations, and then frozen and used as the reward model in RLAF (Eqs. 22-23). The generator is explicitly optimized to maximize this evaluator's scores. Replaying MCA/FS-LTR through the same evaluator therefore scores the baselines out-of-distribution with a model whose score function the EGA-V1 generator has been trained to exploit. No ground-truth click labels appear in eCTR/eRPM, and no calibration or unbiasedness check for the evaluator is reported. The online A/B (Section 5.6) is genuinely independent evidence, but it runs for one week on a low-traffic slot, so it cannot by itself support the strong architecture-level conclusion. Unless the evaluator is shown to be an unbiased estimator of actual CTR for arbitrary sequences, the offline improvements in Table 1 may be substantially manufactured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EGA-V1, an end-to-end generative architecture that replaces a multi-stage cascaded advertising pipeline (recall, pre-ranking, ranking, auction) with a single model. The main components are a Hybrid Feature Service that decouples user and ad features, a RecFormer with cluster-attention for modeling user interest and ad externalities, and an AucFormer with a non-autoregressive generator, a permutation-aware evaluator, and a payment network. Training is done in two stages: pre-training on click/purchase signals, and post-training with reinforcement learning from auction feedback plus a payment-network loss. The paper claims offline improvements in eCTR, eRPM, recall, AUC, and incentive compatibility over MCA and FS-LTR baselines, and online gains of +5.2% CTR, +13.6% RPM, and +3.1% ROI in a one-week A/B test on a low-traffic slot.","tokens_in":17735,"tokens_out":8353,"duration_ms":78980,"significance":"If the evaluation were trustworthy, this would be a meaningful industrial contribution: a single non-autoregressive model that handles both ranking and allocation over a large candidate set, with a concrete system design for feature serving and latency control. The paper is strong in architectural detail and in making the externality and auction-consistency problems explicit. Its main weakness is that the offline evidence is generated by the proposed model's own evaluator, which is also the RLAF reward model, so the headline offline gains may be partially self-referential. The online A/B test is independent but is not accompanied by significance testing and is limited to one week on a low-traffic slot. The complexity analysis also does not reproduce the claimed 0.97 FLOPs ratio. These are load-bearing issues for the central claim that one model outperforms the full cascade pipeline.","major_comments":[{"comment":"The offline metrics eCTR and eRPM are computed using q_ctr from the permutation-aware evaluator of Eq. (18). That evaluator is trained with Eq. (21), is frozen and used as the reward model in the RLAF objective of Eqs. (22)-(23), and the generator is explicitly optimized to maximize scores produced by it. Therefore the offline comparison of EGA-V1 against MCA and FS-LTR in Table 1 is self-referential: the baselines are scored out-of-distribution by a model whose score function the proposed generator has been trained to exploit. No calibration check, no unbiasedness argument, and no ground-truth click labels are reported for the eCTR/eRPM values. The authors should either validate the evaluator against actual click labels for arbitrary sequences, report offline metrics computed with real user feedback, or provide a separate held-out evaluation that does not use the reward model as the judge.","section":"5.1.2, Eq. (18), Eqs. (21)-(23)"},{"comment":"The online A/B test is the only independent evidence for the architecture-level claim, but Table 4 reports only relative changes (+5.2% CTR, +13.6% RPM, +3.1% ROI) with no confidence intervals, p-values, sample sizes, or traffic volume for the low-traffic slot. The abstract states that the improvements are 'statistically significant,' yet no significance test is described. Without error bars or a significance statement, the one-week A/B result cannot by itself support the strong conclusion that EGA-V1 outperforms the deployed MCA pipeline. The authors should add significance tests, confidence intervals, and traffic/impression counts, or soften the claim accordingly.","section":"5.6, Table 4"},{"comment":"The claimed FLOPs ratio of 0.97 is not reproduced by the paper's own formulas. With the hyperparameters from Section 5.1.4 (N=10^5, L=1000, d=128, N_c=128, m=6, m_k=2, alpha=0.033, N_r=3300), the simplified ratio (2 m_k N L d^2)/(m_r N_r 4 L^2 d) evaluates to approximately 0.65, not 0.97; the full expressions in Eqs. (30)-(33) do not yield 0.97 either. In addition, some printed terms in Eq. (32) appear dimensionally inconsistent as written (e.g., N_ad^2 without a d factor). The authors should correct the formulas, state the exact assumptions used for the ratio, and provide a reproducible per-module FLOPs table.","section":"Appendix A.1"}],"minor_comments":[{"comment":"The abstract and conclusion claim 'extensive offline evaluations on public benchmarks,' but Section 5.1.1 describes only the proprietary Meituan dataset; no public benchmark is used anywhere in the experiments. This should be corrected or public experiments should be added.","section":"Abstract and Section 5.1.1"},{"comment":"The text says 'Judging from the online experimental results in Table 2,' but Table 2 reports offline eCTR and eRPM. The wording should be changed to avoid implying that the ablation results are from online A/B tests.","section":"Section 5.3, Table 2"},{"comment":"The paragraph before Figure 5 is incomplete: 'Besides, we Results in Figure 5 reveal:' is not a finished sentence, and the figure itself is not described with axis labels or a legend. Please complete the text and make the figure self-contained.","section":"Section 5.5"},{"comment":"The definition of m_k is unclear as printed: the footnote appears to define m_k = ceil(m^2/m_c), which is inconsistent with the hyperparameter values m=6, m_c=2, m_k=2 in Section 5.1.4. Please clarify the intended formula and notation.","section":"Section 3.3, footnote 3"},{"comment":"There are several small editorial issues: 'calcalated' in Appendix A.1, 'eletment-wise' near Eq. (14), and the notation in Eq. (17) where the softmax is written componentwise as z_{i;k} = Softmax([...])_i but the indexing is confusing. Please proofread and standardize the notation.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The evaluation transparency issue is the main blocker: the offline metrics are generated by the model's own reward evaluator, and the online test lacks significance testing. This is fixable within the scope of a revision, so I do not recommend rejection, but the central claim cannot be accepted without either a validated evaluator or independent offline labels. The FLOPs inconsistency in Appendix A.1 should also be corrected because it is explicitly used to support the efficiency argument."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read EGA-V1 carefully and mostly agree with your take, though I'd weight the circularity issue even more heavily. The architecture is interesting—HFS, cluster attention, NAR generator, and the payment network are combined into a coherent whole, and the online A/B is genuinely encouraging: +5.2% CTR, +13.6% RPM, +3.1% ROI over a deployed MCA. That's real evidence, even if it's one week on a low-traffic slot with no significance tests.\n\nThe load-bearing flaw is the offline evaluation. eCTR and eRPM are computed with the permutation-aware evaluator (Eq. 18), which is a trained component of EGA-V1 and also serves as the RLAF reward model (Eqs. 21–23). The generator is optimized to maximize this evaluator's outputs. Replaying MCA/FS-LTR through the same evaluator scores the baselines out-of-distribution with a model that EGA-V1 has been trained to exploit. No calibration or unbiasedness check is presented. So the offline gains (+8.3% eCTR, +11.4% eRPM) are probably inflated, and the central claim that a single model beats the cascade rests mostly on the A/B.\n\nSmaller issues: the abstract claims 'public benchmarks' but only a transformed proprietary dataset is used; the complexity derivation in Appendix A.1 doesn't reproduce its own 0.97 ratio—plugging in the stated hyperparameters gives roughly 0.65, and the AucFormer's O(N^2) term is omitted; Table 2's caption says 'online' but the metrics are offline.\n\nCredit where it's due: the combination is novel, the writing is clear, the IC metric (Ψ=2.3%) is interesting, and the system is deployed, so the engineering is real. This paper deserves a serious referee, but not as-is. The authors need to show the evaluator is an unbiased CTR estimator across architectures, or substitute a held-out judge, and fix the complexity analysis.\n\nI'd bring it to a reading group to discuss the self-referential evaluation pitfall, but I wouldn't cite the offline numbers in my own work until the circularity is resolved.","headline":"Promising industrial architecture with a genuine online A/B, but the offline evaluation is circular and the complexity analysis has a mismatch—engage, but with caution.","tokens_in":18232,"tokens_out":5793,"would_cite":false,"duration_ms":54000,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single end-to-end generative model, EGA-V1, can replace the multi-stage cascade architecture for online advertising ranking, producing the final ad sequence directly from the full city-level candidate pool and beating the cascade on…","keywords":["online advertising","end-to-end learning","multi-stage cascade architecture","generative ranking","non-autoregressive generation","cluster attention","reinforcement learning from auction feedback","ad auction design"],"falsifier":"Train an independent permutation-aware CTR evaluator on logged exposures from the deployed cascade only, with no EGA-V1 representations or RLAF labels, and re-run the offline eCTR/eRPM comparison; if EGA-V1's advantage shrinks to zero while the online A/B gains also fail to reproduce, the central unification claim is refuted.","tokens_in":17154,"feed_emoji":"📈","tokens_out":8197,"duration_ms":78111,"temperature":0.7,"pith_summary":"This paper argues that the entire online-advertising ranking pipeline—recall, pre-ranking, ranking, and auction—can be collapsed into one model that generates the final ad sequence directly from the full candidate set in location-based services. It reports that this single-model architecture lifts click-through rate by 5.2%, revenue per mille by 13.6%, and advertiser ROI by 3.1% in online A/B tests against a fully deployed multi-stage cascade, with only a 2.2% relative increase in response time. Offline, it reports eRPM of 217.1 against 194.9 for the strongest cascade baseline, an 11.4% gain, and an incentive-compatibility regret of 2.3% versus 9.1%. If true, the presumed trade-off between computational cost and holistic optimization in advertising disappears: a unified model can account for interactions among all candidate ads instead of only those that survive earlier filtering stages.","feed_headline":"One model replaces ad pipeline: CTR +5.2%, RPM +13.6%","feed_subtitle":"End-to-end generative ranking over the full city ad pool beats multi-stage cascades in online A/B tests.","key_machinery":"The central object is the cluster-attention block, which replaces the $N \\times N$ attention matrix with a learnable cluster matrix $\\mathbf{S} \\in \\mathbb{R}^{N \\times N_c}$ that groups keys and values into $N_c$ surrogate tokens, cutting attention cost from $O(N^2 d)$ to $O(N N_c d)$. Global Cluster-Former applies this mechanism within the candidate-ad set and within the user behavior sequence, while Mid-fusion Interest-Former applies it across the two sequences through target attention and context attention, capturing both user interests and ad externalities. AucFormer then maps the resulting representations to slots: slot tokens $\\mathbf{T}$ produce an allocation matrix $\\mathbf{A} = \\mathbf{H}_{\\text{ad}} \\mathbf{T}^{\\top}$, a bid-weighted softmax turns it into allocation probabilities, and a permutation-aware evaluator predicts order-dependent pCTR that also serves as the reward model in reinforcement learning. A payment network, trained with a Lagrangian dual objective, sets payments below bids and approximately satisfies incentive compatibility. The Hybrid Feature Service feeds the whole system by storing ad embeddings locally and user features remotely, so feature retrieval does not become the bottleneck.","core_discovery":"On the paper's own terms, EGA-V1 establishes that a single non-autoregressive generative model can perform the whole advertising ranking-and-auction task. The model narrows the candidate universe to ads in the same city, roughly $10^5$ candidates per request, and directly outputs the winning sequence for $K$ slots through a bid-weighted allocation matrix, bypassing the stage-by-stage filtering of multi-stage cascades. The reported offline numbers are Recall@50 0.513 versus 0.426 for FS-LTR, AUC 0.754 versus 0.743, eCTR 6.652% versus 6.140%, eRPM 217.1 versus 194.9, and an incentive-compatibility regret of 2.3% versus 9.1%. Online, the paper reports CTR +5.2%, RPM +13.6%, and ROI +3.1% against the deployed cascade. The core assertion is that externality modeling across the full candidate set, enabled by cluster attention and non-autoregressive generation, is what produces these gains.","pith_inferences":["Because the offline replay evaluator is trained on EGA-V1's own representations and doubles as the RL reward model, its eCTR and eRPM scores may be biased toward sequences EGA-V1 generates; the online A/B metrics are the independent evidence that would settle the question.","The same cluster-attention-plus-NAR-generator-plus-permutation-aware-evaluator recipe could transfer to other large-candidate selection tasks with strong item interactions, such as job search, ride-hailing dispatch, or organic feed ranking, wherever the candidate set can be narrowed by geography or session context.","The city-level narrowing to roughly $10^5$ candidates is what makes full-corpus externality modeling tractable; in non-LBS settings without a comparable locality constraint, the unification would need an additional retrieval mechanism or a redefined candidate universe.","The 2.3% IC regret is measured under scaled-bid perturbations; testing with adversarial or strategically correlated bid distributions would reveal whether the approximate incentive compatibility is structural or an artifact of the test distribution."],"forward_implications":["If the results hold, advertising platforms can model externalities across the full city-level candidate pool instead of only within a filtered subset, since the same model sees all candidates at once.","The non-autoregressive design keeps latency manageable: processing hundreds of times more candidates than the cascade's ranking model costs only 2.2% more response time in the paper's online measurements.","The bi-stage training recipe—pre-train on user engagement, then reinforce on auction feedback—lets the generator optimize revenue and advertiser utility directly, and the paper's offline IC metric drops from 9.1% to 2.3%.","Cluster attention compensates for the removal of explicit cross-features, so feature storage and transmission can be simplified with much smaller accuracy loss than in cascade models.","Scaling the model by stacking cluster-attention blocks gives steady gains with diminishing returns, so the architecture supports future scale-up rather than requiring a pipeline redesign."],"supporting_citations":[{"why":"It supplies the surrogate-token clustering mechanism that cluster-attention adapts to reduce attention complexity.","marker":"[24]"},{"why":"It provides the multi-head attention and FLOPs model that EGA-V1's modules replace or build on.","marker":"[25]"},{"why":"It supplies the permutation-level externality framework and payment network design, including the IC derivation, that AucFormer extends.","marker":"[37]"},{"why":"It supplies the neural auction payment and ex-post regret framework used in the payment network objective.","marker":"[17]"},{"why":"It defines the strongest offline baseline (FS-LTR) and the multi-stage consistency approach that EGA-V1 claims to surpass.","marker":"[35]"},{"why":"It provides the unified generative retrieve-and-rank precedent and the iterative preference alignment idea that informs the bi-stage training.","marker":"[2]"},{"why":"It motivates generative recommendation at scale while illustrating the autoregressive latency limitation that EGA-V1's non-autoregressive generator addresses.","marker":"[30]"},{"why":"It supplies the matching-based auction design with externalities that underlies the IC/IR constraints and allocation formulation.","marker":"[11]"}],"fun_headline_variants":["One model, whole ad pool: CTR +5.2%, RPM +13.6%","Generative ranking ends ad cascade era","Full-pool ad generation: +13.6% RPM in A/B test","Single model beats cascades: city-wide ad ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the offline replay evaluator—a permutation-aware CTR model trained on EGA-V1's own representations and reused as the RL reward model—scores EGA-V1's output and the cascade baselines without favoring EGA-V1.","fun_headline_variants_meta":{"raw":{"variants":["One model, whole ad pool: CTR +5.2%, RPM +13.6%","Generative ranking ends ad cascade era","Full-pool ad generation: +13.6% RPM in A/B test","Single model beats cascades: city-wide ad ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3254,"prompt_tokens":1032,"completion_tokens":2222,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":2147}},"tokens_in":648,"tokens_out":2222,"duration_ms":131032,"temperature":1.0,"reasoning_tokens":2147,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:07:19.338998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an independent permutation-aware CTR evaluator on logged exposures from the deployed cascade only, with no EGA-V1 representations or RLAF labels, and re-run the offline eCTR/eRPM comparison; if EGA-V1's advantage shrinks to zero while the online A/B gains also fail to reproduce, the central unification claim is refuted.","supporting_citations":[{"cited_title":"CAST: Clustering Self-Attention using Surrogate Tokens for Efficient Transformers","cited_arxiv_id":"2402.04239","evidence_quote":"It supplies the surrogate-token clustering mechanism that cluster-attention adapts to reduce attention complexity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the multi-head attention and FLOPs model that EGA-V1's modules replace or build on."},{"cited_title":"A Hybrid Cross-Stage Coordination Pre-ranking Model for Online Recommendation Systems","cited_arxiv_id":"2502.10284","evidence_quote":"It defines the strongest offline baseline (FS-LTR) and the multi-stage consistency approach that EGA-V1 claims to surpass."}],"review_version":1}