{"id":"ce337389-3765-4746-b0e6-aab2fa552977","arxiv_id":"2502.06097","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training a generative re-ranker on relative scores of neighboring item lists, plus a sampling-based non-autoregressive decoder, improves CTR and GMV in Meituan's food delivery recommender.","lead":"A Meituan research team trained a recommender re-ranking model to improve a list of recommendations by studying what happens when one item in the list is swapped for another. They report better click rates and revenue in an online test on the Meituan food delivery platform, plus gains on the public Taobao Ad dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains rely on HR@10% computed by NLGR-E, the same evaluator that provided training rewards; §5.1.3 admits HR only measures evaluator-generator consistency. Independent validation of NLGR-E on counterfactual lists is needed.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: NLGR-E must produce rewards that are faithful proxies for real user utility on counterfactual lists. I agree. The paper's own §5.1.3 concedes that HR measures consistency between evaluator and generator, not utility, which is a manuscript passage that flags the limitation. Since NLGR-G's loss is a direct function of NLGR-E's reward (Eq. 16, Eq. 18), a high HR can be achieved by fitting NLGR-E's inductive biases rather than improving real user satisfaction. The online A/B test is a genuine independent positive signal, but it is reported without confidence intervals, uses a PRM variant as baseline, and therefore does not establish that NLGR outperforms the state of the art. The missing NAR4Rec baseline and undisclosed hyperparameters (temperature τ, stop-condition thresholds) further reduce confidence but are secondary. The correct verdict remains CONDITIONAL: the method is plausible and the online deployment is encouraging, but the offline SOTA claim needs an independent list-reward evaluation before full acceptance.","tokens_in":12844,"tokens_out":3939,"duration_ms":39447,"concrete_test":"Conduct a randomized-list validation using the online serving logs: randomly sample sessions where NLGR-G's output list was exposed, and collect per-list actual click and conversion labels. Compute the true list reward using Eq. 14 with actual labels, and compare NLGR-E's predicted reward ranking to the true reward ranking on these exposed NLGR lists (e.g., rank correlation or NDCG@10). If NLGR-E's ranking of generated lists does not match actual outcomes, Table 3's HR is not evidence of user utility. A complementary check is to re-run the offline HR evaluation with an independent oracle (human raters, or a separately trained listwise model that never saw NLGR-G's outputs) scoring all candidate lists; if NLGR's advantage over DCDR and GRN disappears, the circularity concern is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5.2.1 ('significantly and consistently outperforms the state-of-the-art approaches in all metrics on both datasets') depends on Table 3, where HR is computed by ranking NLGR-G's generated lists with NLGR-E (§5.1.3). But NLGR-G is trained to maximize NLGR-E's reward through the counterfactual loss in Eq. 16 and Eq. 18. This creates a closed loop: the generator is optimized to fit NLGR-E's preferences, and the same evaluator then grades the generator. Section 5.1.3 explicitly states that HR 'measures the consistency between the evaluator and generator' and that 'the generator cannot be fully and fairly evaluated.' That is a stated limitation, and it directly undercuts using HR as evidence of user utility. Moreover, NLGR-E is trained only on exposed lists (§4.3.1), whereas NLGR-G's neighbor lists are counterfactual and were never shown to users. The evaluator therefore operates under distribution shift, with no validation on the lists it is asked to score. The online A/B test is the only independent signal, but it is reported without confidence intervals or significance tests and compares against a PRM variant, not against the generative baselines GRN or DCDR. The offline 'state-of-the-art' claim is therefore not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NLGR, a generative reranking framework in the evaluator-generator paradigm. It introduces two main ideas: (i) a neighbor-list based training procedure in which the generator NLGR-G is trained using relative rewards computed by an evaluator NLGR-E on the current list and its single-item-replacement neighbor lists, and (ii) a sampling-based non-autoregressive generation method comprising a Position Decision Unit (PDU) and a Candidate Retrieval Unit (CRU) that flexibly jumps to neighbor lists. Experiments are conducted on the Taobao Ad public dataset and a large Meituan industrial dataset, with offline metrics (AUC, LogLoss, NDCG, HR) and an online A/B test on Meituan. The paper claims significant and consistent improvement over six baselines in all offline metrics on both datasets, and online CTR +3.25% and GMV +3.07% versus a PRM-based baseline, with deployment on the Meituan platform.","tokens_in":13109,"tokens_out":4626,"duration_ms":38410,"significance":"The neighbor-list relative reward mechanism targets a real limitation of evaluator-generator reranking paradigms, and the non-autoregressive sampling decoder is a plausible efficiency-oriented alternative to sequential decoding. If the reported results are valid, the method would be a practically useful contribution, especially given the successful deployment at Meituan. However, the primary offline metric for the generator (HR) is computed by the same evaluator that supplies the training rewards, creating a circular evaluation. The online A/B test is independent but lacks statistical detail and does not compare against the generative baselines. These issues currently temper the strength of the claims, though the core ideas are worth publishing if the validation is strengthened.","major_comments":[{"comment":"The HR metric used to evaluate NLGR-G is computed by ranking the generated lists with NLGR-E, the same evaluator that provides the training rewards in Eqs. (15) and (16). NLGR-G is explicitly optimized to maximize NLGR-E's estimated reward, so HR measures evaluator-generator consistency rather than user utility. The paper itself acknowledges this in Section 5.1.3 ('the generator cannot be fully and fairly evaluated'). This circularity directly undermines the central claim in Section 5.2.1 that NLGR 'significantly and consistently outperforms the state-of-the-art approaches in all metrics on both datasets,' because the offline generator gains (e.g., HR@10% of 0.8369 on Meituan) are in large part a measure of fit to the training evaluator. Please provide an independent evaluation of the generator, such as online A/B tests against GRN and DCDR with proper statistical reporting, or a counterfactual validation of NLGR-E on held-out lists with ground-truth user feedback.","section":"Section 5.1.3, Table 3"},{"comment":"NLGR-E is trained only on exposed lists (real user logs), yet it is applied to score neighbor lists in the counterfactual space that were never shown to users. The paper does not validate NLGR-E's reliability under this distribution shift. Since both the training rewards and the offline HR metric depend entirely on this extrapolation, the central derivation rests on an untested assumption. Please add an analysis of NLGR-E's calibration or accuracy on counterfactual lists, for example using logged bandit data with propensity scores or a small online validation where counterfactual lists are exposed to a control group.","section":"Section 4.3.1, Section 4.3.2"},{"comment":"Table 2 reports AUC, LogLoss, and NDCG for all models, but Section 5.1.3 states that these metrics evaluate NLGR-E, not the generator. It is unclear whether the AUC/NDCG values for the baselines are computed using the same NLGR-E as a scoring function, using each model's own internal scores, or using ground-truth labels on the reranked lists. A fair comparison requires that the same evaluation protocol be applied to all methods. Please clarify the exact computation and ensure the comparison is apples-to-apples; if AUC/NDCG are computed on NLGR-E's predictions, they mainly measure the quality of the evaluator, which is a different claim than the reranking performance of the full system.","section":"Section 5.2.1, Table 2"},{"comment":"The online A/B test is the only independent evidence of real-world improvement, but it is reported without confidence intervals, p-values, or details on the traffic split and number of users. The reported gains (CTR +3.25%, GMV +3.07%) are compared only against a 'variant of PRM' and not against the generative baselines GRN or DCDR. Given that the offline metric is circular, the online experiment carries the main burden of proof for the paper's practical claims. Please include full statistical details and, if feasible, an online comparison with a stronger generative baseline.","section":"Section 5.4, Table 6"}],"minor_comments":[{"comment":"There are multiple typos: 'Metuan' should be 'Meituan' in Tables 2, 4, 5 and in Section 5.2.1; Eq. (14) uses 'NLGC-E' instead of 'NLGR-E'; Section 5.1.2 has a missing space in 'self-attention. mechanism'.","section":"Throughout"},{"comment":"The symbol $n$ is used both for the Gumbel noise and for the size of the candidate set in Section 4.2.2; please use distinct notation to avoid confusion.","section":"Section 4.2.1, Eq. (7)"},{"comment":"The sentence 'the newly inserted item is $c = \\argmax(r^c_k)$' reuses $c$ which already denotes the candidate representation; use $k^*$ or a different variable for the selected index.","section":"Section 4.2.2, Eq. (11)"},{"comment":"The Gumbel-softmax temperature $\\tau$ is introduced in Eq. (7) but its value is not reported in the implementation details; please add it.","section":"Section 5.1.4"},{"comment":"The stop condition in Section 4.2.2 ('or the values of $r^p_j$ and $r^c_k$ are too low') is vague; specify the threshold or mechanism used in practice.","section":"Section 5.4"},{"comment":"References [1] and [2] are duplicates of the same DLCM paper, and references [36] and [37] are duplicates of the same GOMIA paper; please consolidate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a WWW Companion short paper, so some lack of detail is understandable, but the circular HR evaluation is a substantive concern that should not be brushed aside. The online deployment is a strong point, but the reported A/B test is too thin to independently support the SOTA claim. I would encourage the authors to either add a proper counterfactual validation of the evaluator or report the online experiment with full statistical rigor and a generative baseline comparison. If neither is possible, the claims should be softened to reflect that the offline results are internal consistency checks rather than user utility measurements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the neighbor-list relative-reward training and the PDU/CRU sampling decoder are genuinely new combinations, and the Meituan deployment gives the paper real industrial weight. The headline offline HR numbers, though, should not be read as evidence about user utility. Section 5.1.3 says it plainly: HR measures consistency between evaluator and generator, and \"the generator cannot be fully and fairly evaluated.\" Since NLGR-G is trained against NLGR-E's rewards (Eqs. 15-16) and then graded by NLGR-E in Table 3, the big HR lifts (0.4091 vs 0.2160 on Taobao, 0.8369 vs 0.7573 on Meituan) are mostly self-consistency. The evaluator itself sees only exposed lists in training, while the neighbor lists it scores are counterfactual; there's no validation of NLGR-E on those lists. That's the load-bearing gap.\n\nWhat the paper does well: it identifies a real goal-inconsistency problem, and the relative-reward trick is a sensible way to give the generator a directional signal in combinatorial space rather than a single scalar. The sampling-based non-autoregressive decoder is a reasonable efficiency-motivated alternative to sequential decoding, and the ablations show both pieces matter. The online A/B test is independent and positive (CTR +3.25%, GMV +3.07%), though it compares against a PRM variant rather than GRN/DCDR, and there are no confidence intervals. The authors also deserve credit for stating the HR limitation explicitly instead of burying it.\n\nMinor soft spots: NAR4Rec is cited in related work but missing from the baseline table; temperature tau and the stop condition are not specified; no code. None of these change the central method, but they make replication harder.\n\nWho this is for: practitioners working on listwise reranking in production, and anyone studying evaluator-generator training loops. The method is worth taking seriously, but the paper needs an independent offline evaluation—a separate reward model, or some counterfactual ground truth—before the strong claims in Section 5.2.1 are supported. If this crossed my desk, I'd send it to review; the idea is useful and the deployment is real, but I'd ask for that extra evaluation or a tone-down of the claims.","headline":"A useful industrial neighbor-list training trick, but the headline offline HR gains are computed by the same evaluator that trained the generator, so only the online A/B test is independent evidence.","tokens_in":13719,"tokens_out":2376,"would_cite":true,"duration_ms":21595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a generative reranker trained with neighbor-list relative rewards and a sampling-based non-autoregressive decoder outperforms existing reranking methods on both a public and an industrial dataset, and that it lifts…","keywords":["generative reranking","neighbor lists","evaluator-generator paradigm","non-autoregressive generation","Gumbel-softmax","listwise recommendation","CTR prediction","online A/B test"],"falsifier":"Serve the lists generated by NLGR-G to a small fraction of online traffic and compare NLGR-E's predicted reward for each served list with the actual observed click and conversion behavior; if the rank correlation is weak for these generated lists, the relative-reward training signal is not a faithful proxy for user utility.","tokens_in":12615,"feed_emoji":"📈","tokens_out":4694,"duration_ms":38535,"temperature":0.7,"pith_summary":"This paper proposes a reranking method, NLGR, for recommender systems that reorders an initial list of candidates to maximize list-level user engagement. It claims that training the generator on 'neighbor lists'—lists that differ from the current one by a single item—resolves a goal inconsistency between the evaluator and the generator, so the generator learns to move toward better regions of the combinatorial space rather than imitating historical exposure. It also claims that a sampling-based non-autoregressive decoder, which picks an item position and then a replacement item, explores that space more flexibly than one-by-one autoregressive generation. The authors report consistent offline gains over six baselines on the Taobao Ad and Meituan datasets, and an online A/B test on Meituan with CTR up 3.25% and GMV up 3.07%.","feed_headline":"Neighbor-list reranking lifts Meituan CTR 3.25% and GMV 3.07%","feed_subtitle":"Training on single-item neighbor lists with non-autoregressive sampling beats six baselines on two datasets.","key_machinery":"The central object is the neighbor list: a list that differs from the current candidate list by exactly one item. The relative reward $r_j - r_o$ for each position $j$, computed by the evaluator NLGR-E, converts the evaluator's listwise scores into a differentiable training signal that tells the generator which single-item move improves utility. The generator's sampling-based non-autoregressive decoder consists of the Position Decision Unit (PDU), which chooses the position to replace via Gumbel-softmax sampling, and the Candidate Retrieval Unit (CRU), which chooses the replacement item, allowing the model to jump to any neighbor list in one step.","core_discovery":"On the paper's own terms, the discovery is that the goal inconsistency between evaluator and generator in generative reranking can be resolved by training the generator on neighbor lists: for each candidate list, the evaluator scores the list and each single-item replacement, and the generator's objective is to maximize the sum of relative rewards. The paper also claims that generating a replacement by first sampling the position to change (Position Decision Unit) and then sampling the item to insert (Candidate Retrieval Unit), both with Gumbel-softmax, yields a non-autoregressive search that jumps flexibly between neighbor lists and outperforms one-by-one sequence generation. It reports consistent offline gains over all baselines on Taobao Ad and Meituan, and an online A/B test on Meituan showing CTR +3.25% and GMV +3.07%.","pith_inferences":["The same neighbor-list relative-reward training could apply to other combinatorial list-optimization tasks, such as ad slate optimization or search result diversification, wherever a differentiable evaluator exists.","A testable extension is to replace the hand-designed reward combination (the business parameters $k_1$ and $k_2$) with a learned reward, which could remove the need for per-business tuning.","Because the offline HR metric is computed by ranking generated lists with the same NLGR-E, the offline gains partly measure generator-evaluator consistency; the online A/B test is the cleaner evidence that real user utility improved."],"forward_implications":["Only the generator is served online; the evaluator is used only to train it, so deploying NLGR adds no new online evaluation cost beyond the generator's own inference.","The neighbor-list construction requires $m$ evaluator calls per training list, which is tractable for realistic list sizes (e.g., $m=4$ on Meituan, $m=5$ on Taobao Ad).","The method's gains are consistent across two datasets of very different scale and feature richness, suggesting the training signal transfers to diverse recommender settings.","The online A/B result implies that the offline generator improvements translate into commercially meaningful user engagement and revenue at industrial scale."],"supporting_citations":[{"why":"GRN: the generative rerank network baseline that defines the evaluator-generator paradigm this paper extends and compares against.","marker":"[13]"},{"why":"DCDR: the diffusion-based reranking baseline that is the strongest generative comparator in the experiments.","marker":"[20]"},{"why":"PRM: the listwise self-attention reranking baseline and the online baseline in the A/B test.","marker":"[26]"},{"why":"NAR4Rec: prior non-autoregressive generative reranking work that motivates the proposed sampling-based decoder.","marker":"[28]"},{"why":"DIF: the decoupled feature-fusion approach that inspires the D-Attention unit in the evaluator.","marker":"[34]"},{"why":"Gumbel-softmax: the categorical reparameterization technique used to make the position and item sampling differentiable.","marker":"[17]"},{"why":"Seq2Slate: the pointer-network sequence generation approach used as the autoregressive ablation baseline.","marker":"[4]"},{"why":"Hit Ratio: defines the HR@10% and HR@1% metrics used to evaluate the generator offline.","marker":"[3]"}],"fun_headline_variants":["Neighbor-list reranking: non-autoregressive gains on Meituan","Meituan CTR +3.25% from neighbor-list non-autoregressive search","Rerank with neighbor lists: +3.25% CTR, +3.07% GMV on Meituan","Neighbor-list generative rerank lifts Meituan CTR and GMV"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that NLGR-E's estimated list rewards are faithful proxies for real user utility when applied to lists that were never shown to users; if the evaluator is biased on these counterfactual lists, the generator may simply overfit to the evaluator, and the offline and possibly online gains would not reflect genuine user satisfaction.","fun_headline_variants_meta":{"raw":{"variants":["Neighbor-list reranking: non-autoregressive gains on Meituan","Meituan CTR +3.25% from neighbor-list non-autoregressive search","Rerank with neighbor lists: +3.25% CTR, +3.07% GMV on Meituan","Neighbor-list generative rerank lifts Meituan CTR and GMV"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1638,"prompt_tokens":972,"completion_tokens":666,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":572}},"tokens_in":588,"tokens_out":666,"duration_ms":5814,"temperature":1.0,"reasoning_tokens":572,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:47:01.631251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Serve the lists generated by NLGR-G to a small fraction of online traffic and compare NLGR-E's predicted reward for each served list with the actual observed click and conversion behavior; if the rank correlation is weak for these generated lists, the relative-reward training signal is not a faithful proxy for user utility.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DCDR: the diffusion-based reranking baseline that is the strongest generative comparator in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DIF: the decoupled feature-fusion approach that inspires the D-Attention unit in the evaluator."}],"review_version":1}