{"id":"53f86adf-c3ed-42ea-9dad-965ae03441ca","arxiv_id":"2607.18689","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An exposure-based policy-gradient estimator for ranking is proposed; the appendix retracts the key derivation, and the reported experiments were not rerun with the corrected estimator.","lead":"This paper proposes reinforcement-learning estimators that let ranking models optimize exposure-based metrics through ordinary auto-differentiation, claiming faster and stabler convergence than custom-gradient methods. An appendix retracts the core gradient derivation and says the experiments were not rerun, so the headline claims are not yet supported.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix A retracts Eq. 33, the exposure-based estimator used in all main experiments; central claim unsupported.","rationale":"The reader's weakest assumption precisely identifies the load-bearing flaw: Eq. 33's baseline correction is invalid because it ignores the unplaced-document probability. The paper's own appendix confirms this and admits the experiments were not rerun with the corrected estimator. This is a correctness issue, not a matter of consensus. The central contribution—a plug-and-play auto-differentiable exposure-based estimator—is therefore empirically unsupported. The experimental results in Section 7 cannot be attributed to the proposed method's intended gradient. While the marginalization and baseline-correction ideas for utility-based estimators are interesting and may be valid, they are not the paper's stated central claim. The verdict of REJECT is appropriate because the key empirical claims rest on a known-incorrect estimator, and no rerun is provided to salvage them.","tokens_in":23284,"tokens_out":2150,"duration_ms":23208,"concrete_test":"Rerun the Section 7 RL-for-LTR experiments (Table 1) on MSLR-Web30k and Istella-S using the corrected exposure-based estimator from Listing 3/Eq. 39, with identical neural architecture (3-layer [1024,512,256] ReLU for all methods except PL-Rank's 2-layer sigmoid), Adamax lr=1e-3, batch size 128, 25 random seeds, and N ∈ {2,5,10,25,50,100,250,500,1000}. Select checkpoints by validation NDCG@10 over 15,000 epochs, and compare final test NDCG@10 and convergence speed against PL-Rank, Standard, Placement, and Marginalize-All. If the corrected estimator no longer shows significant improvement (two-sided t-test, p<0.01) for N≥5, the central claim is refuted; if it still does, the claim survives but must be restated with the corrected estimator.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim—that the exposure-based estimator (Eqs. 33–34) converges faster and achieves higher NDCG@10 than PL-Rank for N≥5—rests on the unbiasedness of Eq. 33. Appendix A explicitly shows this is false: Eq. 36 notes that Σ_k P(y_k=d|π) ≤ 1, so the leave-one-out baseline b_d does not cancel inside the expectation because the probability of the document not being placed is omitted. Consequently, Eq. 33 is a biased estimator of ∂θ_d/∂f, not the unbiased policy gradient claimed in Sec. 5.1. The main experiments (Tables 1–2, Figures 2–5) were run with this incorrect estimator (Listing 1), and the corrected estimator (Listing 3, Eq. 39) was not rerun, as the appendix states: 'the experiments could not be repeated with the corrected estimator.' Therefore, the empirical evidence for the headline result is invalidated. The other contributions—baseline corrections and marginalization for utility-based estimators—remain sound, but they do not rescue the exposure-based claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes several policy-gradient estimators for learning-to-rank (LTR) with Plackett-Luce ranking models, focusing on variance reduction via baseline corrections and partial marginalization, and introduces an 'exposure-based' estimator that places gradient estimation behind an exposure distribution to enable plug-and-play auto-differentiation. The authors claim that this exposure-based estimator converges faster and achieves significantly higher NDCG@10 at convergence than PL-Rank and other RL-for-LTR estimators on two benchmark datasets, and that it simplifies optimization of exposure-based losses. The paper also reports instability of PL-Rank under 32-bit floating point and provides implementation listings.","tokens_in":23591,"tokens_out":2468,"duration_ms":23084,"significance":"If the central claim were correct, the exposure-based estimator would be a practical and important contribution: it would let practitioners optimize any differentiable exposure-based loss without custom gradient implementations, while remaining computationally competitive on GPUs. The paper is generally well written and transparent, and it provides code listings and publicly available implementations, which are strengths. However, the exposure-based estimator as presented in the main text (Eq. 33) is explicitly retracted in Appendix A, and the main experiments were run with that incorrect estimator. The corrected estimator was not evaluated, so the headline experimental results do not support the paper's central claim. The ancillary contributions — baseline corrections and marginalization for utility-based estimators — appear sound, but they do not rescue the exposure-based claim. The Appendix's explicit acknowledgment that 'the experiments could not be repeated with the corrected estimator' is a decisive missing-support element.","major_comments":[{"comment":"The paper itself states that Eq. 33 is incorrect: the leave-one-out baseline b_d does not cancel because the probability of a document not being placed is omitted (Eq. 36 shows Σ_k P(y_k=d|π) ≤ 1). Since Eq. 33 is the estimator used in all main experiments (Section 7, Listing 1), the reported NDCG@10 improvements in Tables 1–2 and Figures 2–5 are based on a biased gradient estimator. This directly invalidates the central claim in Section 7.3 that 'the exposure-based estimator provides significant improvements in performance at convergence and learning speed over other RL for LTR estimators, as long as N≥5.'","section":"Appendix A, Eq. 36"},{"comment":"The headline conclusion rests on experiments that were not rerun with the corrected estimator. Appendix A provides a corrected formula (Eq. 39) and implementation (Listing 3), but states that the experiments could not be repeated before the deadline. Without experiments using the corrected estimator, there is no empirical evidence that the exposure-based estimator outperforms the other methods; the published learning curves and significance tests are not trustworthy evidence for the proposed method.","section":"Section 7.3 / Conclusion"},{"comment":"The derivation of the exposure-based gradient estimator is load-bearing for the entire method. The chain rule (Eq. 34) requires an unbiased estimate of ∂θ_d/∂f. Appendix A shows that the proposed estimator is biased. The corrected estimator introduces an additional term for the document not being placed in any position. Consequently, the claimed 'seamless plug-and-play integration with auto-differentiation' (Abstract, Section 5.2) is not achieved by the method evaluated in the paper.","section":"Section 5.1, Eq. 33-34"},{"comment":"Listing 1 implements the incorrect estimator (Eq. 33) and is the one used in the experiments. Listing 3 implements the corrected estimator (Eq. 39). The paper never reports results with Listing 3. The gap between the submitted main text and its own appendix is an explicit limitation: the central empirical evidence is retracted but the main-text claims are not revised. A revised manuscript would need new experiments, not just a rewording.","section":"Listing 1 vs Listing 3"}],"minor_comments":[{"comment":"The text says 'when we consider the loss values reached in Table 1' but should refer to Table 2, which reports the non-traditional loss values.","section":"Section 7.5"},{"comment":"The notation 'δθ_d/δf' uses δ both as functional derivative and as partial derivative; consider standard partial-derivative notation to avoid ambiguity.","section":"Section 5.1, Eq. 35"},{"comment":"The caption says 'without baseline corrections' but PL-Rank never has baseline corrections; it might be clearer to state 'with b=0 for all estimators' explicitly.","section":"Section 7.1, Fig. 2"},{"comment":"The phrase 'for each sampled ranking of length K, D rankings of length one are considered...' is slightly confusing because the initial prefix is length zero; clarify that the marginalization enumerates placements from each prefix.","section":"Section 4.4"}],"recommendation":"reject","confidential_remarks":"The Appendix is candid about the error, but the paper as submitted is not sound. The central empirical claim is unsupported by evidence, and the corrected estimator has not been evaluated. This is a standard case for rejection with a clear path to resubmission after substantial new experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the paper's central claim is not supported by its own evidence. Appendix A explicitly states that Eq. 33, the exposure-based estimator used in all main experiments, is incorrect. The baseline subtraction doesn't cancel because the probability of a document not being placed is omitted. The authors provide a corrected estimator but admit the experiments could not be rerun before the deadline. So Tables 1–2 and Figures 2–5 were produced with the biased estimator, and any conclusion about the exposure-based estimator's convergence speed or NDCG@10 is unsupported.\n\nThat said, the paper is not a waste. The baseline-corrected placement estimator (Eq. 25) and the marginalize-first/all estimators are new, as far as I know. The marginalize-all estimator's behavior under baseline corrections—it does worse with baselines, not better—is a genuinely interesting observation about estimator/reward alignment. The exposure-distribution abstraction is a clean idea for auto-differentiation frameworks, and the implementation is clear and reproducible in JAX. The appendix is refreshingly candid: the authors identify exactly where the derivation fails, derive the corrected estimator, and say plainly that the re-run is unfinished. That's good research hygiene.\n\nThe soft spots beyond the main one: the PL-Rank instability claim is confounded. PL-Rank was run with a smaller network, sigmoid activations, 32-bit floats, on GPU, and for more epochs than prior work—all plausible explanations for the divergence. Calling this a fair comparison is not quite right because the architecture differs. Also, the non-traditional losses comparison replaces PL-Rank with the placement estimator, so it doesn't actually test the case against the method it claims to replace.\n\nProportionally: the marginalization and baseline analysis could be a solid workshop paper on its own. The corrected exposure estimator is a real contribution waiting for experiments. But as it stands, the headline result—exposure-based is faster and better than PL-Rank—is not established. The corrected estimator might behave similarly, or it might not; until the experiments are rerun, we don't know.\n\nWho gets value from this: people working on policy-gradient estimation for LTR, or studying honest self-correction in ML papers. For a reading group, it's a useful case study in how a subtle derivation gap can sink a headline claim. I'd send it to peer review only because the appendix already contains the fix; the work could come back stronger after a re-run. My own verdict, though, is reject in current form.","headline":"The paper retracts its own main estimator in an appendix, so the headline experiments are invalidated, but the marginalization and baseline-correction ideas have real merit.","tokens_in":23998,"tokens_out":2633,"would_cite":false,"duration_ms":81038,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that an exposure-based, auto-differentiable estimator makes reinforcement learning for ranking faster, stabler, and easier to use than existing custom-gradient methods—while an appendix retracts the estimator's derivation a","keywords":["Learning to Rank","Reinforcement Learning","Policy Gradient","Exposure","Plackett-Luce","Baseline Correction","Marginalization","Auto-Differentiation"],"falsifier":"For a small Plackett-Luce model (e.g., 3 documents, K=2), compute the exact exposure gradient by finite differences and compare it to the gradient from Eq. 33; the mismatch equals the omitted unplaced-document term identified in Eq. 36.","tokens_in":23215,"feed_emoji":"🎯","tokens_out":6582,"duration_ms":60425,"temperature":0.7,"pith_summary":"The authors set out to make reinforcement learning for learning-to-rank practical: instead of writing custom gradients, they estimate the gradient of a document-exposure distribution from sampled Plackett-Luce rankings, using leave-one-out baselines and partial marginalization to cut variance. Their headline claim is that this exposure-based estimator converges much faster and to higher NDCG@10 than the existing PL-Rank custom gradient, and that it works in a plug-and-play way with auto-differentiation for any exposure-based loss. The paper's own appendix, however, retracts the key derivation: subtracting a per-document baseline from each position weight inside the sum over positions is not unbiased, because a document may be absent from the ranking. The experiments in the main text use the uncorrected estimator and were not rerun, so the empirical claims rest on a known-incorrect gradient.","feed_headline":"Exposure-based RL-to-rank: faster convergence, retracted derivation","feed_subtitle":"The plug-and-play gradient estimator for ranking losses misses the unplaced-document outcome, so the headline experiments rest on an error.","key_machinery":"The central object is the exposure-based gradient estimator (Eqs. 33–34): for each sampled ranking, it sums over positions k the baseline-corrected position weight θ_k−b_d multiplied by the derivative of the placement probability plus the placement probability times the log-prefix derivative. This is wrapped in a function that returns the estimated exposure distribution, so any loss written as a differentiable function of exposure (relevance utility, fairness ratios, KL-divergence distillation) gets an automatic gradient through the chain rule. The abstraction's correctness collapses on the baseline: the retraction shows the exposure estimate has K+1 outcomes—placement in one of K positions,","core_discovery":"On its own terms, the paper claims the policy-gradient problem for ranking can be pushed behind the exposure distribution: estimate each document's expected exposure by marginalizing over sampled prefixes, correct with leave-one-out baselines, and let the chain rule hand any differentiable exposure-based loss its gradient. The authors report that this exposure-based estimator (Eqs. 33–34) reaches significantly higher NDCG@10 at convergence than PL-Rank and other RL-for-LTR estimators for N≥5, converges in about a third of the epochs, and stays stable where PL-Rank's custom gradient diverges under 32-bit floats. Appendix A retracts Eq. 33: since Σ_k P(y_k=d|π) ≤ 1, the per-position baseline d","pith_inferences":["A natural next step is to rerun the NDCG@10 comparisons with the corrected estimator; there is no evidence yet that the reported speed and convergence advantages survive.","The same baseline-vs-absence failure pattern may affect any estimator that treats exposure as a sum over positions without accounting for non-placement, suggesting a general caution for exposure-based RL formulations.","If the PL-Rank instability claim holds, previously published CPU-era comparisons of RL-to-rank estimators may need re-examination under GPU and 32-bit float conditions."],"forward_implications":["If the estimator were unbiased, any exposure-based ranking objective could be optimized in a few lines of differentiable code, removing the need for custom-gradient implementations.","The reported results imply that with N≥5 samples, the exposure-based estimator matches or beats standard and placement estimators in speed and converged NDCG@10, and with N≥100 on MSLR the gains are statistically significant.","The paper claims PL-Rank's unnormalized intermediate quantities make it numerically unstable when run for many epochs at 32-bit precision, a failure mode not visible in earlier short runs.","The corrected estimator preserves the plug-and-play exposure abstraction and adds an unplaced-document probability term, but its empirical performance is unverified in this paper."],"fun_headline_variants":["Exposure-based RL-to-rank retracts key derivation","Retracted equation sinks exposure-based ranking claims","RL-to-rank speedup relies on retracted math","Leave-one-out baseline fails: paper pulls its own proof","Unplaced-document oversight breaks RL-to-rank estimator"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"That the leave-one-out baseline b_d cancels inside the per-position sum, leaving an unbiased gradient of expected exposure; the appendix itself shows this fails because a document can be unplaced, so Σ_k P(y_k=d|π) ≤ 1.","fun_headline_variants_meta":{"raw":{"variants":["Exposure-based RL-to-rank retracts key derivation","Retracted equation sinks exposure-based ranking claims","RL-to-rank speedup relies on retracted math","Leave-one-out baseline fails: paper pulls its own proof","Unplaced-document oversight breaks RL-to-rank estimator"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1570,"prompt_tokens":813,"completion_tokens":757,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":682}},"tokens_in":557,"tokens_out":757,"duration_ms":13000,"temperature":1.0,"reasoning_tokens":682,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:37:22.659780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a small Plackett-Luce model (e.g., 3 documents, K=2), compute the exact exposure gradient by finite differences and compare it to the gradient from Eq. 33; the mismatch equals the omitted unplaced-document term identified in Eq. 36.","supporting_citations":[],"review_version":1}