Pith. sign in

REVIEW 3 major objections 5 minor 78 references

Ranking policies improve when reward gains are measured per unit of ranking change, not by reward alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:53 UTC pith:HCALVUB6

load-bearing objection Plausible heuristic with a broad empirical sweep, but Eq. (5) is not a true Kendall-tau distance—it's not invariant under candidate relabeling, which undermines the paper's central mechanism. the 3 major comments →

arxiv 2607.25268 v1 pith:HCALVUB6 submitted 2026-07-28 cs.IR cs.AI

Structure-aware Relative Policy Optimization for Ranking

Yiteng Tu , Weihang Su , Zitao Su , Yiqun Liu , Min Zhang , Qingyao Ai This is my paper
classification cs.IR cs.AI
keywords learning to rankreinforcement learningrelative policy optimizationlistwise rankingKendall tau distancecoarse-grained feedbackLLM reranking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing reinforcement-learning methods for ranking score each sampled list by its scalar reward, ignoring how similar or different two lists are as permutations. This paper proposes SRPO, which measures each list's advantage as the reward it gains per unit of structural change, using a top-weighted Kendall-tau distance between lists. Rewarding efficient local reorderings while down-weighting disruptive global reshuffles makes listwise RL more stable, more sample-efficient, and less sensitive to noisy feedback. In experiments on three learning-to-rank benchmarks, a fairness objective, and an LLM-based reranker, SRPO consistently outperforms reward-only GRPO and other baselines.

Core claim

Core claim: with coarse-grained listwise rewards, a sampled permutation's advantage should reflect reward gained per unit of structural change, not reward alone. SRPO divides pairwise reward differences by a top-weighted Kendall-tau distance, aggregates the resulting structure-normalized preferences into a bounded, zero-centered contrastive advantage, and updates the policy at each position. This favors small local reorderings and suppresses drastic global reshuffles. Experiments show consistent gains over reward-only GRPO on three learning-to-rank benchmarks, better fairness-relevance trade-offs, strong performance with two to eight samples per query, and improved out-of-domain reranking ge

What carries the argument

The central mechanism is the top-weighted Kendall-tau distance d(L_i,L_j)=Σ_{a<b} w_a w_b I[ordered differently by L_i and L_j], with position weights w_a=1/[log2(a+1)]^η, which counts pairwise disagreements between two rankings while amplifying disagreements near the top. SRPO normalizes each pairwise reward difference ΔR_ij by this distance, giving a structure-normalized preference S_ij = ΔR_ij / (d+ε). These preferences are aggregated into a contrastive advantage via tanh and scaled by the group reward's standard deviation, producing a bounded, zero-sum advantage. The same advantage is applied per ranking position rather than to the whole list.

Load-bearing premise

The load-bearing premise is that the top-weighted Kendall-tau distance, with its tunable position-weight exponent η, captures the 'right' geometry of ranking changes—so that dividing reward differences by this distance assigns credit proportionally to genuinely useful improvements.

What would settle it

Decisive observation: on a held-out set of queries with known relevance labels, compute for each sampled list its SRPO advantage and its actual marginal contribution to the ranking metric (e.g., leave-one-out NDCG gain over the policy's expected list). If the Spearman correlation between SRPO advantage and marginal contribution is not higher than the same correlation for a reward-only advantage, the structural normalization is not adding useful signal.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SRPO's advantage estimation is sample-efficient: strong results with two to eight sampled lists per query make listwise RL practical in settings where repeated feedback is scarce.
  • The gains transfer across three different objectives—relevance, exposure fairness, and cross-domain reranking—indicating that structural normalization is a general property of ranking geometry rather than a reward-specific trick.
  • Because the advantage is bounded and zero-centered, policy updates remain conservative under noisy or extreme rewards, which should reduce catastrophic forgetting and over-optimization during RL training.
  • The position-weighting ablation shows that top-position disagreements carry the most informative signal, implying that ranking metrics' emphasis on top positions is being reflected in the learning signal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to replace the hand-set position weights and temperature with quantities derived from the target ranking metric's gain function, which would make the distance normalization self-tune to the reward.
  • The same 'reward per unit of structural change' idea could apply to other structured output problems—set selection, graph alignment, or itinerary planning—wherever a ground metric between outputs exists.
  • Since the method suppresses globally disruptive reorderings, it may act as an implicit regularizer; one test would be checking whether SRPO preserves a pretrained ranker's ability when fine-tuning on a new domain.
  • The paper's stability claim could be stress-tested by measuring gradient variance directly; the prediction is that SRPO's per-step gradient norm variance is lower than GRPO's under identical sampling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SRPO, a structure-aware variant of GRPO for listwise ranking. SRPO defines a top-weighted Kendall-tau distance between sampled permutations, normalizes pairwise reward differences by this distance, aggregates the resulting preferences into a contrastive advantage, and optimizes the policy at the level of individual ranking decisions. Experiments on three LTR datasets and an LLM-based reranker (E2Rank) claim consistent improvements over GRPO and other RL baselines, including under limited group sizes and noisy rewards.

Significance. The idea of incorporating permutation-level structure into advantage estimation for RL-based ranking is timely and potentially useful. The paper contributes a framework, an open-source implementation, and experiments across multiple ranking scenarios. However, the formal definition of the structural distance in Eq. (5) is not a valid distance between rankings and is not invariant under relabeling of candidates; this directly undermines the claimed mechanism. In addition, the empirical support for the headline claim ('SRPO outperforms GRPO') is weak because results are reported as point estimates without error bars or significance tests against GRPO. The manuscript is not circular, and the code release is a strength, but the central formal defect and the insufficient statistical evidence require substantial revision.

major comments (3)
  1. [§3.2.1, Eq. (5)] The distance in Eq. (5) is not a Kendall-tau distance and is not invariant under relabeling of candidates. It sums over position pairs (a,b) and checks the relative order of the items at those positions in the two lists; these are generally different pairs of items. For k=3, L_i=[A,B,C], L_j=[A,C,B]: with id order A<B<C Eq. (5) gives d=1, but with id order C<A<B (id(A)=2,id(B)=3,id(C)=1) it gives d=3. Thus the same two rankings receive different 'structural discrepancy' depending on the arbitrary input order of X. Since every S_ij divides by d+ε, the advantage and all policy updates depend on this arbitrary preprocessing. The central claim—reward improvement per unit of ranking change—is therefore not well-defined. The fix is to sum over item pairs with weights based on item positions in one of the lists, e.g., d(L_i,L_j)=Σ_{x<y} w_{p_i(x)}w_{p_i(y)} I((p_i(x)-p_i(y))(p_j(x)-p_j(y))<0),
  2. [§5.1, Tables 1/5/6] The central empirical claim that SRPO outperforms GRPO is not statistically supported. Tables report single point estimates without error bars or multiple seeds, and the significance markers are computed against the best supervised baseline, not against GRPO. Many SRPO-vs-GRPO margins are tiny—e.g., Table 1, Istella G=8 ERR@3: 0.7041 vs 0.7035; Table 5, MSLR G=8 NDCG@10: 0.4530 vs 0.4520—well within typical seed variance for LTR. Please report mean±std over at least 3 seeds and run pairwise significance tests (e.g., paired t-test or Wilcoxon) between SRPO and GRPO for each dataset and group size. Without this, the abstract's claim of improved effectiveness and stability is not established.
  3. [§5.3, Table 3] The ablation results are not interpretable until Eq. (5) is corrected. The 'w/o PW' variant removes weights from the position-pair distance, but since that distance is not a valid structural measure, the ablation tests an ill-specified quantity. A correct top-weighted Kendall-tau distance may respond differently to position weighting. The conclusions drawn from Table 3 (e.g., that top-weighted distance is beneficial, that tanh and std placement matter) should be re-evaluated after fixing the metric.
minor comments (5)
  1. [Figure 3] The fairness figure reports training curves and trade-off scatter plots without final numeric values or error bars. Please add a table with final fair@10 and NDCG@10 values (mean±std) to support the claimed Pareto dominance quantitatively.
  2. [Table 2] Several values are run together in the BEIR block (e.g., '0.38310.4334', '0.74620.3593'). Please fix the formatting so each metric is readable.
  3. [Appendix B] The 'desirable properties' are stated informally. Some, like 'Variance Reduction', follow from antisymmetry of S_ij and oddness of tanh, but 'Dynamic Controllability' is vague and 'Smoother Optimization Landscape' conflates boundedness with smoothness. Please provide precise statements or proofs, or mark these as intuitive motivations.
  4. [§3.2.2] The temperature α is set to 1 with no sensitivity analysis. Given that the advantage estimator is central, please report a small study over α (e.g., 0.5, 1, 2) on at least one dataset.
  5. [§3.1.2] Eq. (4) is described as a simplified one-step GRPO without clipping, following [63,72]. Please clarify whether the official GRPO clipping is omitted and, if so, why this does not affect the comparison with the GRPO baseline, which presumably uses the same simplification.

Circularity Check

0 steps flagged

No significant circularity: SRPO's structure-aware advantage is a new heuristic with independent empirical evaluation; self-citations are peripheral.

full rationale

SRPO's derivation chain is constructive rather than circular. The paper defines a top-weighted Kendall-tau-style distance d(L_i,L_j) in Eq. (5), defines a structure-normalized preference S_ij = ΔR_ij/(d+ε) in Eq. (6), aggregates these into a contrastive advantage in Eq. (7), and inserts this advantage into an action-level policy objective in Eq. (8). Each of these is a definition or design choice, not a derivation of the empirical result; the claim that structure normalization improves effectiveness and stability is tested against GRPO and other baselines on held-out LTR and reranking benchmarks (Tables 1–6, Figures 3–5). No fitted parameter is renamed as a prediction: η is selected by validation and α is fixed, and the paper explicitly treats the properties in Appendix B as properties of the formula (e.g., tanh oddness, boundedness), not as a proof of downstream ranking gains. The self-citations to the authors' prior work [62, 63] supply baseline implementations and training conventions (GRPO adapted to ranking, simplified one-step objective, fairness reward details), but the central SRPO mechanism does not rest on any self-cited uniqueness theorem or imported ansatz. The skeptic's concern that Eq. (5) is not invariant under relabeling of candidate IDs is a formal correctness/robustness issue, not a circularity: it challenges whether the distance is well-defined, but it does not make the method's output equivalent to its input. The paper is therefore self-contained with respect to the circularity criteria.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on a heuristic advantage estimator with two tuned hyperparameters (η, α) and asserted but unproven properties (Appendix B). No new physical entities are introduced. The Plackett-Luce policy and simulated listwise rewards are inherited from prior work.

free parameters (3)
  • η (top-weighted distance scale) = not reported; chosen from {1,2,3}
    Eq (5): weights disagreements at position a by 1/[log2(a+1)]^η. Grid-searched on validation data per dataset (Appendix E.1.2); controls how strongly top-position inversions dominate the structural distance.
  • α (contrastive temperature) = 1
    Eq (7): sharpness of the tanh pairwise comparison; set to 1 without sensitivity analysis, yet it modulates the scale of the advantage.
  • β (KL penalty weight) = range [0, 0.1]
    Common GRPO hyperparameter; grid-searched per method in Appendix E.1.2. Not unique to SRPO but affects all RL comparisons.
axioms (5)
  • domain assumption Plackett-Luce policy over permutations
    Eq (2): the ranking policy is a PL distribution over scores; inherited from prior work [63] and standard in listwise RL ranking.
  • domain assumption A scalar listwise reward R(L) is available and faithful
    Eq (1): in LTR, rewards are simulated from relevance labels via NDCG/ERR; in reranking, NDCG@10 is used. If coarse-grained feedback is noisy or misaligned with true user utility, the optimization target is misspecified.
  • ad hoc to paper Top-weighted Kendall-tau distance is the appropriate structural discrepancy measure
    Eq (5): the paper asserts that disagreements at top positions matter more via weights 1/[log2(a+1)]^η; no derivation connects this distance to ranking-metric gradients or to the geometry of policy updates.
  • ad hoc to paper tanh-bounded pairwise aggregation yields stable advantages
    Eq (7) and Appendix B: 'variance reduction', 'smoother landscape', and 'dynamic controllability' are claimed as properties without proofs; the estimator is a heuristic.
  • domain assumption One-step simplified GRPO without clipping is a valid optimization objective
    Eq (4): follows [63,72] and omits the clipping operation; standard in this line of work but an unproved simplification.

pith-pipeline@v1.3.0-alltime-deepseek · 21491 in / 9356 out tokens · 89332 ms · 2026-08-01T02:53:35.780465+00:00 · methodology

0 comments
read the original abstract

Ranking is a fundamental component of modern information access systems. Reinforcement learning (RL) provides a flexible framework for directly optimizing coarse-grained feedback and system-level objectives defined over the complete ranking list. However, existing RL-based ranking methods typically treat each sampled permutation as an atomic output and evaluate it primarily through a scalar reward, overlooking the structural relationships among different ranking lists. Consequently, permutations with similar rewards but substantially different permutation patterns may receive comparable optimization signals, potentially leading to inaccurate credit assignment and overly aggressive policy updates. To address this limitation, we propose SRPO, a \textbf{S}tructure-aware \textbf{R}elative \textbf{P}olicy \textbf{O}ptimization framework for listwise ranking. SRPO measures the discrepancy between sampled permutations using a top-weighted Kendall-tau distance and normalizes their pairwise reward differences by the corresponding distances. It quantifies the reward improvement per unit of ranking change, thereby emphasizing efficient local refinements, particularly those involving top-ranked positions. Experimental results across two ranking scenarios demonstrate that explicitly modeling permutation-level differences improves the effectiveness and stability of listwise ranking, with particularly favorable performance in limited-feedback and complex list-level optimization settings.

Figures

Figures reproduced from arXiv: 2607.25268 by Min Zhang, Qingyao Ai, Weihang Su, Yiqun Liu, Yiteng Tu, Zitao Su.

Figure 1
Figure 1. Figure 1: An example demonstrates the impact of the ranking [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An illustration of our SRPO and its comparison with GRPO (tailored for ranking in [63]). [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The fairness results and relevance-fairness trade-off of all RL algorithms. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of the group size 𝐺, i.e., the number of sam￾pled lists for each query, on Yahoo test NDCG@10. ratio and KL divergence are computed over the probability of the complete list, treating each sampled list as a single action [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Validation curves of all algorithms and reward trajectories of listwise RL methods on Yahoo. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 19 linked inside Pith

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. 2018. Learning a deep listwise context model for ranking refinement. InThe 41st international ACM SIGIR conference on research & development in information retrieval. 135–144

    2018

  3. [3]

    Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018. Unbi- ased learning to rank with unbiased propensity estimation. InThe 41st interna- tional ACM SIGIR conference on research & development in information retrieval. 385–394

    2018

  4. [4]

    Qingyao Ai, Tao Yang, Huazheng Wang, and Jiaxin Mao. 2021. Unbiased learning to rank: online or offline?ACM Transactions on Information Systems (TOIS)39, 2 (2021), 1–29

    2021

  5. [5]

    Christopher Burges, Robert Ragno, and Quoc Le. 2006. Learning to rank with nonsmooth cost functions.Advances in neural information processing systems19 (2006)

    2006

  6. [6]

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. InProceedings of the 22nd international conference on Machine learning. 89–96

    2005

  7. [7]

    Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview.Learning11, 23-581 (2010), 81

    2010

  8. [8]

    Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: from pairwise approach to listwise approach. InProceedings of the 24th international conference on Machine learning. 129–136

    2007

  9. [9]

    Olivier Chapelle and Yi Chang. 2011. Yahoo! learning to rank challenge overview. InProceedings of the learning to rank challenge. PMLR, 1–24

    2011

  10. [10]

    Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009. Expected reciprocal rank for graded relevance. InProceedings of the 18th ACM conference on Information and knowledge management. 621–630

    2009

  11. [11]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216 4, 5 (2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. 2019. Top-k off-policy correction for a REINFORCE recommender system. InProceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 456–464

    2019

  13. [13]

    Xu Chen, Yongfeng Zhang, Qingyao Ai, Hongteng Xu, Junchi Yan, and Zheng Qin. 2017. Personalized key frame recommendation. InProceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. 315–324

    2017

  14. [14]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M Voorhees, and Ian Soboroff. 2025. Overview of the TREC 2022 deep learning track.arXiv preprint arXiv:2507.10865(2025)

    Pith/arXiv arXiv 2025

  15. [15]

    Aryan Deshwal, Syrine Belakaria, Janardhan Rao Doppa, and Dae Hyun Kim

  16. [16]

    Georges Dupret and Mounia Lalmas. 2013. Absence time and user engagement: evaluating ranking functions. InProceedings of the sixth ACM international con- ference on Web search and data mining. 173–182

    2013

  17. [17]

    Juan Feng, Zuo-Jun Max Shen, and Roger Lezhou Zhan. 2007. Ranked items auctions and online advertisement.Production and Operations Management16, 4 (2007), 510–522

    2007

  18. [18]

    Ge Gao, Jonathan D Chang, Claire Cardie, Kianté Brantley, and Thorsten Joachim

  19. [19]

    Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. 2004. Variance reduction techniques for gradient estimates in reinforcement learning.Journal of Machine Learning Research5, Nov (2004), 1471–1530

    2004

  20. [20]

    Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. 2016. Q-prop: Sample-efficient policy gradient with an off-policy critic.arXiv preprint arXiv:1611.02247(2016)

    Pith/arXiv arXiv 2016

  21. [21]

    Yulong Gu, Lixin Zou, and Chenliang Li. 2026. Deep Learning to Rank in Industrial Search Engines, Recommender Systems, and Online Advertising: An Overview and New Perspectives.ACM Transactions on Information Systems44, 4 (2026), 1–52

    2026

  22. [22]

    1954.Statistical theory of extreme values and some practical applications: a series of lectures

    Emil Julius Gumbel. 1954.Statistical theory of extreme values and some practical applications: a series of lectures. Vol. 33. US Government Printing Office

    1954

  23. [23]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incen- tivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  24. [24]

    Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W Bruce Croft, and Xueqi Cheng. 2020. A deep look into neural ranking models for information retrieval.Information Processing & Management57, 6 (2020), 102067

    2020

  25. [25]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Yunlong Jiao and Jean-Philippe Vert. 2015. The Kendall and Mallows kernels for permutations. InInternational Conference on Machine Learning. PMLR, 1935– 1944

    2015

  27. [27]

    Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay

  28. [28]

    Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search.ACM Transactions on Information Systems (TOIS)25, 2 (2007), 7–es

    2007

  29. [29]

    Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Rein- forcement learning: A survey.Journal of artificial intelligence research4 (1996), 237–285

    1996

  30. [30]

    Maryam Karimzadehgan, Wei Li, Ruofei Zhang, and Jianchang Mao. 2011. A sto- chastic learning-to-rank algorithm and its application to contextual advertising. InProceedings of the 20th international conference on World wide web. 377–386

    2011

  31. [31]

    Xiaojie Ke, Shuai Zhang, Liansheng Sun, Yongjin Wang, Hengjun Jiang, Xiangkun Liu, Cunxin Gu, Jian Xu, and Guanjun Jiang. 2026. ResRank: Unifying Retrieval and Listwise Reranking via End-to-End Joint Training with Residual Passage Compression.arXiv preprint arXiv:2604.22180(2026)

    Pith/arXiv arXiv 2026

  32. [32]

    Maurice G Kendall. 1938. A new measure of rank correlation.Biometrika30, 1-2 (1938), 81–93

    1938

  33. [33]

    Anna Korba, Alexandre Garcia, and Florence d’Alché Buc. 2018. A structured prediction approach for label ranking.Advances in neural information processing systems31 (2018)

    2018

  34. [34]

    Qi Liu, Yanzhao Zhang, Mingxin Li, Dingkun Long, Pengjun Xie, and Jiaxin Mao. 2025. E2Rank: Your Text Embedding can Also be an Effective and Efficient Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Yiteng Tu, Weihang Su, Zitao Su, Yiqun Liu, Min Zhang, and Qingyao Ai Listwise Reranker.arXiv preprint arXiv:2510.22733(2025)

    arXiv 2025

  35. [35]

    Tie-Yan Liu. 2009. Learning to rank for information retrieval.Foundations and Trends®in Information Retrieval3, 3 (2009), 225–331

    2009

  36. [36]

    Yiqun Liu, Chao Wang, Ke Zhou, Jianyun Nie, Min Zhang, and Shaoping Ma. 2014. From skimming to reading: A two-stage examination model for web search. In Proceedings of the 23rd ACM international conference on information and knowledge management. 849–858

    2014

  37. [37]

    Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Salvatore Trani. 2016. Post-learning optimization of tree ensembles for efficient ranking. InProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 949–952

    2016

  38. [38]

    1959.Individual choice behavior

    R Duncan Luce et al. 1959.Individual choice behavior. Vol. 4. Wiley New York

    1959

  39. [39]

    Marco Morik, Ashudeep Singh, Jessica Hong, and Thorsten Joachims. 2020. Con- trolling fairness and bias in dynamic learning-to-rank. InProceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval. 429–438

    2020

  40. [40]

    Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Docu- ment ranking with a pretrained sequence-to-sequence model. InFindings of the association for computational linguistics: EMNLP 2020. 708–718

    2020

  41. [41]

    Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT.arXiv preprint arXiv:1910.14424(2019)

    Pith/arXiv arXiv 2019

  42. [42]

    Harrie Oosterhuis. 2021. Computationally efficient optimization of plackett-luce ranking models for relevance and fairness. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1023–1032

    2021

  43. [43]

    Harrie Oosterhuis. 2022. Learning-to-rank at the speed of sampling: Plackett-luce gradient estimation with minimal computational complexity. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2266–2271

    2022

  44. [44]

    Harrie Oosterhuis and Maarten de Rijke. 2018. Ranking for relevance and display preferences in complex presentation layouts. InThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 845–854

    2018

  45. [45]

    Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen

  46. [46]

    Robin L Plackett. 1975. The analysis of permutations.Journal of the Royal Statistical Society Series C: Applied Statistics24, 2 (1975), 193–202

    1975

  47. [47]

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. arXiv preprint arXiv:2309.15088(2023)

    Pith/arXiv arXiv 2023

  48. [48]

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze!arXiv preprint arXiv:2312.02724(2023)

    Pith/arXiv arXiv 2023

  49. [49]

    Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 datasets.arXiv preprint arXiv:1306.2597(2013)

    Pith/arXiv arXiv 2013

  50. [50]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, et al. 2024. Large language models are effective text rankers with pairwise ranking prompting. InFindings of the Association for Computational Linguistics: NAACL 2024. 1504–1518

    2024

  51. [51]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel

  52. [52]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  53. [53]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    Pith/arXiv arXiv 2024

  54. [54]

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms. InInternational conference on machine learning. Pmlr, 387–395

    2014

  55. [55]

    Ashudeep Singh and Thorsten Joachims. 2019. Policy learning for fairness in ranking.Advances in neural information processing systems32 (2019)

    2019

  56. [56]

    Rion Snow, Brendan O’connor, Dan Jurafsky, and Andrew Y Ng. 2008. Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks. InProceedings of the 2008 conference on empirical methods in natural language processing. 254–263

    2008

  57. [57]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? inves- tigating large language models as re-ranking agents. InProceedings of the 2023 conference on empirical methods in natural language processing. 14918–14937

    2023

  58. [58]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  59. [59]

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems12 (1999)

    1999

  60. [60]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663(2021)

    Pith/arXiv arXiv 2021

  61. [61]

    Anh Tran, Tao Yang, and Qingyao Ai. 2021. ULTRA: an unbiased learning to rank algorithm toolbox. InProceedings of the 30th ACM International Conference on Information & Knowledge Management. 4613–4622

    2021

  62. [62]

    Yiteng Tu, Weihang Su, Shuguang Han, Yiqun Liu, and Qingyao Ai. 2026. Equity vs. Equality: Optimizing Ranking Fairness for Tailored Provider Needs.arXiv preprint arXiv:2602.00495(2026)

    arXiv 2026

  63. [63]

    Yiteng Tu, Zhichao Xu, Tao Yang, Weihang Su, Yujia Zhou, Yiqun Liu, Fen Lin, Qin Liu, and Qingyao Ai. 2022. Reinforcement Learning to Rank Using Coarse- grained Rewards.arXiv e-prints(2022), arXiv–2208

    2022

  64. [64]

    1998.Reinforcement learning: An intro- duction

    Richard S Sutton, Andrew G Barto, et al. 1998.Reinforcement learning: An intro- duction. Vol. 1. MIT press Cambridge

    1998

  65. [65]

    Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. InProceedings of the 25th international conference on Machine learning. 1192–1199

    2008

  66. [66]

    Jun Xu, Zeng Wei, Long Xia, Yanyan Lan, Dawei Yin, Xueqi Cheng, and Ji-Rong Wen. 2020. Reinforcement learning to rank with pairwise policy gradient. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 509–518

    2020

  67. [67]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  68. [68]

    Tao Yang and Qingyao Ai. 2021. Maximizing marginal fairness for dynamic learning to rank. InProceedings of the Web Conference 2021. 137–145

    2021

  69. [69]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al . 2026. Dapo: An open-source llm reinforcement learning system at scale.Advances in Neural Information Processing Systems38 (2026), 113222–113244

    2026

  70. [70]

    Zeng Wei, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2017. Reinforce- ment learning to rank with Markov decision process. InProceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. 945–948

    2017

  71. [71]

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. 2025. Group sequence policy optimization.arXiv preprint arXiv:2507.18071(2025)

    Pith/arXiv arXiv 2025

  72. [72]

    ∗" indicates signifi- cantly better than the best supervised learning method (un- derlined) at the 𝑝< 0.05level using the two-tailed pairwise t-test

    Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuc- con. 2025. Rank-r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning.arXiv preprint arXiv:2503.06034(2025). A Efficent Plackett-Luce Sampling Direct sampling from the ranking score distribution can be compu- tationally expensive. Following prior works [42,...

    Pith/arXiv arXiv 2025

  73. [76]

    Yongfeng Zhang, Qingyao Ai, Xu Chen, and W Bruce Croft. 2017. Joint repre- sentation learning for top-n recommendation with heterogeneous information sources. InProceedings of the 2017 ACM on Conference on Information and Knowl- edge Management. 1449–1458

    2017

  74. [2015]

    High-dimensional continuous control using generalized advantage estima- tion.arXiv preprint arXiv:1506.02438(2015)

    Pith/arXiv arXiv 2015

  75. [2017]

    InAcm Sigir Forum, Vol

    Accurately interpreting clickthrough data as implicit feedback. InAcm Sigir Forum, Vol. 51. Acm New York, NY, USA, 4–11

  76. [2020]

    InProceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval

    Setrank: Learning a permutation-invariant ranking model for information retrieval. InProceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval. 499–508

  77. [2022]

    InProceedings of the AAAI conference on artificial intelligence, Vol

    Bayesian optimization over permutation spaces. InProceedings of the AAAI conference on artificial intelligence, Vol. 36. 6515–6523

  78. [2023]

    Policy-gradient training of language models for ranking.arXiv preprint arXiv:2310.04407(2023)

    Pith/arXiv arXiv 2023