Document Similarity Enhanced IPS Estimation for Unbiased Learning to Rank

Graham McDonald; Iadh Ounis; Zeyan Liang

REVIEW 4 major objections 5 minor 34 references

Document Similarity Enhanced IPS Estimation for Unbiased Learning to Rank

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Similarity-aware propensity weights improve unbiased learning to rank, with significant NDCG gains over the Doubly Robust estimator in top-30 and top-50 settings.

desk verdict Useful incremental idea, but the headline gain is the best of many tuned variants and needs more careful significance testing before the claim is taken at face value. read the letter →

arxiv 2507.07909 v1 pith:FEKHKBBE submitted 2025-07-10 cs.IR

Zeyan Liang , Graham McDonald , Iadh Ounis This is my paper · ORCID

classification cs.IR

keywords unbiasedlearningtorankinversepropensityscoringpositionbiasdocumentsimilarityNDCGclickdebiasingcounterfactual

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Position-bias correction in learning to rank usually reweights clicks by the inverse of the probability that a document is seen at its rank. The paper argues that this ignores a useful signal: low-ranked documents that resemble the documents at the top of the historical ranking are likely relevant too. It therefore proposes IPSsim, which weights each click by a convex combination of the inverse propensity and the average similarity between the clicked document and the top-ranked documents. On two standard public LTR datasets with simulated clicks, the paper reports that IPSsim outperforms the existing IPS, RPS, Doubly Robust, and MIPS-style estimators, with the gains concentrated in top-30 and top-50 evaluation settings (for example, a statistically significant ~3% NDCG improvement over the Doubly Robust estimator at n=50). If the claim holds, similarity is an effective additional debiasing signal that requires no extra side information beyond the document features already present in LTR datasets.

What carries the argument

The load-bearing object is the IPSsim estimator (Eq. 2), which blends the standard inverse-propensity weight with an average-similarity term: each click on document i is weighted by (1-alpha)/rho_i + alpha*Avgsim(i,T), where Avgsim(i,T) is the mean pairwise similarity between i and the documents in T, the set of documents highly ranked in the historical ranking. The hyperparameter alpha is learned on a validation set in increments of 0.1, and three feature-based similarity measures (cosine, Euclidean, Manhattan) are tested, all normalized to the unit interval. The estimator also keeps the trust-bias correction term from the affine IPS formulation. This convex blend is what carries the argument: it gives low-ranked documents that resemble top-ranked documents a larger effective weight, on the hypothesis that such documents are likely relevant despite their low rank.

What would settle it

If the same NDCG improvements persist when Avgsim(i,T) is computed from randomly shuffled document features (so similarity is pure noise) while alpha is re-tuned, then the claimed benefit is not attributable to document similarity and the central hypothesis fails.

Watch

Extended reading notes

Core claim

The central claim is that incorporating document similarity into inverse-propensity estimation yields a more effective debiasing estimator for unbiased learning to rank. For each document i, IPSsim replaces the standard propensity weight 1/rho_i with (1-alpha)/rho_i + alpha*Avgsim(i,T), where T is the set of documents ranked at the top of the historical ranking and alpha in [0,1] is tuned on validation data. The paper reports that this estimator improves NDCG over Naive, IPS, RPS, DR, MIPSLTR, and DRM baselines on ISTELLA-S and MSLTR-WEB30K, particularly when n>=30 and when more training clicks are available. The intended mechanism is that low-ranked documents similar to top-ranked documents receive a boosted effective weight, counteracting the propensity penalty they incur for being ranked low.

Load-bearing premise

The load-bearing premise is that a convex blend of the inverse propensity and average similarity to top-ranked documents remains an effective debiasing weight, even though IPSsim is not an inverse propensity and the paper provides no bias or consistency proof for it.

Editorial extensions

If this is right

The paper reports that IPSsim achieves statistically significant NDCG gains over existing IPS estimators in top-n settings with n>=30, including a 2.88% improvement over the Doubly Robust estimator on ISTELLA-S.
The gains over baselines grow as the number of training clicks increases from 10^4 to 10^8.
The choice of similarity measure has little effect on IPSsim's performance, with the cosine variant often the best.
The method needs only the document feature vectors already present in LTR datasets, not external side information.
The authors propose adapting IPSsim to transformer-based ranking models and datasets with real user clicks as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because IPSsim is not an inverse propensity and has no unbiasedness proof, part of its empirical gain may come from variance reduction or regularization rather than from correctly recovering relevance; a variance decomposition across the five runs would separate these effects.
The same convex similarity correction could be applied to the imputation term of the Doubly Robust estimator or to RPS, not just to the propensity weight; the paper does not test these combinations.
Since alpha is tuned per dataset and top-n setting, the method's success depends on the validation set being representative; a sensitivity analysis across validation splits would show how fragile the gains are.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a modest, plausible extension of IPS for ULTR, but the headline claim overstates what the tables show. The idea of blending inverse propensity with average similarity to top-ranked documents is not in the prior work they cite, so there is a small novel kernel. The evaluation on ISTELLA-S and MSLTR-WEB30K follows a standard semi-synthetic setup, and the authors are transparent that they are testing a hypothesis rather than proving unbiasedness.

The main problem is the significance testing. Alpha is tuned per variant on the validation set, and the tables underline the best of six IPSsim configurations; the t-test then compares baselines against that underlined configuration without any correction for the selection. Many cells show IPSsim exactly matching the logging policy (e.g., all IPSsim variants at n=10 on MSLR-WEB30K), so the method frequently does nothing. Claiming that IPSsim 'is more effective than existing IPS estimators' based on a few selected n>=30 cells is an overstatement. The most honest reading is: in some top-n settings, the best tuned variant beats DR by a couple of NDCG points.

There are also missing experimental details: lambda in exp(-lambda*d) is never specified, and the target ranker architecture is not given. Both should be in the paper.

That said, this is not a bad paper. The authors don't oversell the theory; they explicitly say 'hypothesise.' The distribution analysis in Figures 1 and 2 is a nice attempt to explain dataset differences. The comparison with MIPS and DRM is reasonable.

For a short SIGIR paper, this could be worth referee time if the authors provide code, specify the missing hyperparameters, and redo the significance testing with a multiple-comparison correction or a pre-registered configuration. As it stands, the central claim should be read as conditional. I would not cite it as a definitive result, but it is a legitimate increment for ULTR researchers.

Referee Report

4 major / 5 minor

Summary. The paper proposes IPSsim, an extension of Inverse Propensity Scoring (IPS) for unbiased learning to rank. Instead of weighting each clicked document by the inverse propensity 1/rho_i alone, Eq. (2) uses a convex combination (1-alpha)/rho_i + alpha*Avgsim(i,T), where Avgsim is the average similarity of a document to the top-ranked documents in the historical ranking. The method is evaluated on the ISTELLA-S and MSLTR-WEB30K datasets under simulated clicks following Oosterhuis's setup, across top-n settings n=10,30,50 and click counts 10^4,10^6,10^8, comparing against Naive, IPS, RPS, DR, MIPSLTR, and DRM. The authors report that IPSsim outperforms existing IPS estimators particularly for n>=30, with a claimed statistically significant ~3% NDCG improvement over DR. The paper contains no theoretical bias or consistency analysis of the modified estimator; alpha and |T| are selected on validation data, and the reported headline result is the best among several tuned configurations.

Significance. If the central empirical claim were robust, the contribution would be a simple and practical way to inject document similarity into existing IPS-style debiasing for learning to rank, with improvements in top-heavy rankings on two standard public benchmarks. The paper has clear strengths: it follows a widely used semi-synthetic evaluation protocol, uses two large datasets, varies top-n and click volume, reports results for three similarity instantiations, and provides significance tests over five runs. The learned alpha values are reported for every configuration, which is transparent. However, the current evidence does not yet support the headline claim. The modified estimator is not an inverse propensity score and no unbiasedness argument is given; the reported gains are selected as the best of many validation-tuned configurations without multiple-comparison correction; and a large fraction of table cells show IPSsim exactly matching the logging policy, indicating the method often has no effect. These issues are load-bearing for the paper's central claim, so the manuscript needs substantial revision before the empirical conclusion can be accepted.

major comments (4)

[Section 3, Eq. (2)] The proposed weight (1-alpha)/rho_i + alpha*Avgsim(i,T) is not an inverse propensity score, and the paper provides no unbiasedness or consistency argument under the affine click model described in Section 4. For any alpha > 0, the second term introduces bias unless Avgsim(i,T) happens to be proportional to the true inverse propensity; the conditions under which this holds are neither stated nor tested. The authors' own Section 3 frames the connection as a 'hypothesis' rather than a derivation. Additionally, Eq. (1) defines T as documents highly ranked in H, not necessarily relevant, although the motivating hypothesis refers to 'highly ranked relevant documents'. Please provide a bias/variance decomposition for Eq. (2) under the click model, state the conditions on Avgsim under which the estimator is unbiased, or at minimum compare the empirical bias of IPSsim and IPS against known ground-truth propensities in a controlled experiment.
[Section 4, Tables 1 and 2] Alpha is learned on the validation set over a grid in increments of 0.1, |T| is selected from {5,10}, and three similarity measures are tested, giving 60 tuned configurations per dataset. The tables underline the best IPSsim variant for each setting, and the significance tests compare only that underlined configuration against the baselines, with no correction for multiple comparisons. The reported 2.88%/'~3%' improvement is therefore the maximum of many configurations, not a property of the estimator family. Please report the performance of all configurations (or a summary such as mean/median across configurations), describe the model-selection procedure, and account for the selection step in the significance testing, for example by using a separate model-selection holdout or a multiple-comparison correction.
[Tables 1 and 2] Many table cells show IPSsim exactly matching the logging policy (LP), e.g., all IPSsim variants at MSLTR-WEB30K n=10 with 10^4 clicks equal LP=0.4734, and numerous other cells in both tables are identical to LP. This indicates that in many settings the learned weights produce no effective debiasing, which directly conflicts with the abstract's claim that 'our IPSsim estimator is more effective than the existing IPS estimators'. The paper should quantify the consistency of the improvement, for example by reporting the number of settings (out of 18 per dataset) in which each IPSsim variant outperforms DR, and by discussing the settings where IPSsim has no effect.
[Abstract and Section 6] The headline example is internally inconsistent. The abstract states a ~3% NDCG improvement over DR when n=50, while Section 6 states a 2.88% improvement when n=30. Table 1 shows that for n=30 with 10^6 clicks, DR=0.8048 and the best IPSsim is 0.8280 (a 2.88% relative gain), whereas for n=50 with 10^6 clicks, DR=0.8135 and the best IPSsim is 0.8344 (about 2.57% relative gain). The authors should correct the abstract and Section 6 so that the claimed improvement matches the table, and should specify the exact comparison (dataset, click count, IPSsim variant) for every numerical claim.

minor comments (5)

[Section 3, Eq. (1)] There is a typographical fragment 'wheresim' after the equation; the sentence should be completed and the notation sim(i,i_m) defined before first use.
[Section 4] The click counts in Tables 1 and 2 are printed as '104', '106', and '108', which is easily misread as one hundred and four rather than 10^4; please use superscripts or explicit notation.
[Section 4, click model] The trust-bias correction term \hat{\epsilon}_{k_h(i)} in Eq. (2) is not defined precisely; the text says it is 'a trust bias correction' but does not give the formula or a precise reference to the version used.
[Section 5] The paper reports mean NDCG with significance triangles but no standard deviations or confidence intervals; adding these would help readers assess the stability of the reported gains over the five runs.
[Section 4, MIPSLTR/DRM] The description of MIPSLTR and DRM in Eqs. (3) is somewhat terse; in particular, the role of the sum over s documents and the exact definition of M(k|i_j) should be clarified for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IPSsim is an empirically evaluated new estimator; alpha tuning on validation is standard model selection and test-set evaluation is external.

full rationale

The paper's central claim is empirical: IPSsim improves NDCG over IPS, RPS, DR, MIPSLTR, and DRM baselines on held-out test splits of two public datasets. The derivation chain is: (1) a stated hypothesis that similar documents share relevance; (2) a new estimator, Eq. 2, formed by a convex combination of inverse propensity and average similarity; (3) evaluation with simulated clicks following Oosterhuis's setup. I checked each circularity pattern. There are no self-citations: none of the cited references are authored by Liang, McDonald, or Ounis, and no uniqueness theorem is imported from the authors' prior work. Eq. 2 does not reduce to Eq. 1 or to the NDCG target by construction; the target metric uses expert relevance labels and a separate test split. The alpha parameter is learned on the validation set (Section 4, 'IPSsim Parameters'), and the learned alpha values are reported in Tables 1 and 2, so the headline result is indeed a selected configuration. However, this is standard hyperparameter tuning rather than fitting the prediction target: the test evaluation is not forced by the tuning, and many IPSsim configurations underperform the baselines. The absence of a formal unbiasedness or consistency proof for the similarity-weighted estimator is a theoretical weakness and a correctness risk, but it is not circularity. The inconsistency between the abstract (n=50, approximately 3% improvement) and the conclusion (n=30, 2.88% improvement) is a reporting error, not a circular reduction. Overall, the paper is self-contained against external benchmarks, and I find no load-bearing circular step.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard simulated click-model assumptions inherited from Oosterhuis [22], and on the paper-specific hypothesis that feature-based similarity to top-ranked documents is a valid debiasing signal. Four free parameters (alpha, clipping threshold, |T|, and the unspecified lambda) are introduced, which restricts the generality of the empirical conclusions.

free parameters (4)

alpha = Values shown in parentheses in Tables 1 and 2, e.g., 0.9, 0.6, 0.1
Blending weight between the standard IPS term and the similarity term; selected from {0.1, ..., 0.9} on the validation set for each dataset, top-n setting, and click count.
clipping threshold tau = 10 / sqrt(number of clicks), e.g., 0.1 for 10^4 clicks
Heuristic used to cap large inverse propensity values; no theoretical derivation or sensitivity analysis is provided.
number of top documents |T| = 5 or 10
Two values are tested; the paper does not justify why these specific values are appropriate.
lambda in distance-to-similarity mapping = Not stated
The paper defines sim(x,y) = exp(-lambda d(x,y)) but never specifies the value of lambda, leaving the Euclidean and Manhattan similarity variants underspecified.

assumptions (4)

domain assumption The examination hypothesis: the probability of observing a document at position k is P(O=1|k) = (1 + (k-1)/5)^-2
Adopted from Oosterhuis [22] to simulate clicks; the entire evaluation depends on this position-bias model.
domain assumption The affine click model: users click relevant documents with probability 1 and non-relevant documents with probability 0.1 + 0.6/(1+k/20)
Used to generate simulated clicks, following Oosterhuis [22]; results are conditional on this click model.
domain assumption Documents that are similar tend to have similar relevance labels
Cited to MacAvaney et al. [18]; this motivates the IPSsim correction but is not demonstrated on the two datasets used.
ad hoc to paper Low-ranked documents similar to highly ranked documents are likely relevant, and this similarity can be added as a corrective weight inside IPS
The paper's hypothesis in Section 3; the implementation uses top-ranked documents without filtering for relevance, so the premise is not verified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Document Similarity Enhanced IPS Estimation for Unbiased Learning to Rank." pith.science (2026). https://pith.science/paper/FEKHKBBE

@misc{pith2026250707909,
  author       = {Pith},
  title        = {Pith review of: Document Similarity Enhanced IPS Estimation for Unbiased Learning to Rank},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FEKHKBBE}},
  note         = {Machine review of arXiv:2507.07909}
}

read the original abstract

Learning to Rank (LTR) models learn from historical user interactions, such as user clicks. However, there is an inherent bias in the clicks of users due to position bias, i.e., users are more likely to click highly-ranked documents than low-ranked documents. To address this bias when training LTR models, many approaches from the literature re-weight the users' click data using Inverse Propensity Scoring (IPS). IPS re-weights the user's clicks proportionately to the position in the historical ranking that a document was placed when it was clicked since low-ranked documents are less likely to be seen by a user. In this paper, we argue that low-ranked documents that are similar to highly-ranked relevant documents are also likely to be relevant. Moreover, accounting for the similarity of low-ranked documents to highly ranked relevant documents when calculating IPS can more effectively mitigate the effects of position bias. Therefore, we propose an extension to IPS, called IPSsim, that takes into consideration the similarity of documents when estimating IPS. We evaluate our IPSsim estimator using two large publicly available LTR datasets under a number of simulated user click settings, and with different numbers of training clicks. Our experiments show that our IPSsim estimator is more effective than the existing IPS estimators for learning an unbiased LTR model, particularly in top-n settings when n >= 30. For example, when n = 50, our IPSsim estimator achieves a statistically significant ~3% improvement (p < 0.05) in terms of NDCG compared to the Doubly Robust estimator from the literature.

Figures

Figures reproduced from arXiv: 2507.07909 by the authors.

**Figure 2.** Distribution of Queries by Document Count on [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 28 canonical work pages

[1]

Alekh Agarwal, Olivier Chapelle, Miroslav Dudík, and John Langford. 2014. A reliable effective terascale linear learning system.The Journal of Machine Learning Research 15, 1 (2014)

work page 2014
[2]

Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. 2019. A general framework for counterfactual learning-to-rank. In Proc. of SIGIR

work page 2019
[3]

Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Marc Najork
[4]

Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018. Unbi- ased learning to rank with unbiased propensity estimation. In Proc. of SIGIR

work page 2018
[5]

Qingyao Ai, Tao Yang, Huazheng Wang, and Jiaxin Mao. 2021. Unbiased learning to rank: online or offline? ACM Transactions on Information Systems (TOIS) 39, 2 (2021)

work page 2021
[6]

Rocío Cañamares and Pablo Castells. 2018. Should I follow the crowd? A proba- bilistic analysis of the effectiveness of popularity in recommender systems. In Proc. of SIGIR

work page 2018
[7]

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An experi- mental comparison of click position-bias models. In Proc. of WSDM

work page 2008
[8]

Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Transactions on Information Systems (TOIS) 35, 2 (2016)

work page 2016

Show all 34 references

[9]

Richard O Duda, Peter E Hart, and David G Stork. 1973. Pattern classification and scene analysis. Vol. 3. Wiley New York

1973
[10]

Shashank Gupta, Harrie Oosterhuis, and Maarten de Rijke. 2023. Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization. arXiv preprint arXiv:2305.01522 (2023)

2023 arXiv
[11]

Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20, 4 (2002)

2002
[12]

Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In SIGKDD

2002
[13]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay
[14]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS) 25, 2 (2007)

2007
[15]

Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In Proc. of WSDM

2017
[16]

1986.Taxicab geometry: An adventure in non-Euclidean geometry

Eugene F Krause. 1986.Taxicab geometry: An adventure in non-Euclidean geometry. Courier Corporation

1986
[17]

Tie-Yan Liu et al. 2009. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval 3, 3 (2009), 225–331

2009
[18]

Sean MacAvaney, Nicola Tonellotto, and Craig Macdonald. 2022. Adaptive re- ranking with a corpus graph. In Proc. of the 31st ACM International Conference on Information & Knowledge Management . 1491–1500

2022
[19]

Benjamin M Marlin and Richard S Zemel. 2009. Collaborative prediction and ranking with non-random missing data. In Proc. of RecSys

2009
[20]

Serge Moscovici and Claude Faucheux. 1972. Social influence, conformity bias, and the study of active minorities. In Advances in experimental social psychology . Vol. 6

1972
[21]

Harrie Oosterhuis. 2021. Computationally efficient optimization of plackett-luce ranking models for relevance and fairness. In Proc. of SIGIR

2021
[22]

Harrie Oosterhuis. 2023. Doubly Robust Estimation for Correcting Position Bias in Click Feedback for Unbiased Learning to Rank. ACM Transactions on Information Systems 41, 3 (2023)

2023
[23]

Harrie Oosterhuis and Maarten de Rijke. 2020. Policy-aware unbiased learning to rank for top-k rankings. In Proc. of SIGIR

2020
[24]

Mattia Prosperi, Yi Guo, Matt Sperrin, James S Koopman, Jae S Min, Xing He, Shannan Rich, Mo Wang, Iain E Buchan, and Jiang Bian. 2020. Causal inference and counterfactual prediction in machine learning for actionable healthcare. Nature Machine Intelligence 2, 7 (2020)

2020
[25]

Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 datasets. arXiv preprint arXiv:1306.2597 (2013)

2013 arXiv
[26]

Yuta Saito. 2020. Unbiased pairwise learning from biased implicit feedback. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval. 5–12

2020
[27]

Yuta Saito and Thorsten Joachims. 2022. Off-policy evaluation for large action spaces via embeddings. arXiv preprint arXiv:2202.06317 (2022)

2022 arXiv
[28]

Amit Singhal et al. 2001. Modern information retrieval: A brief overview. IEEE Data Eng. Bull. 24, 4 (2001), 35–43

2001
[29]

Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. 2020. When inverse propensity scoring does not work: Affine corrections for unbiased learning to rank. In Proc. of CIKM

2020
[30]

Nan Wang, Zhen Qin, Xuanhui Wang, and Hongning Wang. 2021. Non-clicks mean irrelevant? propensity ratio scoring as a correction. In Proc. of WSDM

2021
[31]

Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to rank with selection bias in personal search. In Proc. of SIGIR

2016
[32]

Yisong Yue, Rajan Patel, and Hein Roehrig. 2010. Beyond position bias: Examining result attractiveness as a source of presentation bias in clickthrough data. InProc. of WWW

2010
[2017]

Accurately interpreting clickthrough data as implicit feedback. In Proc. of SIGIR
[2019]

Addressing trust bias for unbiased learning-to-rank. In Proc. of WWW

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Alekh Agarwal, Olivier Chapelle, Miroslav Dudík, and John Langford. 2014. A reliable effective terascale linear learning system.The Journal of Machine Learning Research 15, 1 (2014)

work page 2014

[2] [2]

Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. 2019. A general framework for counterfactual learning-to-rank. In Proc. of SIGIR

work page 2019

[3] [3]

Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Marc Najork

[4] [4]

Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018. Unbi- ased learning to rank with unbiased propensity estimation. In Proc. of SIGIR

work page 2018

[5] [5]

Qingyao Ai, Tao Yang, Huazheng Wang, and Jiaxin Mao. 2021. Unbiased learning to rank: online or offline? ACM Transactions on Information Systems (TOIS) 39, 2 (2021)

work page 2021

[6] [6]

Rocío Cañamares and Pablo Castells. 2018. Should I follow the crowd? A proba- bilistic analysis of the effectiveness of popularity in recommender systems. In Proc. of SIGIR

work page 2018

[7] [7]

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An experi- mental comparison of click position-bias models. In Proc. of WSDM

work page 2008

[8] [8]

Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Transactions on Information Systems (TOIS) 35, 2 (2016)

work page 2016