Pith. sign in

REVIEW 3 major objections 5 minor 68 references

An Epistemic Position-Based Click Model: From Interactions to Epistemic Distributions of Relevance and Bias

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces the first contextual epistemic click model, replacing point estimates of position bias and relevance with Beta distributions, and reports higher click log-likelihood than a traditional pointwise model.

desk verdict The optimization technology is real; the epistemic-uncertainty headline is not yet earned. read the letter →

arxiv 2607.18712 v1 pith:P2IHOFTO submitted 2026-07-21 cs.IR

Oscar Rolando Ramirez Milian , Harrie Oosterhuis This is my paper
classification cs.IR
keywords clickmodelingposition-basedmodelepistemicuncertaintyevidentialdeeplearningBetadistributionspositionbiastorankquantification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that click modeling does not have to settle for point estimates: in the position-based click model, each position-bias and relevance factor can be represented by a Beta distribution whose spread expresses epistemic uncertainty about the true parameters. The authors show how to train this epistemic model with evidential deep learning by making the Monte Carlo likelihood gradient numerically stable through self-normalization and conditioning on position bias. In semi-synthetic experiments with a known ground-truth PBM, the resulting epistemic model reaches higher test-set log-likelihood than the standard pointwise PBM, converges much faster, and places its distributions around the true parameter values. The payoff, if true, is that search and recommendation systems would know when their estimates of relevance and position bias are uncertain.

What carries the argument

The load-bearing object is the epistemic PBM itself: click probability remains P(C=1|q,d,k)=theta_k*zeta_qd, but theta_k and zeta_qd are now Beta-distributed random variables, with a lookup table for theta_k and a neural network predicting alpha and beta for each query-item pair from features. Training maximizes the marginal likelihood under independent Beta priors; because naive Monte Carlo gradients have impractical variance and underflow, the method uses log-sum-exp for stable likelihood estimation, conditions on position bias to marginalize relevance and lower variance, and self-normalizes gradients with a softmax over log-likelihood samples. Entropy regularization with a per-parameter a

What would settle it

In the paper's semi-synthetic setting, run a coverage check: for many query-item pairs, build 90% credible intervals from the predicted Beta distributions and test whether roughly 90% of ground-truth relevance and position-bias values fall inside them. Consistent under-coverage, or intervals whose width does not grow with prediction error, would show the distributions do not capture epistemic uncertainty even if likelihood is high.

Watch

Extended reading notes

Core claim

The central claim is that an epistemic position-based click model — where theta_k and zeta_qd are independent Beta-distributed random variables rather than single numbers — can be optimized to maximize the marginal likelihood of observed clicks, and that the resulting distributions are informative about uncertainty. The paper's own summary is that, with self-normalization and conditioning, the epistemic PBM appropriately captures uncertainty in its position-bias predictions, and that epistemic click models can provide higher log-likelihood than traditional pointwise click models. The improvement in likelihood is credited to neutral initialization: starting from an almost-uniform distribution

Load-bearing premise

The load-bearing premise is that the spread of the learned Beta distributions is a genuine measure of epistemic uncertainty; the paper supports this mainly by visual inspection of a few position-bias plots and does not calibrate the width to actual estimation error.

Editorial extensions

If this is right

  • Self-normalization and conditioning on position bias are not optional decorations: without them the epistemic model fails to learn, while with them it outperforms a pointwise PBM on test log-likelihood.
  • Epistemic click models can match or exceed pointwise predictive accuracy, so modeling uncertainty does not force a trade-off against accuracy.
  • The learned distributions can be propagated to downstream ranker evaluation, giving an indication of error under distribution shift instead of a single point prediction.
  • Moderate entropy regularization lowers confidence in relevance predictions without harming likelihood, suggesting confidence can be tuned toward conservatism; extreme regularization produces uninformative distributions.
  • This is the first contextual epistemic click model, so the same evidential approach becomes available for other click models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the epistemic-uncertainty claim could be tested objectively in the semi-synthetic setup by computing frequentist coverage of Beta credible intervals around true theta_k and zeta_qd; the paper does not report such a test.
  • Editorial inference: the reported likelihood advantage over the pointwise PBM may depend on initialization; a systematic comparison across pointwise initializations or against ensembles of pointwise models would separate the benefit of uncertainty modeling from the benefit of starting neutral.
  • Editorial inference: because position bias is conditioned out analytically in the estimator, the relevance distributions may absorb misspecification in the position-bias model; conditioning on relevance instead could reveal whether the uncertainty attribution between theta and zeta is stable.
  • Editorial inference: the Beta distributions over position bias could be used directly in inverse-propensity-scoring, yielding not only debiased relevance estimates but also variance estimates for those estimates, which would be a natural next application.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an evidential deep-learning variant of the position-based click model (PBM). Instead of point estimates, the model outputs a Beta distribution for each position-bias parameter θ_k and each query-item relevance parameter ζ_{q,d}, with the independent Beta distributions representing epistemic uncertainty. The optimization objective is the marginal likelihood of the observed clicks. Because the naive Monte Carlo gradient estimator suffers from numerical underflow and high variance, the authors introduce three techniques: log-sum-exp stabilization, conditioning on position-bias samples to reduce variance, and self-normalized gradient estimation. Experiments on semi-synthetic MSLR-Web-10K and Istella-S data compare these techniques against a pointwise PBM and against REINFORCE-style baselines. The self-normalization plus conditioning method achieves higher test log-likelihood than the pointwise PBM, and the learned position-bias distributions appear centered near the true values, while naive estimation fails. The paper also studies entropy regularization and a downstream ranker-evaluation task.

Significance. The paper addresses a genuine gap: existing click models provide only point estimates and no uncertainty quantification. If the epistemic distributions are shown to be meaningful, this would be the first contextual epistemic click model and a useful foundation for uncertainty-aware learning-to-rank. The optimization contributions are solid: the conditioning derivation (Eqs. 28–31) is correct, the variance inequality (Eq. 29) holds, and the experiments convincingly show that the naive estimator fails while the proposed self-normalized and conditioned estimator learns. The semi-synthetic setup, with known ground-truth parameters, provides a golden opportunity to validate uncertainty estimates, but the paper does not currently exploit it. The log-likelihood improvement over the pointwise PBM is interesting, though its interpretation is complicated by differences in initialization.

major comments (3)
  1. [§7.3 and §8] The paper's central claim is that the learned Beta distributions 'appropriately capture uncertainty.' However, Section 7.3 explicitly says this conclusion is reached 'as far as can be judged from visual inspection,' and Section 8 concedes 'there is no objective evaluation of epistemic distributions; thus, whilst we did identify some serious failures of baselines, we are limited in assessing the correctness of the shapes of our epistemic distributions.' Because the experimental setup is semi-synthetic and the true θ_k and ζ_{q,d} are known, the authors can and should directly test whether the predictive distributions are calibrated. For example, they could report empirical coverage of credible intervals for θ_k (and for ζ_{q,d} aggregated by relevance label), or use a proper scoring rule for the predictive distribution of clicks. Without such evidence, the widths of the Beta distributions
  2. [§4.2 and §5.4] Section 4.2 proves that, in the absence of regularization, the optimal Beta distributions can concentrate into Dirac deltas (Eqs. 12–15). The only mechanisms preventing this collapse are entropy regularization with a hand-chosen weight λ (Eq. 40) and early stopping. Yet λ is not selected by any uncertainty-calibration criterion; the experiments merely test a few values, and the modified entropy in Eq. (39) weights position and relevance parameters by 1/K and 1/|D| without further justification. Since the widths of the predicted distributions are precisely the epistemic-uncertainty output, this leaves the central output dependent on a free parameter that is not validated. The authors should either provide a principled way to choose λ, demonstrate that the uncertainty estimates are robust across a reasonable range of λ, or validate the resulting widths against known ground truth.
  3. [§7.2] The claim that 'epistemic click models can provide higher log-likelihood than traditional pointwise click models' is based on a comparison where the pointwise PBM is initialized with default sigmoid logits (predictions near 0.5), while the epistemic PBM starts with ν≈1, i.e., nearly uniform Beta distributions. The authors themselves speculate that the improvement comes from this initialization difference, but no experiment tests a pointwise PBM initialized to the same neutral starting point or otherwise equalizes initialization. As presented, the log-likelihood gain may reflect the initialization scheme rather than the epistemic formulation. Since this result is highlighted in the paper, the comparison should be made fair (e.g., by trying multiple initializations for the pointwise PBM) or the claim should be appropriately qualified.
minor comments (5)
  1. [Abstract] 'anepistemicalternative' and later 'anepistemicclick model' are missing spaces; also 'anepistemic' appears in the abstract and conclusion. Please fix typographical spacing.
  2. [§4.1] 'exuberates variance-related problems' should be 'exacerbates.'
  3. [§6] The experimental setup says 'We perform several experiments' but the tense is inconsistent: 'we perform' followed by 'we apply.' Please standardize.
  4. [Fig. 3 caption] The caption includes 'Predicted k' which appears to be a truncated phrase; it should likely be 'Predicted θ_k' or similar.
  5. [§5.4] Equation (39) introduces a modified entropy H̄′ but the relationship to the standard entropy H in Eq. (16) is not fully discussed. A brief clarification of why the 1/|D| and 1/K scaling is used would help.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: epistemic-uncertainty validation gap is acknowledged, but derivation is self-contained and benchmarked externally.

full rationale

The paper's derivation chain is self-contained. The epistemic objective is defined in Eq. (6) as an expectation of the PBM likelihood under independent Beta distributions; the gradient estimators in Eqs. (10)-(11), conditioning in Eqs. (28)-(31), self-normalization in Eqs. (34)-(37), and entropy regularization in Eqs. (39)-(40) are all derived from this objective. The log-likelihood comparison between epistemic and pointwise PBM (Sec. 7.2, Fig. 2) is an external test-set benchmark, not a function of the model's own fitted values. I checked the self-citations ([30], [41]-[45], [57]): they support background, notation, and prior work on unbiased LTR and Beta-confidence modeling, but the load-bearing evidential-learning premise is attributed to external work [5,37,38,40,53], and no 'uniqueness theorem' from the authors is invoked to force the Beta choice. The one real weakness is stated by the authors in the Conclusion: 'there is no objective evaluation of epistemic distributions; thus, whilst we did identify some serious failures of baselines, we are limited in assessing the correctness of the shapes of our epistemic distributions.' Section 4.2 itself proves that the unregularized optimum collapses to Dirac measures, so the non-degenerate widths are produced by the hand-chosen λ in Eq. (40) plus early stopping; this means the epistemic-uncertainty claim is not strongly validated. However, an unvalidated or hyperparameter-dependent claim is a correctness/verification gap, not circularity: the predicted distributions are not a renamed version of an input, and no fitted parameter is called a prediction. Visual inspection (Fig. 3) is weak evidence, but it is not a circular reduction.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

Everything beyond the standard PBM and Beta math is either a hand-chosen hyperparameter (λ, S, initialization, batch sizes) or a borrowed heuristic (Meinert et al.'s proxy claim). The paper's own Section 4.2 shows the unregularized objective would collapse to point estimates, so the epistemic widths are entirely created by the regularization choice — no new entity with independent falsifiable evidence is introduced.

free parameters (4)
  • Entropy regularization weight λ = 0, 5e-5, 1e-4, 5e-4, 1e-3, 5e-3, 1e-2, 2.5e-1 (explored)
    Controls the width of the epistemic distributions — the very quantity the paper claims to measure. No principled selection rule or calibration check ties λ to a target coverage level; Sec 7.4 shows λ=0.25 destroys likelihood while λ=1e-3 keeps it.
  • MC sample count S / S_pos / S_rel = 60
    Number of samples for gradient estimation; affects variance and bias of the log-likelihood estimate. Chosen by hand (Sec 6).
  • Initialization scheme (ν≈1, logits≈0) and early stopping = ν starts near 1
    The paper attributes the epistemic model's advantage over the pointwise baseline to this initialization/degenerate-distribution start (Sec 7.2, Conclusion). The stopping point also fixes where the regularized objective stops, hence the final width.
  • Batch sizes and architecture = 12k vs 35k pairs per batch; 3×64/64/32
    Different batches for epistemic vs pointwise models confound the convergence comparison; architecture is a design choice (Sec 6).
assumptions (7)
  • domain assumption PBM factorization: P(click)=θ_k·ζ_{q,d}
    Eq. (1); the whole likelihood (3) is built on it. If clicks depend on more than position and item (e.g., context, fatigue), the fitted distributions are mis-specified.
  • domain assumption All clicks are independent Bernoulli given θ and ζ
    Eq. (3) binomial; needed for the product-form likelihood over q,d,k in Eq. (9) and for the factorization in Eq. (28).
  • domain assumption θ_k and ζ_{q,d} are independent a priori (factorized Beta prior)
    Eqs. (4)-(5). The conditioning estimator (Sec 5.2) and the product-form ∏ J_{q,d} rely on independence; any correlation between position bias and relevance would break the variance-reduction derivation.
  • standard math Beta distributions with diverging parameters converge weakly to Dirac deltas; Portmanteau theorem applies
    Sec 4.2, Eqs. (13)-(15); used to prove the optimizer collapses to point masses without regularization.
  • ad hoc to paper Entropy-regularized evidential learning is a useful heuristic proxy for epistemic uncertainty (Meinert et al.)
    Sec 4.2/5.4. This borrowed heuristic is the load-bearing justification that the output widths mean anything; the paper provides no independent verification. It is precisely the premise the authors admit cannot be objectively evaluated.
  • domain assumption Semi-synthetic ground truth: θ_k=1/k, ζ=0.9(y/4)+0.1 faithfully represents PBM behavior
    Sec 6 click generation. All claims about capturing ground-truth relevance/bias rely on this simulated equivalence; no real-user click data is used.
  • domain assumption Logging policy (Plackett-Luce on 30 queries) induces a realistic distribution shift for evaluation
    Sec 6 and Sec 7.5 downstream evaluation. If the logging policy's exposure distribution were unrealistic, the measured generalization and the downstream histograms would be unrepresentative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Epistemic Position-Based Click Model: From Interactions to Epistemic Distributions of Relevance and Bias." pith.science (2026). https://pith.science/paper/P2IHOFTO

@misc{pith2026260718712,
  author       = {Pith},
  title        = {Pith review of: An Epistemic Position-Based Click Model: From Interactions to Epistemic Distributions of Relevance and Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P2IHOFTO}},
  note         = {Machine review of arXiv:2607.18712}
}
read the original abstract

User interactions with rankings are affected by both items' relevances and display positions. Accordingly, click probabilities are often modeled as a product of relevance and position factors; and for improving recommendation and search, one needs to disentangle relevance from position bias. However, existing click models only provide frequentist point-estimates that do not capture any measure of epistemic uncertainty. Consequently, there is no indication of how much confidence one should have in their predictions. In this work, we introduce the first evidential deep-learning approach to form an epistemic alternative to the important position-based click model. Our learned model takes as input item and position features and outputs a beta-distribution for every relevance and position-bias variable of the position-based model. These distributions capture epistemic uncertainty about click probabilities and the underlying effects of attraction and position bias. The main challenge of our approach is its optimization for which we propose approximation and conditioning techniques to provide numerical stability and variance reduction. Our experiments indicate that our approach captures epistemic uncertainty in predictions on previously-unseen data, whereas standard policy gradients fail to learn meaningful distributions. We believe our contribution of the first contextual epistemic click model constitutes an important step in incorporating Bayesian uncertainty into click modeling.

Figures

Figures reproduced from arXiv: 2607.18712 by the authors.

Figure 1
Figure 1. Schematic comparison of the two deep learning instantiations of a position-based click model. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Log-likelihood of test-set clicks as predicted by pointwise and epistemic [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Predicted distributions by epistemic PBMs optimized with different methods and pointwise PBM predictions. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Statistics of the relevance distributions of the epistemic [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The effect of regularization on various statistics. X-axis: training epoch; Y-Axis: values of the indicated metrics. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Position bias distributions obtained with entropy regularization applied with weight [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Pointwise and epistemic predictions of the expected [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 2 linked inside Pith

  1. [32]

    Chao Liu, Fan Guo, and Christos Faloutsos. 2009. Bbm: bayesian browsing model from petabyte-scale data. InProceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 537–546

    2009

  2. [1]

    Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Marc Najork

  3. [2]

    Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018. Unbi- ased learning to rank with unbiased propensity estimation. InThe 41st interna- tional ACM SIGIR conference on research & development in information retrieval. 385–394

    2018

  4. [3]

    Qingyao Ai, Tao Yang, Huazheng Wang, and Jiaxin Mao. 2021. Unbiased learning to rank: online or offline?ACM Transactions on Information Systems (TOIS)39, 2 (2021), 1–29

    2021

  5. [4]

    Réka Albert, Hawoong Jeong, and Albert-László Barabási. 1999. Diameter of the world-wide web.nature401, 6749 (1999), 130–131

    1999

  6. [5]

    Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. 2020. Deep evidential regression.Advances in neural information processing systems33 (2020), 14927–14937

    2020

  7. [6]

    Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. 2022. Pitfalls of epis- temic uncertainty quantification through loss minimisation.Advances in Neural Information Processing Systems35 (2022), 29205–29216

    2022

  8. [7]

    2013.Convergence of probability measures

    Patrick Billingsley. 2013.Convergence of probability measures. John Wiley & Sons

    2013

Show all 68 references
  1. [8]

    Pierre Blanchard, Desmond J Higham, and Nicholas J Higham. 2021. Accurately computing the log-sum-exp and softmax functions.IMA J. Numer. Anal.41, 4 (2021), 2311–2330

    2021

  2. [9]

    Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016. A Neu- ral Click Model for Web Search. InProceedings of the 25th International Conference on World Wide Web(Montréal, Québec, Canada)(WWW ’16). International World Wide Web Conferences Steering Committee...

    2016

  3. [10]

    Alexey Borisov, Martijn Wardenaar, Ilya Markov, and Maarten de Rijke. 2018. A Click Sequence Model for Web Search. InThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval(Ann Arbor, MI, USA)(SIGIR ’18). Association for Computing Machine...

    2018

  4. [11]

    Gabriel Cardoso, Sergey Samsonov, Achille Thin, Eric Moulines, and Jimmy Olsson. 2022. BR-SNIS: Bias Reduced Self-Normalized Importance Sampling. InAdvances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol....

    2022

  5. [12]

    Ye Chen and Tak W Yan. 2012. Position-normalized click prediction in search advertising. InProceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. 795–803

    2012

  6. [13]

    Flavio Chierichetti, Ravi Kumar, and Prabhakar Raghavan. 2011. Optimizing two-dimensional search results presentation. InProceedings of the Fourth ACM International Conference on Web Search and Data Mining(Hong Kong, China) (WSDM ’11). Association for Computing Machinery, New ...

    2011

  7. [14]

    2022.Click models for web search

    Aleksandr Chuklin, Ilya Markov, and Maarten De Rijke. 2022.Click models for web search. Springer Nature

    2022

  8. [15]

    Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An ex- perimental comparison of click position-bias models. InProceedings of the 2008 international conference on web search and data mining. 87–94

    2008

  9. [16]

    Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016. Fast Ranking with Additive Ensembles of Oblivious and Non-Oblivious Regression Trees.ACM Transactions on Information Systems35, 2, Article...

    2016

  10. [17]

    Dupret and Benjamin Piwowarski

    Georges E. Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations.. InProceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval(Singapore, Singapore)(...

    2008

  11. [18]

    Ruocheng Guo, Xiaoting Zhao, Adam Henderson, Liangjie Hong, and Huan Liu. 2020. Debiasing Grid-based Product Search in E-commerce. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining(Virtual Event, CA, USA)(KDD ’20). Association f...

    2020

  12. [19]

    Shashank Gupta, Philipp Hager, Jin Huang, Ali Vardasbi, and Harrie Oosterhuis

  13. [20]

    Philipp Hager, Maarten de Rijke, and Onno Zoeter. 2023. Contrasting Neural Click Models and Pointwise IPS Rankers. InAdvances in Information Retrieval, Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, and Ann...

    2023

  14. [21]

    Philipp Hager, Onno Zoeter, and Maarten de Rijke. 2025. Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank. InProceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR)(...

    2025

  15. [22]

    2013.Monte Carlo methods

    John Hammersley. 2013.Monte Carlo methods. Springer Science & Business Media

    2013

  16. [23]

    Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric and epistemic uncer- tainty in machine learning: An introduction to concepts and methods.Machine learning110, 3 (2021), 457–506

    2021

  17. [24]

    Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques.ACM Transactions on Information Systems (TOIS)20, 4 (2002), 422–446

    2002

  18. [25]

    Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. InProceedings of the tenth ACM interna- tional conference on web search and data mining. 781–789

    2017

  19. [26]

    Mira Juergens, Nis Meinert, Viktor Bengs, Eyke Hüllermeier, and Willem Waege- man. 2024. Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learnin...

    2024

  20. [27]

    Jingwei Kang, Maarten de Rijke, Santiago de Leon-Martinez, and Harrie Ooster- huis. 2025. Rethinking Click Models in Light of Carousel Interfaces: Theory-Based Categorization and Design of Click Models. InProceedings of the 2025 Interna- tional ACM SIGIR Conference on Innovati...

    2025

  21. [28]

    Armen Der Kiureghian and Ove Ditlevsen. 2009. Aleatory or epistemic? Does it matter?Structural Safety31, 2 (2009), 105–112. doi:10.1016/j.strusafe.2008.06.020 Risk Acceptance and Risk Communication

    2009 doi

  22. [29]

    Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. 2022. Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model. InProceedings of the Fifteenth ACM International Conference on Web Search and D...

    2022

  23. [30]

    Norman Knyazev and Harrie Oosterhuis. 2023. A Lightweight Method for Model- ing Confidence in Recommendations with Learned Beta Distributions. InProceed- ings of the 17th ACM Conference on Recommender Systems(Singapore, Singapore) (RecSys ’23). Association for Computing Machin...

    2023

  24. [31]

    Wouter Kool, Herke van Hoof, and Max Welling. 2019. Buy 4 REINFORCE samples, get a baseline for free! (2019)

    2019

  25. [33]

    Jianping Liu, Yingfei Wang, Jian Wang, Meng Wang, and Xintao Chu. 2024. Probabilistic graph model and neural network perspective of click models for web search.Knowledge and Information Systems66, 10 (2024), 5829–5873. doi:10. 1007/s10115-024-02145-z

    2024

  26. [34]

    Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval.Foundations and Trends®in Information Retrieval3, 3 (2009), 225–331. doi:10.1561/1500000016

    2009 doi

  27. [35]

    2001.Propositional, Probabilistic and Evidential Reasoning: Integrating numerical and symbolic approaches

    Weiru Liu. 2001.Propositional, Probabilistic and Evidential Reasoning: Integrating numerical and symbolic approaches. Vol. 77. Springer Science & Business Media

    2001

  28. [36]

    Linyuan Lü, Matúš Medo, Chi Ho Yeung, Yi-Cheng Zhang, Zi-Ke Zhang, and Tao Zhou. 2012. Recommender systems.Physics Reports519, 1 (2012), 1–49. doi:10.1016/j.physrep.2012.02.006 Recommender Systems

    2012 doi

  29. [37]

    Andrey Malinin, Sergey Chervontsev, Ivan Provilkov, and Mark Gales. 2020. Regression prior networks.arXiv preprint arXiv:2006.11590(2020)

    2020 arXiv

  30. [38]

    Andrey Malinin and Mark Gales. 2018. Predictive Uncertainty Estimation via Prior Networks. InAdvances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://pr...

    2018

  31. [39]

    Andrey Malinin and Mark Gales. 2019. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness.Advances in neural information processing systems32 (2019)

    2019

  32. [40]

    Nis Meinert, Jakob Gawlikowski, and Alexander Lavin. 2023. The unreasonable effectiveness of deep evidential regression. InProceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intellig...

    2023 doi

  33. [41]

    2020.Learning from User Interactions with Rankings: A Uni- fication of the Field.Ph

    Harrie Oosterhuis. 2020.Learning from User Interactions with Rankings: A Uni- fication of the Field.Ph. D. Dissertation. Informatics Institute, University of Amsterdam. https://harrieo.github.io//publication/2021-phd-thesis

    2020

  34. [42]

    Harrie Oosterhuis. 2021. Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness. InProceedings of the 44th Interna- tional ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada)(SIGIR ’21). A...

    2021

  35. [43]

    Harrie Oosterhuis. 2022. Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank. InProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval(Madrid, Spain) (ICTIR ’22). Association for Computing Mac...

    2022

  36. [44]

    Harrie Oosterhuis. 2023. Doubly robust estimation for correcting position bias in click feedback for unbiased learning to rank.ACM Transactions on Information Systems41, 3 (2023), 1–33

    2023

  37. [45]

    Harrie Oosterhuis, Rolf Jagerman, and Maarten de Rijke. 2020. Unbiased learning to rank: counterfactual and online approaches. InCompanion Proceedings of the Web Conference 2020. 299–300

    2020

  38. [46]

    Zohreh Ovaisi, Ragib Ahsan, Yifan Zhang, Kathryn Vasilaky, and Elena Zheleva

  39. [47]

    Deep Shankar Pandey and Qi Yu. 2023. Evidential conditional neural processes. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 9389–9397

    2023

  40. [48]

    Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets.CoRR abs/1306.2597 (2013). http://arxiv.org/abs/1306.2597

    2013 arXiv

  41. [49]

    Behnam Rahdari, Peter Brusilovsky, and Branislav Kveton. 2024. Towards Simulation-Based Evaluation of Recommender Systems with Carousel Inter- faces.ACM Trans. Recomm. Syst.2, 1, Article 9 (March 2024), 25 pages. doi:10.1145/3643709

    2024 doi

  42. [50]

    Behnam Rahdari, Branislav Kveton, and Peter Brusilovsky. 2022. The Magic of Carousels: Single vs. Multi-List Recommender Systems. InProceedings of the 33rd ACM Conference on Hypertext and Social Media(Barcelona, Spain)(HT ’22). Association for Computing Machinery, New York, NY...

    2022

  43. [51]

    Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting clicks: estimating the click-through rate for new ads. InProceedings of the 16th International Conference on World Wide Web(Banff, Alberta, Canada)(WWW ’07). Association for Computing Machinery, New York, N...

    2007

  44. [52]

    Mark Sanderson, Monica Lestari Paramita, Paul Clough, and Evangelos Kanoulas

  45. [53]

    Murat Sensoy, Lance Kaplan, and Melih Kandemir. 2018. Evidential deep learning to quantify classification uncertainty.Advances in neural information processing systems31 (2018)

    2018

  46. [54]

    Ava P Soleimany, Alexander Amini, Samuel Goldman, Daniela Rus, Sangeeta N Bhatia, and Connor W Coley. 2021. Evidential deep learning for guided molecular property prediction and discovery.ACS central science7, 8 (2021), 1356–1367

    2021

  47. [55]

    Adith Swaminathan and Thorsten Joachims. 2015. The Self-Normalized Estimator for Counterfactual Learning. InAdvances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. https://proceedin...

    2015

  48. [56]

    Ali Vardasbi, Maarten de Rijke, and Ilya Markov. 2020. Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank. InProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval(Virtual Event, China)(SIGIR...

    2020

  49. [57]

    Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. 2020. When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank. InProceedings of the 29th ACM International Conference on Information & Knowledge Management(Virtual Event, Ireland)(CI...

    2020 doi

  50. [58]

    Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to Rank with Selection Bias in Personal Search. InProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 115–124

    2016

  51. [59]

    Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position bias estimation for unbiased learning to rank in personal search. InProceedings of the eleventh ACM international conference on web search and data mining. 610–618

    2018

  52. [60]

    Ronald J Williams. 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.Machine Learning8, 3-4 (1992), 229–256

    1992

  53. [61]

    Xiaohui Xie, Jiaxin Mao, Maarten de Rijke, Ruizhe Zhang, Min Zhang, and Shaop- ing Ma. 2018. Constructing an Interaction Behavior Model for Web Image Search. InThe 41st International ACM SIGIR Conference on Research & Development in In- formation Retrieval(Ann Arbor, MI, USA)(...

    2018

  54. [62]

    Le Yan, Zhen Qin, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2022. Revisiting Two-tower Models for Unbiased Learning to Rank. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval(Madrid, Spain)...

    2022 doi

  55. [63]

    Jose, and Ke Zhou

    Hai-Tao Yu, Adam Jatowt, Roi Blanco, Joemon M. Jose, and Ke Zhou. 2019. A Rank-biased Neural Network Model for Click Modeling. InProceedings of the 2019 Conference on Human Information Interaction and Retrieval(Glasgow, Scotland UK)(CHIIR ’19). Association for Computing Machin...

    2019

  56. [64]

    Honglei Zhuang, Zhen Qin, Xuanhui Wang, Michael Bendersky, Xinyu Qian, Po Hu, and Dan Chary Chen. 2021. Cross-Positional Attention for Debiasing Clicks. InProceedings of the Web Conference 2021(Ljubljana, Slovenia)(WWW ’21). Association for Computing Machinery, New York, NY, U...

    2021

  57. [2010]

    InProceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

    Do user preferences and evaluation measures line up?. InProceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. 555–562

  58. [2019]

    InThe World Wide Web Conference

    Addressing trust bias for unbiased learning-to-rank. InThe World Wide Web Conference. 4–14

  59. [2020]

    InProceedings of The Web Conference 2020

    Correcting for selection bias in learning-to-rank systems. InProceedings of The Web Conference 2020. 1863–1873

    2020

  60. [2024]

    InProceedings of the 17th ACM International Conference on Web Search and Data Mining(Merida, Mexico)(WSDM ’24)

    Unbiased Learning to Rank: On Recent Advances and Practical Applications. InProceedings of the 17th ACM International Conference on Web Search and Data Mining(Merida, Mexico)(WSDM ’24). Association for Computing Machinery, New York, NY, USA, 1118–1121. doi:10.1145/3616855.3636451

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.