REVIEW 4 major objections 5 minor 19 references
LLMs for estimating positional bias in logged interaction data
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read LLM relevance judgments, used as a conditioning variable in click logs, can estimate position-bias propensities—including row- and column-specific effects in a grid layout—without online experiments.
desk verdict A practical, clean idea for offline propensity estimation with an LLM judge, but the validation doesn't nail the core assumption and the headline gain is under-reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The estimator is Equation (7): the propensity at position p is the average, over LLM score buckets s, of the observed click probability at p divided by the observed click probability at the top position for that same bucket. Conditioning on the LLM score is meant to cancel relevance, leaving only the examination term from the Position-Based Model, and the top position anchors the scale by assumption.
What would settle it
Run a randomized swap experiment on the same Viator grid and compare its propensity matrix with the LLM-derived one; disagreement beyond sampling error in any cell would falsify the estimator. A faster test uses synthetic logs with known examination probabilities and an LLM score only partially correlated with relevance to see whether Equation (7) recovers the injected propensities.
Extended reading notes
Core claim
The paper introduces a method for estimating examination propensities entirely from logged interactions: use zero-shot LLM relevance judgments as a conditioning signal, and estimate P(examination | position) as the ratio P(click | s, p) / P(click | s, position=1), averaged over LLM score values. On 8,439 logged Viator search queries judged by GPT-4o-mini, the propensity curves from three score buckets overlap, supporting the conditional-independence assumption, and the recovered propensities expose row- and column-dependent grid effects that one-dimensional position models miss. An IPS-weighted reranker trained with these propensities matches the production model on NDCG@10 and improves weig
Load-bearing premise
The estimate stands or falls on Assumption 2: once the LLM's score is known, the logged position carries no additional information about true relevance; if the LLM misses relevance signals that correlate with position—such as price, rating, or visual appeal—the recovered propensities are biased.
Editorial extensions
If this is right
- Position-bias estimates can be obtained from existing logs, reducing the need for randomized result-swap experiments.
- Comparing propensity curves across LLM score buckets doubles as a diagnostic: under the conditional-independence assumption, the curves should coincide.
- IPS-weighted rerankers can be trained offline with these propensities, matching production effectiveness on NDCG@10 while improving rank-weighted metrics (wNDCG@10 by about 2 percent).
- For grid and other multi-dimensional interfaces, propensities are row- and column-dependent; one-dimensional position curves are insufficient.
- Because LLM outputs are slow and not used in serving, LLM scores have limited causal influence on logged positions—making them better instruments than lexical scores such as BM25.
Reading between the lines
- The method is an instrumental-variable estimate in disguise: the LLM score is the instrument, and Assumption 2 is an exclusion restriction. A natural stress test is to shift the logging policy and check that the recovered propensities are invariant; movement would signal that some relevance information leaks through position.
- The same score-conditioned ratio could be adapted to other feedback signals—purchase, add-to-cart—and to multi-step examination models, letting the grid row-column structure be learned rather than assumed.
- Stability across buckets only validates the assumption within the information the LLM already sees; signals the LLM systematically misses, such as price, ratings, or imagery, remain a threat. Testing with deliberately incomplete judges would quantify how much residual bias survives.
- If the offline gains replicate in live traffic, LLM-judged propensity estimation could become a standard pre-processing step for training on logged marketplace search data.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an offline method for estimating position-based examination propensities from logged click data using an LLM-as-a-judge relevance score as a conditioning variable. Under the Position-Based Model and two assumptions (top position always examined; relevance conditionally independent of position given the LLM score), the propensity at position p is estimated as the ratio of click rates at p and at position 1 within LLM score strata, averaged over scores (Eqs. (6)-(7)). The method is applied to 8,439 logged queries from Viator's grid search layout. The authors report that propensity curves are stable across LLM score buckets (RQ1), that the recovered propensities reveal row- and column-dependent effects (RQ2), and that an IPS-weighted reranker trained with these propensities matches production NDCG@10 while improving wNDCG@10 by about 2%.
Significance. The core derivation is clean: given PBM and the two stated assumptions, Eq. (7) directly identifies the examination propensity up to the top-slot anchor. The idea of using a zero-shot LLM judge as an independent relevance signal for propensity estimation is novel and potentially valuable, since it avoids expensive online randomized interventions. However, the empirical validation is currently insufficient to support the central claim. The RQ1 bucket-overlap diagnostic does not actually test Assumption 2, no uncertainty quantification is provided for the propensity curves, and the IPS reranker result is reported without experimental details. If the method is made reliable and properly validated, it would be a useful contribution for complex layouts such as grids, but in its present form the key assumption remains unverified.
major comments (4)
- [Section 3, RQ1 (Fig. 2, Eq. (6))] The claim that overlapping bucket curves 'provide empirical support for the conditional-independence assumption' is not supported. Eq. (6) is invariant to the choice of s not only when Eq. (3) holds, but also when the residual position effect has the multiplicative form P(relevant|s,p)=a(s)b(p), which violates Eq. (3). The test therefore cannot detect a broad class of violations. Since the production ranker uses features (price, rating, text, visuals) that GPT-4o-mini may not fully capture, items at higher positions may be more relevant within every LLM score bucket. The paper itself concedes in Section 2 that Eq. (3) is credible only if the LLM score conditions on essentially all relevance-related information, but no evidence is provided. A direct test using logged bookings or human labels stratified by position and score, or an online randomized comparison, is needed.
- [Section 3, Figures 2 and 3] No confidence intervals or error bars are given for the propensity estimates. With 8,439 queries, the ratio estimator in Eq. (7) within score buckets and individual grid cells may have substantial variance. The statements that the curves are 'virtually superimposed' and that row 2, column 3 to 4 shows a 'roughly 10%' increase are quantitative claims that require bootstrap intervals or similar uncertainty quantification. Without them, RQ1 and RQ2 cannot be assessed reliably.
- [Section 4 and Abstract] The IPS reranker experiment is reported in a single sentence: 'IPS-weighted reranker trained with these propensities performed on par with the production model on NDCG@10 but showed better results on weighted NDCG(+2%).' No details are given about the reranker architecture, loss function, how the learned propensities were incorporated, propensity clipping, train/validation splits, number of runs, or statistical significance. This is load-bearing for the claim that the estimated propensities improve ranking. Either a full experimental section with uncertainty quantification is needed, or the claim should be clearly labelled as preliminary and not used as evidence for the method's validity.
- [Section 3, RQ1] Scores below 41 are excluded post hoc with the explanation 'too sparse to analyse.' This removes the bucket where residual position dependence might be strongest: items ranked highly by the production model despite receiving a low LLM score. The paper should report the bucket sizes and, if possible, include a lower-score bucket or a sparsity-aware analysis, since the current exclusion weakens the diagnostic value of RQ1.
minor comments (5)
- [Section 2] The exact LLM prompt, model configuration, temperature, and decoding settings are not provided. As the method is 'LLM-as-a-judge,' the prompt is a key component; without it, the experiments cannot be reproduced.
- [Figure 3] The figure lacks a clear legend/color scale and the axes are not fully labeled. The grid orientation (which dimension is row vs. column) should be stated explicitly in the caption.
- [Equation (7)] The notation S is used both for the set of all possible LLM scores and for subsets of scores (buckets). This is confusing; use a different symbol for the score set, e.g., S_all and S_bucket.
- [Assumption 1] The paper cites eye-tracking evidence that the first result is examined in about 90% of sessions, not 100%. The method estimates propensities relative to the top slot. Please clarify why this relative scaling is sufficient for the IPS use case, or discuss the effect of Assumption 1 being only approximately true.
- [References] Reference [9] is incomplete and formatted inconsistently. Please polish the reference list.
Circularity Check
No significant circularity; the derivation is self-contained and no fitted parameter is renamed as a prediction.
full rationale
The paper's derivation chain is not circular. The propensity estimator in Eq. (6)-(7) is a direct ratio of observed click probabilities conditional on an externally supplied LLM relevance score, under the explicitly stated Position-Based Model and Assumptions 1-2. The LLM-as-a-judge model is external and untrained on the click logs, so it is not a fitted parameter that is later called a prediction. The propensities are not defined in terms of the downstream IPS evaluation; they are estimated from click ratios and then used to reweight a reranker, with the caveat that results are offline and pending live experiments. The only notable weakness is that RQ1's bucket-overlap test is a limited diagnostic for Assumption 2: a multiplicative omitted-variable effect would also produce superimposed curves. That is a statistical validity concern, not circularity, because the estimator does not reduce to its own output by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of known results. The paper is self-contained against external benchmarks (e.g., BM25 and production model comparisons), and the central assumptions are openly stated rather than smuggled in.
Assumptions & free parameters
free parameters (1)
- LLM score bucket thresholds =
41-60, 61-80, 81-100
assumptions (3)
- domain assumption Position-Based Model: P(click | user, item, position) = P(relevant | item, user) * P(examination | position)
- domain assumption Assumption 1: P(examination | position = 1) = 1.0
- ad hoc to paper Assumption 2: P(relevant | position, LLM score) = P(relevant | LLM score)
Cite this review
Pith. "Pith review of LLMs for estimating positional bias in logged interaction data." pith.science (2026). https://pith.science/paper/4QKB2MEQ
@misc{pith2026250903696,
author = {Pith},
title = {Pith review of: LLMs for estimating positional bias in logged interaction data},
year = {2026},
howpublished = {\url{https://pith.science/paper/4QKB2MEQ}},
note = {Machine review of arXiv:2509.03696}
}
read the original abstract
Recommender and search systems commonly rely on Learning To Rank models trained on logged user interactions to order items by predicted relevance. However, such interaction data is often subject to position bias, as users are more likely to click on items that appear higher in the ranking, regardless of their actual relevance. As a result, newly trained models may inherit and reinforce the biases of prior ranking models rather than genuinely improving relevance. A standard approach to mitigate position bias is Inverse Propensity Scoring (IPS), where the model's loss is weighted by the inverse of a propensity function, an estimate of the probability that an item at a given position is examined. However, accurate propensity estimation is challenging, especially in interfaces with complex non-linear layouts. In this paper, we propose a novel method for estimating position bias using Large Language Models (LLMs) applied to logged user interaction data. This approach offers a cost-effective alternative to online experimentation. Our experiments show that propensities estimated with our LLM-as-a-judge approach are stable across score buckets and reveal the row-column effects of Viator's grid layout that simpler heuristics overlook. An IPS-weighted reranker trained with these propensities matches the production model on standard NDCG@10 while improving weighted NDCG@10 by roughly 2%. We will verify these offline gains in forthcoming live-traffic experiments.
Figures
Reference graph
Works this paper leans on
-
[1]
Negar Arabzadeh and Charles LA Clarke. 2025. Benchmarking LLM-based rel- evance judgment methods. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval . 3194–3204
work page 2025
-
[2]
Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th international conference on World wide web. 1–10
work page 2009
-
[3]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198
2016
-
[4]
Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An ex- perimental comparison of click position-bias models. In Proceedings of the 2008 international conference on web search and data mining . 87–94
work page 2008
-
[5]
Fan Guo, Chao Liu, and Yi Min Wang. 2009. Efficient multiple-click models in web search. In Proceedings of the second acm international conference on web search and data mining . 124–131
work page 2009
-
[6]
Kasra Hosseini, Thomas Kober, Josip Krapac, Roland Vollgraf, Weiwei Cheng, and Ana Peleteiro Ramallo. 2025. Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation. In European Conference on Information Retrieval . Springer, 149–163
work page 2025
-
[7]
Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Re- search and Development in Information Retrieval (Salvador, Brazil) (SIGIR ’05). Association for Computing Machinery, New York, NY, USA, ...
-
[8]
Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In Proceedings of the tenth ACM interna- tional conference on web search and data mining . 781–789
work page 2017
Show all 19 references
-
[9]
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al
-
[10]
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085 (2019)
2019 arXiv
-
[11]
Zohreh Ovaisi, Ragib Ahsan, Yifan Zhang, Kathryn Vasilaky, and Elena Zheleva
-
[12]
Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. 2007. In Google we trust: Users’ decisions on rank, position, and relevance. Journal of computer-mediated communication 12, 3 (2007), 801–823
2007
-
[13]
Stephen E Robertson. 1977. The probability ranking principle in IR. Journal of documentation 33, 4 (1977), 294–304
1977
-
[14]
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at TREC-3. (1995)
1995
-
[15]
Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation 28, 1 (1972), 11–21
1972
-
[16]
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? investigat- ing large language models as re-ranking agents. arXiv preprint arXiv:2304.09542 (2023)
2023 arXiv
-
[17]
Xingzhu Wang, Erhan Zhang, Yiqun Chen, Jinghan Xuan, Yucheng Hou, Yitong Xu, Ying Nie, Shuaiqiang Wang, Dawei Yin, and Jiaxin Mao. 2025. Leveraging LLMs to Evaluate Usefulness of Document.arXiv preprint arXiv:2506.08626 (2025)
2025 arXiv
-
[2020]
In Proceedings of The Web Conference 2020
Correcting for selection bias in learning-to-rank systems. In Proceedings of The Web Conference 2020. 1863–1873
2020
-
[2025]
URL https://arxiv
From generation to judgment: Opportunities and challenges of llm-as-a- judge, 2025. URL https://arxiv. org/abs/2411.16594 (2025)
2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.