{"id":"15c84be7-23d0-4d20-a3cc-9dd8559b832d","arxiv_id":"2607.18712","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A position-based click model that outputs Beta distributions over relevance and position-bias parameters, trained with self-normalizing and position-conditioned estimators, outperforms a pointwise PBM at predicting simulated clicks.","lead":"A search-engine click model that normally gives single best guesses for relevance and position bias now returns a full probability distribution over each value, showing how much the model doesn't know. The authors add stability tricks for training these 'epistemic' models, and in simulations the result predicts clicks better than a standard single-value model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Epistemic uncertainty claim is unvalidated: the Beta distribution widths are produced by a hand-chosen entropy regularizer and never tested for calibration/coverage against known ground truth, so the paper's central claim that they 'appropriately capture uncertainty' is unsupported.","rationale":"The reader's weakest_assumption correctly identifies the central vulnerability: the spread of the learned Beta distributions is asserted to represent epistemic uncertainty based on a heuristic entropy regularizer, with no objective evaluation. The paper explicitly acknowledges this gap in the conclusion. My analysis confirms this is the single most load-bearing concern: the claimed contribution is an 'epistemic' click model, and the entire epistemic content lies in the widths of the Beta distributions. The paper's own Section 4.2 shows that without regularization, optimization collapses to Dirac deltas, so the non-degenerate widths are almost entirely a product of the hand-chosen λ and early-stopping/training trajectory. The semi-synthetic experiments provide ground truth, but the evaluation never checks whether the predicted credible intervals actually contain the true parameters at the expected frequencies. This is an addressable issue: a coverage/calibration audit would settle it. The optimization techniques (self-normalization and conditioning) are well-motivated and appear to work as demonstrated, so I do not dispute the engineering contributions or the log-likelihood improvements. But those results do not substantiate the epistemic-uncertainty claim without a validation of distributional calibration. Hence the verdict remains CONDITIONAL: the paper should either add such a test or soften the uncertainty claim. I agree with the reader's weakest assumption and see no need to escalate to REJECT because the concern is fixable and the authors are transparent about the limitation.","tokens_in":22205,"tokens_out":7331,"duration_ms":71486,"concrete_test":"In the paper's semi-synthetic setup, for the self-norm+cond model with λ=0 and λ=0.001, compute the empirical frequentist coverage of the 80% and 95% credible intervals of the predicted Beta distributions: (a) for each position bias θ_k, and (b) for a random sample of 1000 held-out query-item pairs ζ_{q,d}, using the known ground-truth values. Repeat over the 15 runs. Compare coverage to nominal levels and also to a bootstrap baseline over the training interactions. If coverage deviates substantially from nominal (e.g., <60% for the 80% interval) or varies strongly with λ, the distribution widths do not represent epistemic uncertainty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the predicted Beta distributions capture epistemic uncertainty (Sec. 7.3, Conclusion). However, Section 4.2 proves that without regularization the optimal distributions collapse to Dirac deltas (Eqs. 12-15). The only mechanism producing non-degenerate distributions is the entropy regularizer with a hand-chosen weight λ (Eqs. 16-18, 39-40). The paper itself concedes in the Conclusion: 'there is no objective evaluation of epistemic distributions; thus, whilst we did identify some serious failures of baselines, we are limited in assessing the correctness of the shapes of our epistemic distributions.' The experiments only visually inspect position-bias distributions (Fig. 3) and report average/s.s.d. of relevance distribution parameters (Fig. 4); no calibration, coverage, or proper-scoring test is performed. In this semi-synthetic setting, the true θ_k and ζ_{q,d} are known, so such a test is straightforward. If the widths are not calibrated to the actual uncertainty, the epistemic-uncertainty claim fails even though the optimization method works and the point predictions (product of means) are accurate. This is load-bearing because the novelty over pointwise PBMs rests on the meaningfulness of these distributions, not on the log-likelihood gain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an evidential deep-learning variant of the position-based click model (PBM). Instead of point estimates, the model outputs a Beta distribution for each position-bias parameter θ_k and each query-item relevance parameter ζ_{q,d}, with the independent Beta distributions representing epistemic uncertainty. The optimization objective is the marginal likelihood of the observed clicks. Because the naive Monte Carlo gradient estimator suffers from numerical underflow and high variance, the authors introduce three techniques: log-sum-exp stabilization, conditioning on position-bias samples to reduce variance, and self-normalized gradient estimation. Experiments on semi-synthetic MSLR-Web-10K and Istella-S data compare these techniques against a pointwise PBM and against REINFORCE-style baselines. The self-normalization plus conditioning method achieves higher test log-likelihood than the pointwise PBM, and the learned position-bias distributions appear centered near the true values, while naive estimation fails. The paper also studies entropy regularization and a downstream ranker-evaluation task.","tokens_in":22504,"tokens_out":4785,"duration_ms":43046,"significance":"The paper addresses a genuine gap: existing click models provide only point estimates and no uncertainty quantification. If the epistemic distributions are shown to be meaningful, this would be the first contextual epistemic click model and a useful foundation for uncertainty-aware learning-to-rank. The optimization contributions are solid: the conditioning derivation (Eqs. 28–31) is correct, the variance inequality (Eq. 29) holds, and the experiments convincingly show that the naive estimator fails while the proposed self-normalized and conditioned estimator learns. The semi-synthetic setup, with known ground-truth parameters, provides a golden opportunity to validate uncertainty estimates, but the paper does not currently exploit it. The log-likelihood improvement over the pointwise PBM is interesting, though its interpretation is complicated by differences in initialization.","major_comments":[{"comment":"The paper's central claim is that the learned Beta distributions 'appropriately capture uncertainty.' However, Section 7.3 explicitly says this conclusion is reached 'as far as can be judged from visual inspection,' and Section 8 concedes 'there is no objective evaluation of epistemic distributions; thus, whilst we did identify some serious failures of baselines, we are limited in assessing the correctness of the shapes of our epistemic distributions.' Because the experimental setup is semi-synthetic and the true θ_k and ζ_{q,d} are known, the authors can and should directly test whether the predictive distributions are calibrated. For example, they could report empirical coverage of credible intervals for θ_k (and for ζ_{q,d} aggregated by relevance label), or use a proper scoring rule for the predictive distribution of clicks. Without such evidence, the widths of the Beta distributions","section":"§7.3 and §8"},{"comment":"Section 4.2 proves that, in the absence of regularization, the optimal Beta distributions can concentrate into Dirac deltas (Eqs. 12–15). The only mechanisms preventing this collapse are entropy regularization with a hand-chosen weight λ (Eq. 40) and early stopping. Yet λ is not selected by any uncertainty-calibration criterion; the experiments merely test a few values, and the modified entropy in Eq. (39) weights position and relevance parameters by 1/K and 1/|D| without further justification. Since the widths of the predicted distributions are precisely the epistemic-uncertainty output, this leaves the central output dependent on a free parameter that is not validated. The authors should either provide a principled way to choose λ, demonstrate that the uncertainty estimates are robust across a reasonable range of λ, or validate the resulting widths against known ground truth.","section":"§4.2 and §5.4"},{"comment":"The claim that 'epistemic click models can provide higher log-likelihood than traditional pointwise click models' is based on a comparison where the pointwise PBM is initialized with default sigmoid logits (predictions near 0.5), while the epistemic PBM starts with ν≈1, i.e., nearly uniform Beta distributions. The authors themselves speculate that the improvement comes from this initialization difference, but no experiment tests a pointwise PBM initialized to the same neutral starting point or otherwise equalizes initialization. As presented, the log-likelihood gain may reflect the initialization scheme rather than the epistemic formulation. Since this result is highlighted in the paper, the comparison should be made fair (e.g., by trying multiple initializations for the pointwise PBM) or the claim should be appropriately qualified.","section":"§7.2"}],"minor_comments":[{"comment":"'anepistemicalternative' and later 'anepistemicclick model' are missing spaces; also 'anepistemic' appears in the abstract and conclusion. Please fix typographical spacing.","section":"Abstract"},{"comment":"'exuberates variance-related problems' should be 'exacerbates.'","section":"§4.1"},{"comment":"The experimental setup says 'We perform several experiments' but the tense is inconsistent: 'we perform' followed by 'we apply.' Please standardize.","section":"§6"},{"comment":"The caption includes 'Predicted k' which appears to be a truncated phrase; it should likely be 'Predicted θ_k' or similar.","section":"Fig. 3 caption"},{"comment":"Equation (39) introduces a modified entropy H̄′ but the relationship to the standard entropy H in Eq. (16) is not fully discussed. A brief clarification of why the 1/|D| and 1/K scaling is used would help.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: the optimization machinery is the real contribution; the epistemic-uncertainty headline is not yet supported. If you read it for the estimator, you'll find a genuinely useful trick. If you read it for the claimed uncertainty quantification, you'll be disappointed.\n\nWhat's new and good: the paper proposes a contextual PBM with Beta-distributed relevance and position-bias parameters, and solves the marginal-likelihood optimization by conditioning on position-bias samples and self-normalizing the importance weights. The derivation in Secs. 5.2–5.3 looks correct; the variance inequality (29) is standard but the estimator is applied sensibly. The experiments show a clear win: the naive REINFORCE estimator barely learns, while the conditioned self-normalized version consistently reaches higher log-likelihood on both datasets. That is a solid empirical result, and the authors are honest about the weaknesses of evidential deep learning (Sec. 4.2) and about the lack of an objective evaluation of the distributions (Conclusion). I trust the math and the main empirical claim about optimization.\n\nThe soft spots, in proportion. The biggest one is the central claim: the width of the Beta distributions is supposed to capture epistemic uncertainty, but the only thing preventing collapse to Dirac deltas is entropy regularization with a hand-picked λ. The paper itself says there is no objective evaluation of the epistemic distributions. In a semi-synthetic setup where the true θ and ζ are known, a calibration or coverage test is easy, and its absence leaves the headline claim resting on visual inspection of Figure 3. That is not nothing, but it is not enough for the abstract's statement that the model 'captures epistemic uncertainty'. Second, the experimental setup samples training interactions from the test-set queries too, so the model has already seen clicks for those queries during training; the 'previously-unseen' claim is overstated. Third, the comparison with the pointwise PBM is confounded by different batch sizes and initialization; the pointwise baseline hasn't converged in 1500 epochs, so 'higher log-likelihood than pointwise' is suggestive, not conclusive. Minor: the 'first epistemic click model' framing should be contrasted more carefully with BBM [32], which is Bayesian albeit non-contextual.\n\nWho is this for: anyone working on click models or unbiased LTR who wants a way to get distributions over position bias and relevance for risk-aware ranking. The estimator alone is worth a referee's time. My recommendation: send to peer review, but request calibration/coverage experiments and a fair baseline comparison before acceptance.","headline":"The optimization technology is real; the epistemic-uncertainty headline is not yet earned.","tokens_in":23010,"tokens_out":4625,"would_cite":true,"duration_ms":39929,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the first contextual epistemic click model, replacing point estimates of position bias and relevance with Beta distributions, and reports higher click log-likelihood than a traditional pointwise model.","keywords":["click modeling","position-based click model","epistemic uncertainty","evidential deep learning","Beta distributions","position bias","learning to rank","uncertainty quantification"],"falsifier":"In the paper's semi-synthetic setting, run a coverage check: for many query-item pairs, build 90% credible intervals from the predicted Beta distributions and test whether roughly 90% of ground-truth relevance and position-bias values fall inside them. Consistent under-coverage, or intervals whose width does not grow with prediction error, would show the distributions do not capture epistemic uncertainty even if likelihood is high.","tokens_in":22028,"feed_emoji":"🖱️","tokens_out":5168,"duration_ms":68028,"temperature":0.7,"pith_summary":"This paper claims that click modeling does not have to settle for point estimates: in the position-based click model, each position-bias and relevance factor can be represented by a Beta distribution whose spread expresses epistemic uncertainty about the true parameters. The authors show how to train this epistemic model with evidential deep learning by making the Monte Carlo likelihood gradient numerically stable through self-normalization and conditioning on position bias. In semi-synthetic experiments with a known ground-truth PBM, the resulting epistemic model reaches higher test-set log-likelihood than the standard pointwise PBM, converges much faster, and places its distributions around the true parameter values. The payoff, if true, is that search and recommendation systems would know when their estimates of relevance and position bias are uncertain.","feed_headline":"Epistemic click model beats pointwise PBM","feed_subtitle":"First model to output confidence in relevance and position bias while improving test log-likelihood.","key_machinery":"The load-bearing object is the epistemic PBM itself: click probability remains P(C=1|q,d,k)=theta_k*zeta_qd, but theta_k and zeta_qd are now Beta-distributed random variables, with a lookup table for theta_k and a neural network predicting alpha and beta for each query-item pair from features. Training maximizes the marginal likelihood under independent Beta priors; because naive Monte Carlo gradients have impractical variance and underflow, the method uses log-sum-exp for stable likelihood estimation, conditions on position bias to marginalize relevance and lower variance, and self-normalizes gradients with a softmax over log-likelihood samples. Entropy regularization with a per-parameter a","core_discovery":"The central claim is that an epistemic position-based click model — where theta_k and zeta_qd are independent Beta-distributed random variables rather than single numbers — can be optimized to maximize the marginal likelihood of observed clicks, and that the resulting distributions are informative about uncertainty. The paper's own summary is that, with self-normalization and conditioning, the epistemic PBM appropriately captures uncertainty in its position-bias predictions, and that epistemic click models can provide higher log-likelihood than traditional pointwise click models. The improvement in likelihood is credited to neutral initialization: starting from an almost-uniform distribution","pith_inferences":["Editorial inference: the epistemic-uncertainty claim could be tested objectively in the semi-synthetic setup by computing frequentist coverage of Beta credible intervals around true theta_k and zeta_qd; the paper does not report such a test.","Editorial inference: the reported likelihood advantage over the pointwise PBM may depend on initialization; a systematic comparison across pointwise initializations or against ensembles of pointwise models would separate the benefit of uncertainty modeling from the benefit of starting neutral.","Editorial inference: because position bias is conditioned out analytically in the estimator, the relevance distributions may absorb misspecification in the position-bias model; conditioning on relevance instead could reveal whether the uncertainty attribution between theta and zeta is stable.","Editorial inference: the Beta distributions over position bias could be used directly in inverse-propensity-scoring, yielding not only debiased relevance estimates but also variance estimates for those estimates, which would be a natural next application."],"forward_implications":["Self-normalization and conditioning on position bias are not optional decorations: without them the epistemic model fails to learn, while with them it outperforms a pointwise PBM on test log-likelihood.","Epistemic click models can match or exceed pointwise predictive accuracy, so modeling uncertainty does not force a trade-off against accuracy.","The learned distributions can be propagated to downstream ranker evaluation, giving an indication of error under distribution shift instead of a single point prediction.","Moderate entropy regularization lowers confidence in relevance predictions without harming likelihood, suggesting confidence can be tuned toward conservatism; extreme regularization produces uninformative distributions.","This is the first contextual epistemic click model, so the same evidential approach becomes available for other click models."],"fun_headline_variants":["Click model that outputs confidence in relevance and bias","Epistemic approach captures uncertainty in click predictions","First click model to give uncertainty estimates over pointwise","Self-normalized optimization improves epistemic click model","Beta distributions reveal relevance and position bias confidence"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the spread of the learned Beta distributions is a genuine measure of epistemic uncertainty; the paper supports this mainly by visual inspection of a few position-bias plots and does not calibrate the width to actual estimation error.","fun_headline_variants_meta":{"raw":{"variants":["Click model that outputs confidence in relevance and bias","Epistemic approach captures uncertainty in click predictions","First click model to give uncertainty estimates over pointwise","Self-normalized optimization improves epistemic click model","Beta distributions reveal relevance and position bias confidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1691,"prompt_tokens":716,"completion_tokens":975,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":905}},"tokens_in":460,"tokens_out":975,"duration_ms":10088,"temperature":1.0,"reasoning_tokens":905,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:34:58.900318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In the paper's semi-synthetic setting, run a coverage check: for many query-item pairs, build 90% credible intervals from the predicted Beta distributions and test whether roughly 90% of ground-truth relevance and position-bias values fall inside them. Consistent under-coverage, or intervals whose width does not grow with prediction error, would show the distributions do not capture epistemic uncertainty even if likelihood is high.","supporting_citations":[],"review_version":1}