REVIEW 3 major objections 3 minor 31 references
Correcting the LogQ Correction: Revisiting Sampled Softmax for Large-Scale Retrieval
T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The standard logQ correction of sampled softmax still carries a finite-sample bias because it treats the deterministic positive item as a sampled negative; the paper's refined loss removes that bias and reports consistent retrieval gains…
desk verdict Sound derivation, modest and inconsistent empirical gains; the 'consistent improvements' claim in the abstract outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the gradient decomposition identity that separates the deterministic positive from the sampled negatives, combined with weighted (self-normalized) importance sampling. In the identity $\nabla_\theta L_{\mathrm{softmax}}(u,p) = (1-P_\theta(p|u))[-\nabla_\theta f(u,p) + \mathbb{E}_{d\sim P_\theta(\cdot|u,d\neq p)}\nabla_\theta f(u,d)]$, the factor $(1-P_\theta(p|u))$ becomes the per-example sample weight $w_{up}$, and the expectation is Monte Carlo-estimated from negatives drawn from $Q'(d)$, the proposal distribution with the positive item excluded. The same negatives are reused to estimate $P_\theta(p|u)$ by importance sampling, $\hat P(p|u)=\frac{e^{f(u,p)}}{e^{f(u,p)} + \frac{1}{n}\sum_{i=1}^{n} e^{f(u,d_i)-\log Q'(d_i)}}$, so the whole correction is a rescaling of the sampled-softmax loss by a model-confidence weight, with the positive item removed from the normalizer.
What would settle it
On MovieLens-1M, with a fixed seed and $n=256$ negatives per example, compute the angle or L2 distance between the exact full-softmax gradient and each sampled gradient (uncorrected, standard logQ, and refined loss) over a held-out batch; the refined gradient should be the closest on average, and the gap between the two corrections should shrink as $n$ grows. If the standard logQ gradient is already closer to the true gradient, or if the reported Recall@20 gains disappear when $w_{up}$ is replaced by its expectation under the true model, the paper's bias-removal mechanism is not what produced the improvements.
Extended reading notes
Core claim
The paper's central claim is that the logQ correction [29], derived from weighted importance sampling, applies the sampling correction to the one term that is not sampled: the positive item in the loss denominator appears with probability 1, not with probability $Q(p)$. Decomposing the full softmax gradient as $\nabla_\theta L_{\mathrm{softmax}}(u,p) = (1-P_\theta(p|u))\left[-\nabla_\theta f(u,p) + \mathbb{E}_{d\sim P_\theta(\cdot|u,d\neq p)}[\nabla_\theta f(u,d)]\right]$, the paper replaces the conditional expectation by weighted importance sampling over a proposal $Q'$ supported on $\mathcal{D}\setminus\{p\}$, giving the loss $L_{\mathrm{ours}}(u,p)=-\operatorname{sg}(1-P_\theta(p|u))\log\frac{e^{f(u,p)}}{\sum_{i} e^{f(u,d_i)-\log Q'(d_i)}}$. The positive item no longer appears in the denominator, and the stop-gradient weight $w_{up}=1-P_\theta(p|u)$ rescales the entire example: easy positives, already well separated from the sampled negatives, contribute little, while uncertain positives are amplified. The paper further shows the standard logQ gradient equals $(1-v_{up})\nabla L_{\mathrm{original}}$ and that both corrections converge in probability to the full softmax gradient as $n\to\infty$, so the difference is a finite-sample bias that the new loss is designed to remove. Empirically, the refined loss reports consistent gains over standard logQ on public benchmarks and in a large-scale industrial retrieval system.
Load-bearing premise
The load-bearing premise is that the estimated misclassification probability $1 - \hat P(p|u)$, computed from the same 256 sampled negatives used in the loss, is accurate enough that rescaling every training example by it does not inject more noise than it removes.
Editorial extensions
If this is right
- With in-batch negatives, the exact proposal $Q'$ has the closed form $Q'(d) = \frac{N}{N - \#p} Q(d)$, so the refined correction is computable exactly on any dataset for which item frequencies are known.
- Because the positive item is removed from the denominator, the popularity over-penalization is corrected on the sampled negatives only; the paper attributes the reported recall gains at large $k$ in the production setup to this removal.
- The interpretable weight $w_{up} = 1 - \hat P(p|u)$ automatically down-weights easy positives and up-weights hard ones during training, so the loss adapts per example as the model becomes more confident.
- The method adds negligible computational overhead and requires no change to model architecture or data pipeline, making it a drop-in replacement for standard logQ correction in existing retrieval pipelines.
- Both the standard and refined losses converge to the full-softmax gradient as the number of negatives grows, so the practical value of the refinement is largest at the small sample sizes (for example, $n=256$) used in real systems.
Reading between the lines
- An obvious extension the paper does not pursue: the deterministic-positive oversight is present in any self-normalized sampled-softmax loss, including the output layers of language models trained with sampled softmax, so the refined weighting could transfer there.
- Because the paper reuses the same $n$ negatives to estimate $P(p|u)$, the weight $1 - \hat P(p|u)$ is itself a ratio of positive score to average negative score; on rare items this ratio is high-variance, and one testable extension would estimate the weight from a larger separate negative sample or a calibrated baseline.
- A direct corollary of the convergence argument is that the advantage of the refined loss over standard logQ should shrink as $n$ grows; a reader testing at very large negative counts should expect the two losses to merge, and at very small $n$ the gap should be largest.
- The public-dataset evidence reported here covers MovieLens-1M and Steam only; the authors note they could not reproduce the Gowalla baseline with the public implementation and set that dataset aside, so the claim of consistent public gains should be read as applying to the two datasets actually benchmarked.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper revisits the logQ correction used in sampled-softmax training for two-tower retrieval models. It observes that in standard practice the positive item is included in the softmax denominator deterministically while negatives are Monte Carlo sampled, and it argues that this mismatch introduces bias. The proposed loss, L_ours, removes the positive from the denominator and rescales the sampled-softmax gradient by a stop-gradient weight w_up = sg(1 - P_hat(p|u)), where P_hat is obtained by self-normalized importance sampling from the same set of negatives. Appendix B shows that the resulting gradient estimate is consistent with the full-softmax gradient as the number of negatives grows. The paper evaluates the method on MovieLens-1M and Steam under leave-one-out and temporal splits, and on a proprietary industrial dataset, comparing improved logQ against standard logQ.
Significance. If the claimed improvement were established, the paper would provide a principled, low-cost modification to a widely used industrial technique, with an interpretable per-example weight. The derivation in Section 3.3 and Appendix B is self-contained and correct as a consistency argument, and the authors make their code publicly available. However, the central claim of consistent improvements is not supported by the reported tables, and the paper does not provide a finite-sample bias/variance analysis that would justify the new estimator over standard logQ. The work is therefore potentially useful but currently overclaims its empirical and theoretical contributions.
major comments (3)
- [Abstract; Section 4.3; Table 1] The claim of 'consistent improvements' over standard logQ is contradicted by the paper's own Table 1. Improved logQ is worse than standard logQ in 4 of the 16 academic comparisons, including Steam leave-one-out in-batch NDCG@20 (0.0719 vs 0.0727) and R@20 (0.1689 vs 0.1698), and MovieLens temporal-split mixed NDCG@20 (0.1281 vs 0.1292) and R@20 (0.2792 vs 0.2800). In Table 2, improved logQ is also worse than standard at Recall@10 for both Consumption and Engagement. No error bars or significance tests are reported, so these differences may be within noise. The authors should report per-seed variability or paired tests and should revise the abstract and conclusion to describe the observed pattern, e.g., comparable or better on most settings with gains concentrated at higher recall values.
- [Section 3.3; Appendix B] The derivation establishes only consistency as n goes to infinity, not that the finite-sample bias of L_ours is smaller than that of standard logQ. Appendix A shows that standard logQ is also consistent for the same target gradient, so the paper's claimed advantage must rest on a finite-sample property that is never proved or measured. Moreover, the weight w_up is computed from the same n=256 negatives used in the loss, making w_up a self-normalized importance-sampling estimate with O(1/n) bias and variance that can be large when the proposal Q' is heavy-tailed relative to the model distribution. Because w_up multiplies the entire loss, noise in this estimate directly rescales the gradient; for a rare positive, a single high-weight negative can push P_hat(p|u) toward 0 and w_up toward 1, eliminating the intended downweighting. The paper needs an explicit finite-sample bias/variance comparison or an ablation that isolates the effect of the weight.
- [Section 4.2; Table 2] In the industrial evaluation, improved logQ is worse than standard logQ at Recall@10 for both Consumption (0.0700 vs 0.0756) and Engagement (0.0279 vs 0.0304), and the claimed advantage appears only at Recall@1000. This head-versus-tail pattern is not explained or tested. Since the paper explicitly states that high values of k are the relevant regime for retrieval, the result may be acceptable, but without confidence intervals or multiple production runs, the observed differences are consistent with noise. The authors should either provide a mechanism for why the correction behaves differently at small k or report uncertainty.
minor comments (3)
- [Section 4.1; Table 1] The text says all results are averaged over 5 independent runs, but the table does not report standard deviations or any measure of dispersion; this makes it impossible to judge whether differences of 0.0001-0.001 are meaningful.
- [Section 4.3] The sentence 'while slightly trailing standard logQ only on MovieLens (temporal split)' is inconsistent with Table 1, where improved logQ also trails standard logQ for in-batch negatives on Steam leave-one-out (both NDCG@20 and R@20). Please make the summary of results match the table.
- [Section 4; Section 3.3] The notation Q' is used both for the distribution conditioned on excluding the positive item and for the log-probability replacement in the loss; the relation Q'(d) = Q(d)/(1-Q(p)) should be stated explicitly in Section 4, and the effect of the deduplication step on this distribution should be clarified.
Circularity Check
No significant circularity: the refined logQ loss is derived from the softmax gradient without fitting to target results.
full rationale
The paper's central derivation is self-contained. L_ours is obtained by decomposing the exact softmax gradient into a term for the positive item and an expectation over the remaining catalog, then replacing that expectation with a weighted importance-sampling estimate over sampled negatives. The weight w_up = sg(1 - P_theta(p|u)) is an algebraic byproduct of this decomposition, not a parameter fitted to retrieval metrics; it is computed from current model logits via a consistent importance-sampling estimate and is detached with a stop-gradient. No equation in the paper reduces the claimed improvement to the definition of the weight, and no result is imported from a self-citation chain: the cited prior work (Bengio and Senecal, Yi et al., Yang et al.) is external and used as background or baseline. The empirical claim of consistent improvements is testable against external benchmarks and is not guaranteed by construction. The main substantive concerns raised by the paper's own text — that the self-normalized estimate of P(p|u) uses the same negatives as the loss and may have high variance for rare positives, and that Table 1 shows a few cases where improved logQ trails standard logQ — are statistical and correctness risks, not circularity. Accordingly, the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- standard math Weighted, self-normalized importance sampling yields a consistent estimator of the softmax gradient as the number of negatives n grows.
- domain assumption The in-batch item distribution equals the empirical unigram distribution Q, and after discarding positives the renormalized distribution Q' is a valid proposal.
- domain assumption Approximating log Q'(d) by log Q(d) changes performance negligibly.
- domain assumption Recall@k with a 2M-item index is an appropriate proxy for retrieval quality.
Cite this review
Pith. "Pith review of Correcting the LogQ Correction: Revisiting Sampled Softmax for Large-Scale Retrieval." pith.science (2026). https://pith.science/paper/44MKPKEX
@misc{pith2026250709331,
author = {Pith},
title = {Pith review of: Correcting the LogQ Correction: Revisiting Sampled Softmax for Large-Scale Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/44MKPKEX}},
note = {Machine review of arXiv:2507.09331}
}
read the original abstract
Two-tower neural networks are a popular architecture for the retrieval stage in recommender systems. These models are typically trained with a softmax loss over the item catalog. However, in web-scale settings, the item catalog is often prohibitively large, making full softmax infeasible. A common solution is sampled softmax, which approximates the full softmax using a small number of sampled negatives. One practical and widely adopted approach is to use in-batch negatives, where negatives are drawn from items in the current mini-batch. However, this introduces a bias: items that appear more frequently in the batch (i.e., popular items) are penalized more heavily. To mitigate this issue, a popular industry technique known as logQ correction adjusts the logits during training by subtracting the log-probability of an item appearing in the batch. This correction is derived by analyzing the bias in the gradient and applying importance sampling, effectively twice, using the in-batch distribution as a proposal distribution. While this approach improves model quality, it does not fully eliminate the bias. In this work, we revisit the derivation of logQ correction and show that it overlooks a subtle but important detail: the positive item in the denominator is not Monte Carlo-sampled - it is always present with probability 1. We propose a refined correction formula that accounts for this. Notably, our loss introduces an interpretable sample weight that reflects the model's uncertainty - the probability of misclassification under the current parameters. We evaluate our method on both public and proprietary datasets, demonstrating consistent improvements over the standard logQ correction.
Reference graph
Works this paper leans on
-
[2]
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res. 3, null (March 2003), 1137–1155
work page 2003
-
[3]
Yoshua Bengio and Jean-Sébastien Senecal. 2003. Quick Training of Probabilistic Neural Nets by Importance Sampling. In Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. R4), Christopher M. Bishop and Brendan J. Frey (Eds.). PMLR, 17–24. https://proceedings.mlr.press/r4...
work page 2003
-
[4]
Eunjoon Cho, Seth A. Myers, and Jure Leskovec. 2011. Friendship and mobility: user movement in location-based social networks. InProceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Diego, California, USA) (KDD ’11). Association for Computing Machinery, New York, NY, USA, 1082–1090. doi:10.1145/2020408.2020579
arXiv 2011
-
[5]
Graham Cormode and S. Muthukrishnan. 2005. An improved data stream sum- mary: the count-min sketch and its applications. Journal of Algorithms 55, 1 (2005), 58–75. doi:10.1016/j.jalgor.2003.12.001
-
[6]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. New York, NY, USA
2016
-
[7]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy...
2019
-
[8]
F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (Dec. 2015), 19 pages. doi:10.1145/2827872
doi:10.1145/2827872 2015
-
[9]
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk
Show all 31 references
-
[10]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management (San Francisco, Ca...
2013
-
[11]
Yitong Ji, Aixin Sun, Jie Zhang, and Chenliang Li. 2023. A Critical Study on Data Leakage in Recommender System Offline Evaluation. ACM Trans. Inf. Syst. 41, 3, Article 75 (Feb. 2023), 27 pages. doi:10.1145/3569930
2023 doi
-
[12]
Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Rec- ommendation. In 2018 IEEE International Conference on Data Mining (ICDM) . 197–206. doi:10.1109/ICDM.2018.00035
2018
-
[13]
Anton Klenitskiy and Alexey Vasilev. 2023. Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?. In Proceedings of the 17th ACM Conference on Recommender Systems (Singapore, Singapore) (RecSys ’23). Association for Computing Machinery, New York, NY, USA, 1120–1...
2023 doi
-
[14]
Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. 2021. Embedding-based Product Retrieval in Taobao Search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (KDD ’21). A...
2021
-
[15]
Chi Liu, Jiangxia Cao, Rui Huang, Kai Zheng, Qiang Luo, Kun Gai, and Guorui Zhou. 2024. KuaiFormer: Transformer-Based Retrieval at Kuaishou. arXiv:2411.10057 [cs.IR] https://arxiv.org/abs/2411.10057
2024 arXiv
-
[16]
Zaiqiao Meng, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2020. Ex- ploring Data Splitting Strategies for the Evaluation of Recommendation Models. In Proceedings of the 14th ACM Conference on Recommender Systems (Virtual Event, Brazil) (RecSys ’20). Association for Com...
2020
-
[17]
Priyanka Nigam, Yiwei Song, Vijai Mohan, Vihan Lakshman, Weitian (Allen) Ding, Ankit Shingavi, Choon Hui Teo, Hao Gu, and Bing Yin. 2019. Semantic Product Search. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage, AK...
2019 doi
-
[18]
Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinner- Former: Sequence Modeling for User Representation at Pinterest. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Washington DC, USA) (KDD ’22). Association fo...
2022
-
[19]
Apurva Pathak, Kshitiz Gupta, and Julian McAuley. 2017. Generating and Per- sonalizing Bundle Recommendations on Steam. In Proceedings of the 40th Interna- tional ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR ’17). As...
2017
-
[20]
Aleksandr Vladimirovich Petrov and Craig Macdonald. 2023. gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling. In Proceedings of the 17th ACM Conference on Recommender Systems (Singapore, Singapore) (RecSys ’23). Association for Computi...
2023
-
[21]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme
-
[22]
Aixin Sun. 2023. Take a Fresh Look at Recommender Systems from an Evaluation Standpoint. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Taiwan) (SIGIR ’23). Association for Computing Machinery, New Yo...
2023
-
[23]
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang
-
[24]
Jiaxi Tang and Ke Wang. 2018. Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey, CA, USA) (WSDM ’18). Association for Computing Machinery,...
2018
-
[25]
Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, and Tianyu Qiu. 2024. On the Effectiveness of Sampled Softmax Loss for Item Recom- mendation. ACM Trans. Inf. Syst. 42, 4, Article 98 (March 2024), 26 pages. doi:10.1145/3637061
2024 doi
-
[26]
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith St...
2016 arXiv
-
[27]
Jing Yan, Liu Jiang, Jianfei Cui, Zhichen Zhao, Xingyan Bin, Feng Zhang, and Zuotao Liu. 2024. Trinity: Syncretizing Multi-/Long-Tail/Long-Term Interests All in One. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining (Barcelona, Spain) (K...
2024
-
[28]
Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaom- ing Wang, Taibai Xu, and Ed H. Chi. 2020. Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations. In Companion Proceedings of the Web Conference 2020 (Taipei, Taiwan)(WWW ’2...
2020
-
[29]
Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. 2019. Sampling-bias-corrected neural modeling for large corpus item recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems (Copenhage...
2019
-
[2009]
In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (Montreal, Quebec, Canada) (UAI ’09)
BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (Montreal, Quebec, Canada) (UAI ’09). AUAI Press, Arlington, Virginia, USA, 452–461
-
[2016]
In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.)
Session-based Recommendations with Recurrent Neural Networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1511.06939
2016 arXiv
-
[2019]
In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing, China) (CIKM ’19)
BERT4Rec: Sequential Recommendation with Bidirectional Encoder Rep- resentations from Transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing, China) (CIKM ’19). ACM, New York, NY, USA, 1441–1450. doi:10.1145/335738...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.