Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training

Alexey Zaytsev; Artyom Sabitov; Daniil Volkov

REVIEW 5 major objections 5 minor 31 references

Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Under a fixed memory budget, sampled-softmax recommender training converges fastest when nearly all memory goes to batch size and almost none to negative sampling.

desk verdict Good question, clean empirical trend, but the theoretical derivation doesn't reach the headline rule and the final-quality claims overreach. read the letter →

arxiv 2608.11061 v1 pith:O73CVVVU submitted 2026-08-11 cs.LG

Artyom Sabitov , Daniil Volkov , Alexey Zaytsev This is my paper · ORCID

classification cs.LG

keywords SequentialRecommendationCross-EntropyLossOptimizationNegativeSamplingSampledSoftmaxIn-BatchNegativesTrainingEfficiencyMemory-EfficientGradientVariance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to settle a practical trade-off in large-catalog recommender training: if memory is fixed, should you use the budget to enlarge the batch or to sample more negative items for the softmax? It argues from stochastic-gradient variance that the answer is to spend almost everything on batch size, taking $n \sim B$ and $k \sim 1$, with $n = k = \sqrt{B}$ as the feasible balanced point when every target class must be in the sampled set. If correct, the rule replaces heuristic grid search with a one-line configuration, which matters because the softmax layer is the dominant memory cost when catalogs reach hundreds of thousands of items. The paper supports the rule with a variance bound for a last-layer logistic-regression model and with experiments on MovieLens-1M, Gowalla, Netflix, and MovieLens-20M showing faster convergence and competitive or better final ranking quality for batch-heavy allocations.

What carries the argument

The carrying mechanism is a two-source variance decomposition for the corrected sampled-softmax gradient of the last layer. The proof splits the gradient into a term from sampled classes and a term from sampled examples, applies the Fenton-Wilkinson approximation (replacing the sum of log-normal exponentiated logits by one log-normal with the same mean and variance) plus the delta method, and arrives at $\mathrm{Var}(g_{n,k}) \le C_y/n + C_{\hat p}/k + \mathrm{const}$. Minimizing this bound under the product constraint $B = n k$ produces the allocation rule $n \sim B$, $k \sim 1$, and the balanced variant $n = k = \sqrt{B}$ under the feasibility constraint $k \ge n$.

What would settle it

Measure the actual last-layer stochastic-gradient variance in the paper's synthetic logistic-regression setup at a fixed budget $B$ while sweeping $k$ from 1 up to the balanced point and beyond. If the variance minimum lands at $k$ much larger than 1, or if the variance does not follow the bound $C_y/n + C_{\hat p}/k$, the allocation rule fails; the paper's own Figure 6 already limits the approximation's validation to $k \ge 16$.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two sources of noise in sampled-softmax training—mini-batch sampling and class sampling—should not split a fixed memory budget equally. For a multiclass logistic-regression model of the final layer, under assumptions that logits are nearly Gaussian with bounded variance and that the positive class is rare among exponentiated logits, the corrected gradient variance satisfies $\mathrm{Var}(g_{n,k}) \le C_y/n + C_{\hat p}/k + \mathrm{const}$ over the constraint $n k = B$. Minimizing this bound yields $n^* \sim B$ and $k^* \sim 1$, and with the added requirement that the sampled class set contain every target class, $n^* = k^* = \sqrt{B}$. The paper concludes that memory should go to batch size first, and reports that batch-heavy configurations converge faster and often reach better final NDCG@10 than negative-heavy configurations at equal memory on synthetic data and on four sequential recommendation benchmarks.

Load-bearing premise

The whole rule rests on the variance estimate staying accurate at very small numbers of negatives, even though the derivation assumes many negatives and the numerical check of the approximation starts at 16.

Editorial extensions

If this is right

With a fixed memory budget $B = n k$, batch-heavy allocations ($n \sim B$, $k \sim 1$) should converge faster than negative-heavy ones at every stage of training, not merely at the end.
The feasible balanced configuration $n = k = \sqrt{B}$, used when each target must appear in the sampled class set, is the recommended default and should beat imbalanced alternatives at equal memory.
Because the variance structure transfers to momentum and Adam-type methods, the allocation rule applies beyond vanilla SGD.
The unbiased correction of Theorem 3.2 did not measurably change optimization dynamics, so the observed convergence gain is attributed to the $(n, k)$ allocation itself, not to the loss correction.
The same $1/n$ plus $1/k$ variance structure suggests the rule extends to other large-output softmax training settings, a broader principle the paper explicitly claims.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper does not run is to measure end-to-end gradient variance in a full transformer recommender at $k = 1, 2, 4$; if earlier layers contribute substantial noise, the optimal allocation could shift away from extreme batch-heavy settings.
The $\sqrt{B}$ rule depends on the feasibility constraint $k \ge n$ that assumes each target class in a batch is distinct and must be included in the sampled set; relaxing that assumption, for example with overlapping in-batch targets, could move the optimum toward smaller $n$.
Since the variance bound contains an additive constant, the advice to maximize $n$ is asymptotic in the budget $B$; at small budgets the constant may dominate and negative-heavy configurations could win, a crossover the synthetic experiments could quantify directly.
The same variance decomposition could be applied to allocate memory between batch size and sampled tokens in language-model softmax heads, where the logit tensor is likewise the dominant memory cost.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper has the right question and a clean empirical trend, but the theory doesn't get to the headline rule, and the claims about final quality outrun the data. I'd send it to review with the expectation of major revision, not desk-reject it.

What's new: the authors frame sampled-softmax training as a fixed budget B = n*k and decompose gradient variance into a mini-batch term and a class-sampling term. That decomposition is a useful way to think about the trade-off. The synthetic experiments in Figure 2 show AUL dropping as n/k increases across several B values, and the real-data results do show faster convergence (lower AUL) for the n=128,k=128 configuration in most cases. They also cite the relevant prior work, including Zhelnin et al. and Klenitskiy & Vasilev, and they don't oversell novelty relative to those empirical observations.

The problems are in the theory and in the final-quality claim. Theorem 3.2's unbiasedness proof only matches first moments approximately and drops -1 terms; that's not enough to establish unbiasedness. Theorem 3.3 derives a variance bound under K >> k >> 1, using the Fenton-Wilkinson approximation and a 1+O(1/k) expansion, and Figure 6 only validates Fenton-Wilkinson for k >= 16. Then Theorem 3.4 minimizes that bound at k* = O(1), i.e., exactly outside the regime where the bound is valid. So the n ~ B, k ~ 1 allocation is not supported. The n = k = sqrt(B) rule used in practice is just the boundary of the k >= n feasibility constraint; it does not follow from the variance minimization.

The experiments are mixed on final NDCG@10. On MovieLens-20M with SGD, (128,128) gives 0.0093 vs 0.0176 for (32,512), so larger n is substantially worse. The abstract's "better final recommendation quality" is too strong. The convergence speed evidence is real, but that's a weaker claim than the abstract makes. The code link wasn't accessible in the review material, so I couldn't verify the reproducibility claims.

In sum: the variance decomposition is a fair formalization, and the observation that larger batches speed up convergence under fixed B is plausible and worth knowing. But the paper's own theorem does not prove the rule, and the rule is not independently validated for final quality. A serious referee could salvage this by re-deriving the variance bound at small k or reframing the contribution as an empirical study. I would not cite it in its current form.

Referee Report

5 major / 5 minor

Summary. The manuscript studies how to split a fixed memory budget B = n k between batch size n and number of sampled negatives k in sampled-softmax training of sequential recommender systems. Section 3 develops an unbiased-correction estimator (Theorem 3.2), a variance bound for the last-layer gradient (Theorem 3.3), and an allocation rule (Theorem 3.4) claiming that n ~ B, k ~ 1, with the practical rule n = k = sqrt(B). Validation is carried out on synthetic data and on four real datasets (MovieLens-1M, Gowalla, Netflix, MovieLens-20M) using SASRec with SGD and Adam. The paper also provides public reproducibility materials.

Significance. The question is well motivated and practically relevant, and the paper identifies a real memory bottleneck in large-catalog recommender training. The manuscript has clear strengths: a controlled synthetic setup, real-data benchmarks across four datasets, reproducibility materials, and an explicit, actionable configuration rule. However, the central theoretical result does not support the headline allocation: the variance bound is minimized outside the regime in which it is derived, the constants in the bound contradict the minimizer computation, and the unbiasedness theorem is only approximate. The empirical evidence is also mixed, e.g., MovieLens-20M SGD NDCG@10 degrades under the recommended configuration. As it stands, the manuscript does not establish its central claim.

major comments (5)

[Sec. 3.3, Appendix A.1] Theorem 3.2 is stated as an exact equality E p''_j = E p_j, but the proof only matches first moments approximately: the correction factors use (alpha + k)/(alpha + K), whereas the ratio E[p_i]/E[p'_i] equals (alpha + k - 1)/(alpha + K - 1) for the target class and K/k times this for negative classes. The proof explicitly neglects the -1 terms. This approximation is not controlled at the k ~ 1 allocations recommended later, where alpha + k need not be much larger than 1. Additionally, the theorem assumes alpha_j is the ratio of true expectations, while the practical estimator uses \hat{alpha}_j computed from current logits and sampled negatives; the randomness of \hat{alpha}_j is not included in the unbiasedness analysis. Since Theorem 3.1 requires a conditionally unbiased gradient estimator, the variance-to-convergence link is not established.
[Sec. 3.4, Appendix A.2] The variance bound in Theorem 3.3 is derived under K >> k >> 1: the proof states 'we assume K >> k >> 1, and, thus, k - 1 ~ k', uses the Fenton-Wilkinson log-normal approximation for the sum S, and expands (1 + (exp(sigma_l^2) - 1)/k)^2 = 1 + O(1/k). Figure 6 validates the approximation only for k >= 16. Theorem 3.4 then minimizes this bound at k* = O(1), exactly outside the derivation regime. At k = 1 the omitted O(1/k) corrections are O(1), so the claimed monotone decrease of the class-sampling variance term in k is not controlled and can reverse. This is the load-bearing step for the n ~ B, k ~ 1 conclusion. The proof also replaces the variance of the batch average by 1/n times a single-example variance, which requires independent class sampling across examples, whereas the setup of Section 3.2 and the in-batch negative implementation use a shared set S; this dependence is not addressed.
[Sec. 3.4 vs. Appendix A.2] The theorem statements and the proof use inconsistent variance formulas. Theorem 3.3 states Var(g_n,k) <= C_y/n + C_hatp/k with C_y ~ O(1/(B K)) and C_hatp ~ O(1/K), while Appendix A.2 concludes Var(g_n,k) <= C_y/(K n) + C_hatp/(B K k). These are not the same up to rescaling, and they lead to different optimizers. Even using the appendix form, the proof of Theorem 3.4 obtains n* = B sqrt(C_y/C_hatp); substituting the stated magnitudes gives n* = O(sqrt(B)) and k* = O(sqrt(B)), not n* ~ B, k* ~ 1. To obtain n* ~ B one needs C_y/C_hatp = O(1), which contradicts the orders declared in Theorem 3.3. The minimizer computation is therefore internally inconsistent.
[Sec. 3.4, Sec. 4.5, Table 2] The practical rule n = k = sqrt(B) is the boundary of the feasibility region, not a consequence of the variance minimization. With B = n k and distinct positive classes, the constraint k >= n forces n <= sqrt(B) and k >= sqrt(B); maximizing n under this constraint gives n = k = sqrt(B). Thus, once a monotone decrease of the variance proxy in n is assumed, the rule is forced and provides no independent support for the theory. Table 2 only tests (32,512), (64,256), and (128,128) for B = 16384, i.e., the high-k side of the feasible region, and no real-data configuration with n > k is evaluated. Section 4.5 states that 'in further experiments we consider k <= n only', which is contradicted by the two rows with k > n in Table 2.
[Table 2] The empirical results are mixed and do not consistently support the headline claim. In MovieLens-1M with SGD, AUL increases from 46.5 at (32,512) to 61.8 at (128,128), so the recommended configuration has worse convergence speed by the paper's own metric, since lower AUL is better. In MovieLens-20M with SGD, AUL improves from 233.1 to 61.5 while NDCG@10 drops from 0.0176 to 0.0093, contradicting the abstract's claim of 'better final recommendation quality'. In Gowalla with SGD, the best NDCG@10 is attained at (64,256), not at (128,128). The abstract's claim that the suggested configuration achieves faster convergence and better final quality is therefore not supported by the reported numbers.

minor comments (5)

[Abstract] The abstract contains duplicated text 'controlled synthetic and synthetic and four real sequential recommendation benchmarks' and a subject-verb disagreement in 'The suggested configuration achieve'.
[Sec. 3.4] The name 'Felton-Wilkinson' is a misspelling of 'Fenton-Wilkinson', and the approximation is attributed to Ref. [12], a normal limit theorem by Marlow, rather than to the standard moment-matching lognormal approximation reference.
[Sec. 3.4, Theorem 3.3] The 'variance of the gradient' is not defined for a matrix-valued random variable; the paper should specify the norm (e.g., Frobenius norm) and state whether the bound is per coordinate or summed over all coordinates.
[Sec. 4.1, Figures 4-5] The convergence curves are plotted against the number of processed batches, but with different batch sizes each batch corresponds to a different amount of training data; the AUL definition should clarify whether Delta_t is wall-clock time, a step count, or an epoch measure, since this affects the comparability of the reported AUL values.
[Sec. 3.4] The word 'Conseqence' in the consequence following Theorem 3.4 should be corrected to 'Consequence'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the allocation rule is obtained by minimizing a derived variance bound, with no fitted parameters and no load-bearing self-citations.

full rationale

The paper's derivation chain is self-contained in the circularity sense. Theorem 3.4 minimizes the variance upper bound of Theorem 3.3, whose constants C_y and C_hatp are derived in Appendix A.2 from assumptions A1-A3 (Gaussian logits, bounded variance, and h_pos/K << h_neg) using the Fenton-Wilkinson log-normal approximation; they are not fitted to any benchmark results. The practical rule n=k=sqrt(B) is explicitly introduced as a compromise for the computational regime k>=n ('However, in practice computations become efficient for k>=n and we use the rule n=k=sqrt(B)'), not as the unconstrained optimum, so the empirical comparison of (32,512), (64,256), and (128,128) is a test of that practical boundary rule, not a circular restatement of the theorem. The only self-citation ([27], with overlapping authors Volkov and Zaytsev) is used for background observations about sequential recommendation training and heuristic tuning; it is not load-bearing for the variance analysis, the unbiased estimator, or the minimization argument. The legitimate weakness is that the variance bound is derived under K >> k >> 1 and Figure 6 validates the log-normal approximation only for k >= 16, while the unconstrained minimizer k* is O(1); minimizing the bound outside its derivation regime is a correctness/validity concern, not an equation-level circularity.

Assumptions & free parameters 1 free parameters · 7 assumptions · 0 invented entities

The central claim rests on unvalidated distributional assumptions (A1-A3), an approximation for sums of log-normals that is checked only for moderately large k, and the assumption that last-layer gradient variance governs total-model convergence. The practical rule n=k=sqrt(B) also depends on the feasibility constraint k >= n.

free parameters (1)

logit variance bound C = 1.2
Hand-chosen threshold in assumption A2 limiting the variance of last-layer logits; it controls the quality of the Fenton-Wilkinson approximation but is not derived from data or theory.

assumptions (7)

domain assumption A1: Last-layer logits l_i are normally distributed with mean zero.
Invoked in Section 3.4 and Appendix A.2 to justify the log-normal approximation for exponentials h_i; only partially supported by [22] for general pre-activations, not validated for the softmax layer in the trained models.
domain assumption A2: logit variance sigma_l^2 is bounded by C=1.2.
Used to justify the Fenton-Wilkinson fit in Figure 6; C is hand-chosen and not verified on real models.
domain assumption A3: h_pos/K << h_neg, meaning the positive-class exponent is much smaller than K times the average negative exponent.
Stated as due to label noise in recommendation systems; no empirical check is provided, and it is load-bearing for dropping the target-class variance contribution in Lemma A.1.
ad hoc to paper The Fenton-Wilkinson approximation: the sum of k lognormal variables is lognormal with matched mean and variance.
Used in Lemma A.1 to estimate the variance of the sampled-softmax denominator; Figure 6 validates it only for k >= 16, while the recommended k* is O(1).
ad hoc to paper The k >> 1 expansion of the variance formula remains valid at the optimum k ~ 1.
The bound is derived for K >> k >> 1 in Appendix A.2 but minimized at k* = sqrt(C_phat/C_y), which is O(1), outside the derivation regime.
standard math Standard SGD and Adam convergence bounds apply and depend linearly on gradient variance (Theorem 3.1, [3]).
Background for translating variance into convergence rate; standard, but only for unbiased estimators with bounded variance, which the corrected estimator only approximately satisfies.
domain assumption The last-layer gradient variance controls total network gradient variance, and most parameters live in the last layer.
Stated in Section 3.2 as justification for the logistic-regression abstraction; not proven for SASRec-like transformer models.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training." pith.science (2026). https://pith.science/paper/O73CVVVU

@misc{pith2026260811061,
  author       = {Pith},
  title        = {Pith review of: Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O73CVVVU}},
  note         = {Machine review of arXiv:2608.11061}
}

abstract

Large-scale neural recommender systems are typically trained with a softmax cross-entropy objective over the full item vocabulary. For a typical large number of possible items $K$, the final classification layer dominates memory, requiring $O(nK)$ logits and gradients to materialize for a batch of $n$ examples. Sampled softmax reduces this cost by restricting the objective to only $k \ll K$ candidate negative items, resulting in an $O(nk)$ memory. However, for a fixed budget $B = n k$, it remains unclear whether one should prioritize larger batches or the inclusion of more negative items. We address this question by analyzing sampled-softmax training under a fixed memory constraint. Under standard smoothness and variance assumptions, our theoretical evidence suggests that the fastest convergence arises from an $ n \sim B, k \sim 1$ allocation. So, an actionable rule is to include as many objects as possible given computational constraints. Our theory is supported by controlled synthetic and synthetic and four real sequential recommendation benchmarks, including MovieLens-20M. The suggested configuration achieve faster convergence and better final recommendation quality than imbalanced alternatives within the same memory constraint. These findings provide a theoretical and empirical foundation for configuring memory during the training of recommender systems. Code, reproducibility materials, and all scripts for generating figures are available at https://anonymous.4open.science/r/LimitedMemoryRule-BBFB

Figures

Figures reproduced from arXiv: 2608.11061 by the authors.

**Figure 1.** The convergence significantly accelerates as we increase the batch size 𝑛 at the expense of reducing the number of negative samples 𝑘. The configurations dominated by larger batch sizes achieve lower loss values much earlier in the optimization process, confirming our theoretical results. Convergence Speed Quantification [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

**Figure 1.** Validation loss dynamics over training iterations for various [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

**Figure 2.** Area Under the Validation Loss Curve (AUL) as a [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: Average training time per epoch as a function of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]

**Figure 5.** Figure 5: Validation NDCG@10 metric dynamics across [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Approximation of the sum of 𝑘 lognormal random variables Lognorm(0, 𝜎𝑙) via CLT and Fenton–Wilkinson. The true distribution is reconstructed using Monte Carlo KDE. Better to view in zoom [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 16 canonical work pages

[1]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normaliza- tion.arXiv preprint arXiv:1607.06450(2016)

arXiv 2016
[2]

Guy Blanc and Steffen Rendle. 2018. Adaptive Sampled Softmax with Kernel- Based Sampling. InProceedings of the 35th International Conference on Machine Learning (ICML ’18). PMLR, 590–599

work page 2018
[3]

Curtis, and Jorge Nocedal

Léon Bottou, Frank E. Curtis, and Jorge Nocedal. 2018. Optimization Methods for Large-Scale Machine Learning.SIAM Rev.60, 2 (2018), 223–311. doi:10.1137/ 16M1080173

work page 2018
[4]

Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. 2020. Lower bounds for finding stationary points I.Mathematical Programming184, 1 (2020), 71–120

work page 2020
[5]

Danil Gusak, Gleb Mezentsev, Ivan Oseledets, and Evgeny Frolov. 2024. RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM ’24). ACM. doi:10.1145/3627673.3679986

arXiv 2024
[6]

Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1–10

work page 2015
[7]

Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In2018 IEEE international conference on data mining (ICDM). IEEE, 197–206

2018
[8]

Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

Show all 31 references

[9]

Kirill Khrylchenko, Vladimir Baikalov, Sergei Makeev, Artem Matveev, and Sergei Liamaev. 2025. Correcting the LogQ Correction: Revisiting Sampled Softmax for Large-Scale Retrieval. InProceedings of the 19th ACM Conference on Recommender Systems (RecSys ’25). ACM, New York, NY,...

2025 arXiv
[10]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Op- timization.International Conference on Learning Representations (ICLR)(2015). arXiv preprint arXiv:1412.6980

2015 arXiv
[11]

Anton Klenitskiy and Alexey Vasilev. 2023. Turning Dross Into Gold Loss: Is BERT4Rec Really Better than SASRec?. InProceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23). ACM, New York, NY, USA. doi:10.1145/ 3604915.3610644

2023
[12]

NA Marlow. 1967. A normal limit theorem for power sums of independent random variables.The Bell System Technical Journal46, 9 (1967), 2081–2089

1967
[13]

Gleb Mezentsev, Danil Gusak, Ivan Oseledets, and Evgeny Frolov. 2024. Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs. InProceedings of the 18th ACM Conference on Recommender Systems (RecSys ’24). ACM, New York, NY, USA. arXiv:2409.18721 [cs...

2024 arXiv
[14]

Aleksandr Petrov and Craig Macdonald. 2023. gSASRec: Reducing Overcon- fidence in Sequential Recommendation Trained with Negative Sampling. In Proceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23). ACM, New York, NY, USA, 116–128. doi:10.1145/3604915.3608783

2023
[15]

Yu, Ananda Theertha Suresh, and Sanjiv Kumar

Ankit Singh Rawat, Jiecao Chen, Felix Xinnan X. Yu, Ananda Theertha Suresh, and Sanjiv Kumar. 2019. Sampled Softmax with Random Fourier Features. In Advances in Neural Information Processing Systems (NeurIPS 2019)

2019
[16]

Shaina Raza, Mizanur Rahman, Safiullah Kamawal, Armin Toroghi, Ananya Raval, Farshad Navah, and Amirmohammad Kazemeini. 2026. A comprehensive review of recommender systems: Transitioning from theory to practice.Computer Science Review59 (2026), 100849

2026
[17]

Francesco Ricci, Lior Rokach, and Bracha Shapira. 2021. Recommender systems: Techniques, applications, and challenges.Recommender systems handbook(2021), 1–35

2021
[18]

Ohad Shamir and Tong Zhang. 2013. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. InInterna- tional conference on machine learning. PMLR, 71–79

2013
[19]

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang
[20]

Bohan Wang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma, and Wei Chen
[21]

Erik Wijmans, Brody Huval, Alexander Hertzberg, Vladlen Koltun, and Philipp Krähenbühl. 2025. Cut your losses in large-vocabulary language models. In International Conference on Learning Representations, Vol. 2025. 68174–68193

2025
[22]

Pierre Wolinski and Julyan Arbel. 2025. Gaussian Pre-Activations in Neural Networks: Myth or Reality?Transactions on Machine Learning Research Journal (2025), 1–50

2025
[23]

Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, and Tianyu Qiu. 2024. On the effectiveness of sampled softmax loss for item recommendation. ACM Transactions on Information Systems42, 4 (2024), 1–26

2024
[24]

Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, Tianyu Qiu, and Xiangnan He. 2024. On the Effectiveness of Sampled Softmax Loss for Item Recommendation.ACM Transactions on Information Systems42, 4 (2024). doi:10.1145/3637061

2024 doi
[25]

Y Yan, T Yang, Z Li, Q Lin, and Y Yang. 2018. A unified analysis of stochastic momentum methods for deep learning. InIJCAI International Joint Conference on Artificial Intelligence

2018
[26]

Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed H. Chi. 2019. Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations. InProceedings of the 13th ACM Conference on Recommender Systems (RecSys ...

2019
[27]

Maxim Zhelnin, Dmitry Redko, Daniil Volkov, Anna Volodkevich, Petr Sok- erin, Valeriy Shevchenko, Egor Shvetsov, Alexey Vasilev, Darya Denisova, Rus- lan Izmailov, and Alexey Zaytsev. 2026. Faster and Memory-Efficient Train- ing of Sequential Recommendation Models for Large Ca...

2026 arXiv
[28]

Pablo Zivic, Hernan Vazquez, and Jorge Sanchez. 2024. Scaling Sequential Rec- ommendation Models with Transformers. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). ACM, New York, NY, USA. doi:10.114...

2024
[2019]

InProceedings of the 28th ACM international conference on information and knowledge management

BERT4Rec: Sequential recommendation with bidirectional encoder rep- resentations from transformer. InProceedings of the 28th ACM international conference on information and knowledge management. 1441–1450
[2020]

arXiv:2001.08361 [cs.LG]

Scaling Laws for Neural Language Models.CoRRabs/2001.08361 (2020). arXiv:2001.08361 [cs.LG]

2020 arXiv
[2024]

arXiv:2403.15146 [cs.LG]

On the Convergence of Adam under Non-Uniform Smoothness: Separability from SGDM and Beyond.CoRRabs/2403.15146 (2024). arXiv:2403.15146 [cs.LG]

2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normaliza- tion.arXiv preprint arXiv:1607.06450(2016)

arXiv 2016

[2] [2]

Guy Blanc and Steffen Rendle. 2018. Adaptive Sampled Softmax with Kernel- Based Sampling. InProceedings of the 35th International Conference on Machine Learning (ICML ’18). PMLR, 590–599

work page 2018

[3] [3]

Curtis, and Jorge Nocedal

Léon Bottou, Frank E. Curtis, and Jorge Nocedal. 2018. Optimization Methods for Large-Scale Machine Learning.SIAM Rev.60, 2 (2018), 223–311. doi:10.1137/ 16M1080173

work page 2018

[4] [4]

Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. 2020. Lower bounds for finding stationary points I.Mathematical Programming184, 1 (2020), 71–120

work page 2020

[5] [5]

Danil Gusak, Gleb Mezentsev, Ivan Oseledets, and Evgeny Frolov. 2024. RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM ’24). ACM. doi:10.1145/3627673.3679986

arXiv 2024

[6] [6]

Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1–10

work page 2015

[7] [7]

Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In2018 IEEE international conference on data mining (ICDM). IEEE, 197–206

2018

[8] [8]

Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei