{"id":"8fcc695a-1520-4f92-9a0a-913c04df0876","arxiv_id":"2508.05673","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SoftmaxLoss@K weights each positive sample by how far its score exceeds an estimated top-K quantile, turning Softmax Loss into a smooth surrogate for NDCG@K that outperforms prior losses by about 6% on average.","lead":"Researchers introduce SoftmaxLoss@K, a new loss function for training recommender systems to directly optimize the truncated ranking metric NDCG@K. Tests on four datasets and three model backbones report roughly 6% average improvement in Recall@K and NDCG@K over existing losses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm C.1 may not implement the loss analyzed in Theorem 3.2: the sampled SL term omits the self term, so the upper-bound guarantee and the 6.03% claim may attach to a different loss.","rationale":"The reader's weakest assumption is the bias of the practical quantile estimator relative to Theorem 3.1. I do not find that to be the most load-bearing issue: because the sampled set is a subset of the full item set, its K-th order statistic cannot exceed the full-set K-th order statistic at the estimation time, so the estimated quantile is not above the ideal one. That preserves the key coefficient argument in Appendix B.2 for H>1, leaving only a staleness issue from the T_β update interval. The more serious problem is internal: Algorithm C.1 line 14 computes the sampled SL term over negative items only, omitting the self term that is essential in Eq. (3.5) and in the proof of Theorem 3.2. With that omission the implemented loss can be negative and is not an upper bound of -log DCG@K, so the theoretical guarantee does not apply to the loss as specified in the algorithm. This is a concrete, checkable discrepancy between the stated method and the analyzed method. The empirical improvements may still hold, and the paper provides code, detailed hyperparameters, and honest appendix discussion, so I am not recommending rejection on suspicion; I recommend a conditional verdict requiring verification of the code and, if needed, correction of either the algorithm or the theoretical claim. If the code includes the '+1' term and line 14 is simply a typographical abbreviation, the concern lands as a documentation issue rather than a correctness failure.","tokens_in":42773,"tokens_out":15356,"duration_ms":186777,"concrete_test":"Inspect the released repository (Tiny-Snow/IR-Benchmark) and check whether the SL@K denominator is `1 + sum_{j in negatives} exp((s_j-s_i)/tau)` or `sum_{j in negatives} exp((s_j-s_i)/tau)` as Algorithm C.1 line 14 states. Then rerun the Electronic/MF configuration once with each denominator. If the shipped code omits the self term and NDCG@20 shifts materially, the upper-bound theorem and the headline improvement attach to a different objective than the one analyzed; if the code includes the self term, line 14 is a typo and the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing condition for the central claim is that the loss actually optimized is the one bounded in Theorem 3.2. As written, this fails. Equation (3.5) defines the inner SL term as log Σ_{j∈I} σ_d(d_uij), and the proof of Theorem 3.2 relies on the self term σ_d(d_uii)=1 to bound log π_ui. Algorithm C.1, however, computes at line 14 L_SL(u,i) = log Σ_{j∈\\hat N_u} σ_d(d_uij), where \\hat N_u is a set of uniformly sampled negative items only; no '+1' and no positive self term appears. This is not a minor sampling artifact: for well-separated positives the sampled sum can fall below 1, making L_SL(u,i) negative, whereas the term it is supposed to bound, log π_ui, is nonnegative. The implemented loss therefore cannot be an upper bound of -log DCG@K in the sense of Theorem 3.2. If the released code follows the algorithm, the reported 6.03% gains are for a different loss than the one analyzed. The reader's quantile concern is less decisive: at the parameter state where the quantile is estimated, the K-th order statistic of a subset cannot exceed the K-th order statistic of the full set, so the biased estimator is downward and the coefficient replacement σ_w ≥ 1/2 in Appendix B.2 still goes through for H>1; the residual gap is quantile staleness over the T_β update interval, not a fundamental violation. The denominator mismatch, by contrast, is a direct break between the theory and the stated algorithm.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SoftmaxLoss@K (SL@K), a weighted softmax loss for optimizing NDCG@K in top-K recommendation. The loss weights each positive instance by a sigmoid of the excess of its score over an estimated top-K quantile, and trains the inner softmax term with exponential activations. The authors derive an upper bound relating -log DCG@K to the new loss (Theorem 3.2), provide a Monte Carlo quantile-estimation guarantee (Theorem 3.1), and report experiments on four recommendation datasets with three backbones, plus extensions to learning-to-rank, sequential recommendation, and link prediction, claiming an average 6.03% improvement over baselines. The manuscript is clearly written and the main theoretical derivation is self-contained given its relaxations, but the algorithm presented in the appendix does not implement the loss analyzed in the main text.","tokens_in":43094,"tokens_out":7107,"duration_ms":81218,"significance":"If the theory/implementation gap is resolved, the paper makes a useful contribution: a computationally cheap modification of softmax loss that explicitly models top-K truncation through a quantile-based weight, with a transparent upper-bound argument and broad empirical evaluation. The strengths are the availability of code, the detailed hyperparameter tables, and the systematic comparison against softmax variants and NDCG@K surrogate losses. The proposed loss is simple and plausible, and the empirical gains, if reproducible for the analyzed loss, would be practically meaningful. The main theoretical claim, however, currently attaches to a different objective than the one specified in the optimization algorithm, which blocks acceptance in its present form.","major_comments":[{"comment":"The loss analyzed in the paper is not the loss implemented in the algorithm. Equation (3.5) defines the inner softmax term as L_SL(u,i) = log Σ_{j∈I} σ_d(d_uij), and the proof of Theorem 3.2 in Appendix B.2 relies on the bound π_ui = Σ_{j∈I} δ(d_uij) ≤ Σ_{j∈I} σ_d(d_uij), which requires the self term j=i contributing σ_d(0)=1. Algorithm C.1, line 14, instead computes L_SL(u,i) = log Σ_{j∈\\hat N_u} σ_d(d_uij), where \\hat N_u is a set of uniformly sampled negative items only and contains no self term and no positive item. Because sampled negative items typically satisfy d_uij<0, the sum can be smaller than 1 and the log can be negative, while the term it is supposed to bound, log π_ui, is nonnegative. The upper-bound relation of Theorem 3.2 therefore does not hold for the objective actually minimized. If the released code follows Algorithm C.1, the reported 6.03% average improvement is for a different loss than the one covered by the theoretical guarantee. Please correct the algorithm to include the self term (for instance, log(1 + Σ_{j∈\\hat N_u} σ_d(d_uij))), or provide a separate theoretical analysis of the negative-only surrogate actually used.","section":"§3.1.2, Eq. (3.5) vs. Algorithm C.1 (line 14)"},{"comment":"Theorem 3.1 provides a finite-sample guarantee for quantile estimation from i.i.d. samples of the score distribution F_u, but the practical estimator described in Appendix C.2 and used in Algorithm C.1 draws \\hat I_u = P_u ∪ \\hat N_u and takes the K-th order statistic of this mixed set. These samples are not i.i.d. from F_u, and the appendix explicitly acknowledges that the resulting estimator has a downward bias. The main text’s statement that the Monte Carlo strategy offers theoretical precision guarantees is therefore not supported by Theorem 3.1 for the implemented estimator. Please either extend the theorem to cover the all-positive-plus-sampled-negatives estimator, or clearly state that the practical estimator is a heuristic with empirical support, and discuss how the downward bias and the quantile-update interval T_β affect the condition σ_w(s_ui − β) ≥ 1/2 used in the proof of Theorem 3.2.","section":"§3.1.3, Theorem 3.1 and Appendix C.2 (negative sampling trick)"},{"comment":"The headline claim of an average 6.03% improvement over the best baseline is computed across 24 Recall@20/NDCG@20 comparisons, but the paper reports no standard deviations, confidence intervals, or significance tests. Some individual improvements are small, for example +1.53% and +2.74% on Gowalla and Book Recall@20 with the MF backbone. Given that the empirical claim is central to the paper, the authors should either report repeated-run statistics or at least state explicitly how the average is aggregated and acknowledge the variability in small-improvement cells.","section":"Section 4.2, Table 2 (6.03% claim)"}],"minor_comments":[{"comment":"There are duplicate references: He et al. 2017 appears as both [27] and [28], and Cao et al. 2007 appears as both [7] and [8]; these should be consolidated.","section":"References"},{"comment":"The sentence about the H^K_u > 1 assumption is repeated almost verbatim in the main text and again in Appendix B.2; one version can be shortened.","section":"Footnotes 4 and 5"},{"comment":"The inequality chain in Eq. (3.3) is dense; in particular the step labeled ③ is correct because 1/H ≤ 1 and log π_ui ≥ 0, but the text could state this explicitly to avoid confusion about the direction of the inequality.","section":"Eq. (3.3)"},{"comment":"Line 14 reuses the symbol L_SL(u,i) for a sum over sampled negatives only, whereas Eq. (3.5) uses the same symbol for a sum over the full item set. Different notation would make the discrepancy between theory and implementation immediately visible.","section":"Algorithm C.1"},{"comment":"The caption of Figure 1 reports a 60.4x speedup while the text says “60 times”; please make the figure caption consistent with the text.","section":"Table 2 and Figure 1"}],"recommendation":"major_revision","confidential_remarks":"I did not inspect the repository, so my main concern rests on Algorithm C.1 as printed. If the released code actually includes the self term in the sampled softmax denominator, Major Comment 1 can be resolved by a correction and a reproducibility statement; if the code follows the printed algorithm, the empirical results currently lack the theoretical backing claimed in Theorem 3.2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is simple and likely useful: a per-user top-K quantile, estimated cheaply, becomes a soft weight on the Softmax Loss. The paper honestly states the H>1 condition and even the looser H=1 factor-of-1/2 bound. The experiments are extensive, with code, hyperparameter tables, and consistent gains across datasets and backbones. The quantization trick is practical and the acknowledged bias is reasonable. This is a credible incremental contribution to how people train top-K recommenders.\n\nThe problem is that the paper analyzes one loss and implements another. Theorem 3.2 bounds the full SL term, log Σ_{j∈I} exp(d_uij), which includes the self-term exp(0)=1. Algorithm C.1 computes instead log Σ_{j∈N̂_u} exp(d_uij) over sampled negatives only. For a well-separated positive, that sum can be below 1, making the loss negative, while the term it is supposed to bound, log π_ui, is nonnegative. So the implemented loss is not an upper bound of -log DCG@K in the theorem's sense. This is not a minor sampling detail; it is a direct break between the stated theory and the stated algorithm. The paper should either add the self-term (+1) in the algorithm or prove the bound for the sampled loss.\n\nThe quantile estimation concern raised by the reader's report is real but less severe: the practical estimator is biased downward, and the paper acknowledges this, so the theoretical coverage of Theorem 3.1 is narrower than advertised. Still, the weight replacement σ_w ≥ 1/2 goes through, so the gap is mostly staleness rather than a structural violation.\n\nMissing error bars also bother me: the headline 6.03% improvement has no variance information, and with four datasets and three backbones, some fluctuation is expected. That is a minor-to-moderate reporting issue, not a fatal one.\n\nBottom line: the method is worth engaging with and the empirical pattern is encouraging, but the theory as written does not cover the implemented loss. A serious referee should ask for a corrected algorithm or a corrected analysis before the guarantee is taken at face value. I would send it to peer review, but with a clear request for that fix plus variance reporting.","headline":"SL@K is a plausible practical upgrade to Softmax Loss for NDCG@K, but the loss actually optimized in Algorithm C.1 omits the self-term and is not the loss bounded in Theorem 3.2, so the theoretical guarantee needs a fix or a scoping clarification.","tokens_in":43715,"tokens_out":3498,"would_cite":false,"duration_ms":40188,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a quantile-weighted softmax loss, SL@K, is a provable surrogate for NDCG@K and beats existing recommendation losses by 6.03% on average.","keywords":["recommender systems","NDCG@K optimization","top-K ranking metrics","surrogate loss","softmax loss","quantile estimation","implicit feedback","learning to rank"],"falsifier":"Train SL@K with the paper's estimator and, each epoch, compute the exact top-K threshold per user from full scores; if for any user with more than one top-K hit the inequality $-\\log \\mathrm{DCG}@K(u) \\le \\mathcal{L}_{\\mathrm{SL}@K}(u)$ fails when the estimated threshold is used, then the practical algorithm violates the guarantee claimed in Theorem 3.2.","tokens_in":42527,"feed_emoji":"🎯","tokens_out":9160,"duration_ms":89825,"temperature":0.7,"pith_summary":"The paper argues that Softmax Loss, a standard surrogate for full-ranking NDCG, is mismatched with NDCG@K, the metric recommender systems are actually judged by, and that closing this gap improves real performance. It introduces SoftmaxLoss@K (SL@K), a weighted softmax loss in which every positive item receives a quantile-based weight that focuses training on items that can enter the top K. The central claim is that, for any user with more than one top-K hit, SL@K is an upper bound of $-\\log \\mathrm{DCG}@K$, so minimizing SL@K is consistent with maximizing NDCG@K. The paper reports an average 6.03% improvement over the best baselines across four datasets and three recommendation backbones, with the gains strongest at small K. If correct, this gives recommenders a surrogate that is as cheap to train as Softmax Loss while explicitly modeling truncation.","feed_headline":"New loss targets NDCG@K directly and beats baselines by 6%","feed_subtitle":"A quantile-weighted softmax loss targets top-K truncation directly; gains grow as K shrinks.","key_machinery":"The load-bearing object is the top-K quantile $\\beta^K_u$, the score threshold below which items fall out of the user's top K. It converts the combinatorial truncation term $\\mathbb{I}(\\pi_{ui}\\le K)$ into the simple comparison $\\mathbb{I}(s_{ui}\\ge \\beta^K_u)$, and smoothing that comparison with a sigmoid produces the per-example weight $w_{ui}=\\sigma_w(s_{ui}-\\beta^K_u)$ that turns ordinary Softmax Loss into SL@K. The second mechanism is the Monte Carlo quantile estimator: sample a few negatives, add all positives, sort, and take the K-th score; Theorem 3.1 bounds its error through a standard empirical-c.d.f. concentration inequality. Together they give a smooth, sampling-friendly surrogate whose gradient concentrates on positive items near or above the threshold.","core_discovery":"The paper's central discovery is that the top-K truncation in NDCG@K can be handled by a per-user threshold, the top-K quantile $\\beta^K_u = \\inf\\{s_{ui} : \\pi_{ui} \\le K\\}$, which separates items inside the top K from the rest. Replacing the threshold indicator with a sigmoid weight and bounding the discounted rank by a softmax sum yields SL@K: $\\mathcal{L}_{\\mathrm{SL}@K}(u)=\\sum_{i\\in P_u}\\sigma_w(s_{ui}-\\beta^K_u)\\log \\sum_{j\\in I}\\exp((s_{uj}-s_{ui})/\\tau_d)$. The paper proves that, whenever the number of top-K hits $H^K_u$ exceeds 1, SL@K is an upper bound of $-\\log \\mathrm{DCG}@K(u)$, and that a Monte Carlo estimator with $N$ sampled scores has quantile error that decays exponentially in $N$ (Theorem 3.1). On this basis the paper claims that optimizing SL@K is a theoretically grounded way to maximize NDCG@K, and the experiments support the claim that the bound transfers into gains on real top-K metrics.","pith_inferences":["Given the paper's admission that the practical quantile estimate is biased downward, SL@K is effectively optimizing a slightly larger cutoff than the specified K; an explicit bias-correction term might tighten the match to NDCG@K, especially for small K.","The exponential error bound in Theorem 3.1 depends on score density near the true quantile; on datasets where many items tie near the K-th position, the practical N=1000 sample could be far from the stated guarantee, suggesting a density-aware sample size rule.","A natural extension, which the paper lists as future work, is incremental quantile updating; if the threshold can be updated online, SL@K becomes usable for streaming and continual recommendation.","Because the main bound requires $H^K_u>1$, users with a single top-K hit get only the weaker half-log bound; weighting users by their hit count could make the objective uniformly aligned with NDCG@K."],"forward_implications":["Training with SL@K should improve NDCG@K and Recall@K relative to full-ranking surrogates, with the largest gains at small K because truncation weighs most heavily there.","SL@K inherits the ease of Softmax Loss: the only added component is a per-user quantile estimate with $O(|U|N\\log N)$ overhead, so existing softmax-based recommender code can adopt it with minimal change.","The moderate gradient distribution reported for SL@K implies it can use sampled negatives without the instability that the paper documents for LambdaLoss@K and SONG@K.","Because the weighting automatically down-weights low-scoring positives, SL@K should degrade more gracefully than SL when false-positive noise is present in the interaction data.","The same loss transfers to other top-K tasks such as learning to rank, sequential recommendation, and link prediction, where the paper reports consistent gains over its baselines."],"supporting_citations":[{"why":"Supplies the Softmax Loss that SL@K extends, including the negative-sampling training procedure and the NDCG upper-bound perspective.","marker":"[82]"},{"why":"PSL's refined softmax upper bound and its experimental protocol are the baseline setup SL@K is built on and compared with.","marker":"[84]"},{"why":"Establishes that softmax cross-entropy bounds -log NDCG for binary relevance, the result SL@K adapts to the truncated NDCG@K setting.","marker":"[4]"},{"why":"Introduces LambdaLoss@K, the truncation-aware surrogate whose exact ranking computation and long-tailed gradients motivate SL@K's efficiency and stability properties.","marker":"[31]"},{"why":"Introduces SONG@K, the bilevel compositional NDCG@K surrogate whose quantile-like threshold and unstable gradients SL@K contrasts with.","marker":"[59]"},{"why":"Supplies the empirical-process concentration inequality used to prove the exponential quantile estimation error bound in Theorem 3.1.","marker":"[49]"}],"fun_headline_variants":["Quantile-based loss optimizes NDCG@K directly, up 6%","SL@K: Smooth upper bound for NDCG@K, proven stable","Break top-K barrier with per-user quantile threshold","Handle top-K truncation, boost NDCG@K by 6%","Direct NDCG@K optimization via quantile-weighted softmax"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical estimator keeps all positive items and samples only negatives, and the paper admits this biases the threshold downward; if that biased threshold marks the wrong positives as top-K, the weighting no longer tracks NDCG@K.","fun_headline_variants_meta":{"raw":{"variants":["Quantile-based loss optimizes NDCG@K directly, up 6%","SL@K: Smooth upper bound for NDCG@K, proven stable","Break top-K barrier with per-user quantile threshold","Handle top-K truncation, boost NDCG@K by 6%","Direct NDCG@K optimization via quantile-weighted softmax"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000389,"raw_usage":{"total_tokens":2098,"prompt_tokens":1038,"completion_tokens":1060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":962}},"tokens_in":654,"tokens_out":1060,"duration_ms":10452,"temperature":1.0,"reasoning_tokens":962,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:37:14.202852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SL@K with the paper's estimator and, each epoch, compute the exact top-K threshold per user from full scores; if for any user with more than one top-K hit the inequality $-\\log \\mathrm{DCG}@K(u) \\le \\mathcal{L}_{\\mathrm{SL}@K}(u)$ fails when the estimated threshold is used, then the practical algorithm violates the guarantee claimed in Theorem 3.2.","supporting_citations":[{"cited_title":"PSL: Rethinking and Improving Softmax Loss from Pairwise Perspective for Recommendation","cited_arxiv_id":"2411.00163","evidence_quote":"PSL's refined softmax upper bound and its experimental protocol are the baseline setup SL@K is built on and compared with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces LambdaLoss@K, the truncation-aware surrogate whose exact ranking computation and long-tailed gradients motivate SL@K's efficiency and stability properties."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the empirical-process concentration inequality used to prove the exponential quantile estimation error bound in Theorem 3.1."}],"review_version":2}