REVIEW 3 major objections 4 minor 37 references
Selective Matching Losses -- Not All Scores Are Created Equal
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A matching loss's link slope can be chosen to emphasize chosen score regions; composite Softmax with nonlinear transform $Q(z)$ extends this to multi-class with ranking sensitivity.
desk verdict A genuinely new construction for selective losses with coherent math, but the application-level claims are ahead of the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the link function $h(z)$ (with primitive $H(z)$) in the matching loss $L_m(\hat s,s)=H(\hat s)-H(s)-(\hat s-s)h(s)$. Its slope $h'(z)$ is the local sensitivity, and the bias-underspecification sensitivity (BUST) analysis computes how an underspecified model's optimal prediction shifts toward high-slope regions: the bias sensitivity is proportional to $h'(w_u)/(h(S_M)-h(S_m))$. In the multi-class case the mechanism is composite Softmax: replacing the standard log partition with $H(z)=\log\sum_k e^{Q(z_k)}$ makes each link component $h_k(z)=q(z_k)p_k(z)$, so the scaling $q=Q'$ and the Softmax probability $p_k$ act as two knobs for region and ranking sensitivity. A monotone nondecreasing $q$ (convex $Q$) is the sufficient condition that makes the induced scalar and multi-class losses convex.
What would settle it
Train the same architecture on a ranking or distillation task with standard Softmax and with composite Softmax using a fixed nonlinear $Q(z)$ (say $q(z)=e^z$), keeping all other hyperparameters identical, and measure whether high-score items improve on a metric that weights high relevance (e.g., NDCG) in the way the sensitivity analysis predicts. If the trained model's weights do not shift toward high-sensitivity features as the BUST derivation prescribes, the central claim fails; a cleaner laboratory check is to perturb one observed score by a controlled $\delta$ in a high-sensitivity region and confirm that the loss gradient scales as $h'(\cdot)$ predicts.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a recipe for designing loss sensitivity rather than inheriting it from a fixed loss. A matching loss is the Bregman divergence $D_H(\hat s,s)$ with gradient $h(\hat s)-h(s)$, where the link $h$ is nondecreasing and $h'(z)$ is the local sensitivity; scalar choices of shifted scaled Sigmoid, exponential, anti-exponential, or $\sinh$ links produce four distinct sensitivity profiles. For multiple classes, the paper constructs composite Softmax probabilities $p_k(z)=e^{Q(z_k)}/\sum_j e^{Q(z_j)}$ and shows the link components become $h_k(z)=q(z_k)p_k(z)$ with $q=Q'$. The resulting multi-class matching loss has $k$-th gradient component $q(\hat s_k)p_k(\hat s)-q(s_k)p_k(s)$, which the authors assert provides flexibility unavailable with standard Softmax: $q$ controls region sensitivity and $p_k$ controls ranking sensitivity, so training can emphasize high scores, low scores, or high norms while remaining a proper Bregman loss.
Load-bearing premise
The claimed link between steeper link slopes and better trained-model behavior rests on the BUST underspecification model, which assumes underspecification acts as a small uniformly distributed interval of feature scores with an added bias; if real network underspecification does not look like that, the practical benefit of selective losses is not established.
Editorial extensions
If this is right
- Scalar regression and classification losses can be tuned to four sensitivity profiles—low norm, high score, low score, and high norm—by choosing among shifted/scaled Sigmoid, exponential, anti-exponential, and $\sinh$ links.
- Multi-class losses built from composite Softmax with nonlinear $Q$ provide both region sensitivity (via $q$) and ranking sensitivity (via $p_k$), which standard Softmax cannot supply because of shift invariance.
- Selective matching losses remain proper scoring rules: at the optimum $h(\hat s)=\mathbb{E}_P h(S)$, so selectivity reweights where model capacity is spent without breaking the expected-link calibration.
- The BUST result implies that under-specified models resolve ambiguity in favor of high-sensitivity regions, so low-score nuisance examples are discounted without enabling them to corrupt high-score predictions.
- The paper identifies high-score sensitivity as the useful profile for retrieval, ranking, distillation, and LLM alignment, and capped high-score sensitivity for dwell-time prediction.
Reading between the lines
- Editorial extension: because the construction defines losses by their gradient, it works even when $H$ has no closed form; learned or piecewise links optimized for downstream metrics are a natural avenue the paper does not explore.
- Editorial extension: the high-norm $\sinh$ link suggests a contrastive loss that suppresses small pairwise score differences and amplifies large ones; the paper mentions ordering robustness but proves no formal noise-robustness guarantee.
- Editorial extension: swapping standard Softmax for composite Softmax in an existing LLM alignment pipeline is a drop-in test of the framework; the paper states the intended application but provides no empirical comparison, so the practical size of the benefit remains open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a framework for designing 'selective' matching losses for scalar and multi-class prediction. A matching loss is defined as a Bregman divergence whose gradient is h(bs)-h(s), where h is a link function; the slope h'(z) is interpreted as local sensitivity. The authors show that shifted/scaled Sigmoid and sinh links produce different region-sensitivity profiles, and they introduce composite Softmax / composite Sigmoid constructions in which the link components factor as q(z_k) p_k(z). Theorems 2-4 give the factorization and sufficient convexity conditions, and extensive appendices catalogue valid and invalid link/scaling choices. The stated motivation is practical: selective losses are claimed to improve dwell-time prediction, retrieval, ranking, distillation, and LLM alignment by emphasizing important score regions and discounting unimportant ones.
Significance. If the construction holds up, the paper offers a principled design recipe for losses with prescribed region and ranking sensitivity, which is a genuinely useful contribution. The Bregman-divergence framing is elegant, and Theorem 2's factorization of the multi-class link into a unary scaling term q(z_k) and a probability term p_k(z) cleanly separates regional and ranking control. The convexity proofs in Theorems 3-4 are self-contained, and the appendices give a helpful catalogue of valid and invalid score transforms. However, the paper's central application claims go well beyond the evidence: there are no experiments, and the bridge from local loss sensitivity to trained-model improvement is an assumption that is not established. The value of the paper at present is as a theoretical/design contribution, not as a demonstrated source of practical gains.
major comments (3)
- [Abstract and Section 7] The abstract and conclusions claim 'substantial advantage' over traditional losses for dwell-time prediction, retrieval, ranking, distillation, and LLM alignment, but the manuscript contains no experiments or benchmarks on any of these tasks. Appendix F is a two-dimensional linear-model illustration, and the figures show loss surfaces and gradient curves, not trained-model performance on real or simulated tasks. This is load-bearing because the stated motivation is practical advantage rather than only mathematical expressiveness. I ask the authors to either provide at least one empirical demonstration with appropriate baselines on one or more of the claimed applications, or to rewrite the application claims as hypotheses and scope them explicitly to the proposed loss construction.
- [Section 2, Theorem 1 and Proposition 1] The transfer from loss sensitivity to trained-model behavior is not established. Theorem 1's BUST result, Eq. (10), computes the optimal displacement of a single collective score under a uniform distribution over S with one small biased interval; Proposition 1 only states that the minimizer of the expected matching loss is bs = h^{-1}(E_P h(S)). Neither result implies that a gradient-trained neural network with many features, non-uniform score distributions, and non-convex optimization will 'resolve underspecification' in favor of high-sensitivity regions, as asserted in the text around Eq. (10) and Section 7. This is a correctness-risk concern with the paper's central practical claim. Please provide a formal transfer theorem under stated assumptions, an empirical check, or an explicit statement that the improved behavior in real models is an assumption rather than a consequence of the theory.
- [Section 5, Eqs. (29)-(30)] The central notions of 'ranking sensitivity' and 'constellation shift invariance' are used informally and are never formally defined. For example, adding a constant to all scores preserves class rankings, but the loss in Eq. (29) is not invariant under this transformation when Q is nonlinear, so the exact intended invariance is unclear. Since the claimed advantage of composite Softmax over per-dimension losses rests on ranking sensitivity, the paper should define these terms precisely (for example, as invariance of the loss or its gradients under specified transformations of the score vector) and state which properties the proposed loss does and does not satisfy.
minor comments (4)
- [Section 2, Eq. (2)] The sentence 'The loss is the additional area covered by h(z) from s to bs' should be worded as signed area or as the area between h(z) and the horizontal line h(s) over the interval between s and bs; for bs < s the integrand h(z)-h(s) is non-positive, so the loss is positive but not an area in the ordinary unsigned sense.
- [Section 5] The terms 'ranking sensitivity' and 'constellation shift invariance' would benefit from formal definitions; at present the reader must infer them from Figures 4-5 and the surrounding text.
- [Appendix G.3] The list of invalid score transforms is useful, but the domain assumptions should be stated explicitly; several entries, such as f(z)=|z|^d, are invalid on certain domains but not on others.
- [Appendix E.2.3] The statement that a composite-Softmax CE loss is 'not guaranteed to be convex' would be easier to verify if a concrete non-convex example or a small counterexample plot were included.
Circularity Check
No material circularity: the loss constructions and sensitivity theorems are derived from stated definitions; the practical application claims are unverified but not circular.
full rationale
The paper's derivation chain is self-contained. Matching losses are defined via link functions and Bregman divergences in Equations (1)-(2), and the connection between link slope and local sensitivity follows from the Taylor expansion in Appendix B, not from any fitted quantity. The BUST/BLUST sensitivity model in Definition 1-2 and Theorem 1 is derived from the matching-loss gradient under an explicitly stated underspecification model; it is an analytical consequence of the loss, not an input that is later relabeled as a prediction. Composite Softmax is derived from first principles: Theorem 2 obtains h_k(z) = q(z_k)p_k(z) from the log-partition function H(z) = log(sum e^{Q(z_k)}) purely by the chain rule, and Theorems 3-4 give sufficient convexity conditions using standard composition arguments. No parameter is fitted to a subset of data and then reported as a prediction, and no claimed result is presupposed by its own definition. The self-citations that appear (e.g., SmeLU from Shamir et al. 2020 and Shamir and Lin 2022, shifted Sigmoid from Hristakeva 2010) are used as examples or prior art, not as load-bearing justification for the central construction. The main weakness is evidential rather than circular: the paper claims substantial advantage for dwell-time prediction, retrieval, ranking, distillation, and LLM alignment without experiments on those tasks, and the transfer from loss sensitivity to trained-model behavior rests on the idealized BUST model. Those are correctness or scope concerns, not circularity.
Assumptions & free parameters
assumptions (5)
- standard math Bregman divergences define matching losses, and the gradient of a matching loss is the difference of the link at predicted and observed scores.
- standard math A monotone non-decreasing link function gives a convex loss.
- standard math The composition of a convex non-decreasing function (Softplus) with a convex Q(z) is convex.
- domain assumption In the BUST model, the collective feature score w is uniformly distributed over S, and a small interval Su has an added bias d.
- standard math The optimal prediction under a matching loss satisfies h(bs) = E_P h(S), requiring h to be invertible on the relevant range.
Cite this review
Pith. "Pith review of Selective Matching Losses -- Not All Scores Are Created Equal." pith.science (2026). https://pith.science/paper/WXW22UAY
@misc{pith2026250604446,
author = {Pith},
title = {Pith review of: Selective Matching Losses -- Not All Scores Are Created Equal},
year = {2026},
howpublished = {\url{https://pith.science/paper/WXW22UAY}},
note = {Machine review of arXiv:2506.04446}
}
read the original abstract
Learning systems match predicted scores to observations over some domain. Often, it is critical to produce accurate predictions in some subset (or region) of the domain, yet less important to accurately predict in other regions. We construct selective matching loss functions by design of increasing link functions over score domains. A matching loss is an integral over the link. A link defines loss sensitivity as function of the score, emphasizing high slope high sensitivity regions over flat ones. Loss asymmetry drives a model and resolves its underspecification to predict better in high sensitivity regions where it is more important, and to distinguish between high and low importance regions. A large variety of selective scalar losses can be designed with scaled and shifted Sigmoid and hyperbolic sine links. Their properties, however, do not extend to multi-class. Applying them per dimension lacks ranking sensitivity that assigns importance according to class score ranking. Utilizing composite Softmax functions, we develop a framework for multidimensional selective losses. We overcome limitations of the standard Softmax function, that is good for classification, but not for distinction between adjacent scores. Selective losses have substantial advantage over traditional losses in applications with more important score regions, including dwell-time prediction, retrieval, ranking with either pointwise, contrastive pairwise, or listwise losses, distillation problems, and fine-tuning alignment of Large Language Models (LLMs).
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Better click quality on display ads improves the user and advertiser experience, 2015
Inside AdWords. Better click quality on display ads improves the user and advertiser experience, 2015
work page 2015
-
[2]
Robust bi-tempered logistic loss based on bregman divergences
Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems, 32, 2019
work page 2019
-
[3]
Locoprop: Enhancing backprop via local loss optimization
Ehsan Amid, Rohan Anil, and Manfred Warmuth. Locoprop: Enhancing backprop via local loss optimization. In International Conference on Artificial Intelligence and Statistics, pages 9626--9642. PMLR, 2022
work page 2022
-
[4]
Exponentially many local minima for single neurons
Peter Auer, Mark Herbster, and Manfred KK Warmuth. Exponentially many local minima for single neurons. Advances in neural information processing systems, 8, 1995
work page 1995
-
[5]
Improving post-click user engagement on native ads via survival analysis
Nicola Barbieri, Fabrizio Silvestri, and Mounia Lalmas. Improving post-click user engagement on native ads via survival analysis. In Proceedings of the 25th International Conference on World Wide Web, pages 761--770, 2016
work page 2016
-
[6]
Asymmetric loss functions for forecasting in criminal justice settings
Richard Berk. Asymmetric loss functions for forecasting in criminal justice settings. Journal of Quantitative Criminology, 27: 0 107--123, 2011
work page 2011
-
[7]
Stephen P Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004
work page 2004
-
[8]
L. M. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics, 7 0 (3): 0 200--217, 1967
work page 1967
Show all 37 references
-
[9]
Loss functions for binary class probability estimation and classification: Structure and applications
Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Working draft, November, 3: 0 13, 2005
2005
-
[10]
Learning to rank using gradient descent
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89--96, 2005
2005
-
[11]
From ranknet to lambdarank to lambdamart: An overview
Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 11 0 (23-581): 0 81, 2010
2010
-
[12]
Learning to rank: from pairwise approach to listwise approach
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pages 129--136, 2007
2007
-
[13]
An iterative row-action method for interval convex programming
Yair Censor and Arnold Lent. An iterative row-action method for interval convex programming. Journal of Optimization theory and Applications, 34 0 (3): 0 321--353, 1981
1981
-
[14]
Online learning to rank with top-k feedback
Sougata Chaudhuri and Ambuj Tewari. Online learning to rank with top-k feedback. Journal of Machine Learning Research, 18 0 (103): 0 1--50, 2017
2017
-
[15]
Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei
Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017
2017
-
[16]
Support-vector networks
Corinna Cortes. Support-vector networks. Machine Learning, 1995
1995
-
[17]
Kl divergence for machine learning
Dibya Ghosh. Kl divergence for machine learning. https://dibyaghosh.com/blog/probability/kldivergence.html, 2023
2023
-
[18]
Worst-case loss bounds for single neurons
David Helmbold, Jyrki Kivinen, and Manfred KK Warmuth. Worst-case loss bounds for single neurons. Advances in neural information processing systems, 8, 1995
1995
-
[19]
Distilling the knowledge in a neural network
Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015
2015 arXiv
-
[20]
Nonlinear Asymmetric Loss with Spectral Decomposition for Glucose Prediction
Maya H Hristakeva. Nonlinear Asymmetric Loss with Spectral Decomposition for Glucose Prediction. University of California, Santa Cruz, 2010
2010
-
[21]
Robust estimation of a location parameter
Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, pages 492--518. Springer, 1992
1992
-
[22]
a rvelin and Jaana Kek \
Kalervo J \"a rvelin and Jaana Kek \"a l \"a inen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20 0 (4): 0 422--446, 2002
2002
-
[23]
Relative loss bounds for multidimensional regression problems
Jyrki Kivinen and Manfred KK Warmuth. Relative loss bounds for multidimensional regression problems. Advances in neural information processing systems, 10, 1997
1997
-
[24]
Learning to rank for information retrieval
Tie-Yan Liu. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 3 0 (3): 0 225--331, 2009
2009
-
[25]
Learning to rank for information retrieval
Tie-Yan Liu. Learning to rank for information retrieval. Springer, 2011
2011
-
[26]
Manning, Prabhakar Raghavan, and Hinrich Sch \"u tze
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch \"u tze. Introduction to Information Retrieval. Cambridge University Press, 2008
2008
-
[27]
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010
2010
-
[28]
Supervised learning: No loss no cry
Richard Nock and Aditya Menon. Supervised learning: No loss no cry. In International Conference on Machine Learning, pages 7370--7380. PMLR, 2020
2020
-
[29]
Direct preference optimization: Your language model is secretly a reward model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023
2023
-
[30]
On ndcg consistency of listwise ranking methods
Pradeep Ravikumar, Ambuj Tewari, and Eunho Yang. On ndcg consistency of listwise ranking methods. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 618--626. JMLR Workshop and Conference Proceedings, 2011
2011
-
[31]
Composite binary losses
Mark D Reid and Robert C Williamson. Composite binary losses. The Journal of Machine Learning Research, 11: 0 2387--2422, 2010
2010
-
[32]
Information, divergence and risk for binary experiments
Mark D Reid and Robert C Williamson. Information, divergence and risk for binary experiments. Journal of Machine Learning Research, 12 0 (3), 2011
2011
-
[33]
Real world large scale recommendation systems reproducibility and smooth activations
Gil I Shamir and Dong Lin. Real world large scale recommendation systems reproducibility and smooth activations. arXiv preprint arXiv:2202.06499, 2022
2022 arXiv
-
[34]
Smooth activations and reproducibility in deep networks
Gil I Shamir, Dong Lin, and Lorenzo Coviello. Smooth activations and reproducibility in deep networks. arXiv preprint arXiv:2010.09931, 2020
2010 arXiv
-
[35]
Learning to approximate a bregman divergence
Ali Siahkamari, Xide Xia, Venkatesh Saligrama, David Casta \ n \'o n, and Brian Kulis. Learning to approximate a bregman divergence. Advances in Neural Information Processing Systems, 33: 0 3603--3612, 2020
2020
-
[36]
Beyond clicks: dwell time for personalization
Xing Yi, Liangjie Hong, Erheng Zhong, Nanthan Nan Liu, and Suju Rajan. Beyond clicks: dwell time for personalization. In Proceedings of the 8th ACM Conference on Recommender systems, pages 113--120, 2014
2014
-
[37]
Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2020
1909 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.