Pith. sign in

REVIEW 3 major objections 3 minor 44 references

Loss Functions and Operators Generated by f-Divergences

T0 review · 3 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper turns cross-entropy into a family of convex f-divergence losses, each with its own softargmax operator, and shows the α=1.5 member beats cross-entropy on image classification, language-model pretraining, fine-tuning, and…

desk verdict Solid theory and a reusable bisection algorithm; the empirical case is strong for SFT and distillation but the pretraining gap and α-selection don't fully support the cross-task headline. read the letter →

arxiv 2501.18537 v2 pith:2UXNT4UC submitted 2025-01-30 cs.LG stat.ML

Vincent Roulet , Tianlin Liu , Nino Vieillard , Michael E. Sander , Mathieu Blondel This is my paper · ORCID
classification cs.LGstat.ML MSC 90C2594A1768T07
keywords f-divergencesFenchel-Younglossesf-softargmaxconvexlossfunctionsalpha-divergenceTsallisentropynext-tokenpredictioncross-entropygeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cross-entropy, the default loss for image classification and next-token prediction, is a Fenchel–Young loss built from KL divergence and the softargmax operator. This paper proposes replacing KL with any f-divergence and allowing a non-uniform reference measure $q$, yielding a family of convex losses each paired with its own f-softargmax operator (a softmax-style map from logits to probabilities), recovering the logistic, sparsemax, and entmax losses as special cases and creating new ones. It proves every such loss can be computed by solving a single one-dimensional root equation via bisection, an algorithm that parallelizes cleanly on GPUs and TPUs. Empirically, the loss generated by the $\alpha$-divergence at $\alpha=1.5$ outperforms cross-entropy on ImageNet (77.56 vs 76.87), 1.2B-parameter language-model pretraining (48.70 vs 48.66), supervised fine-tuning (ROUGE-2 14.27 vs 9.77), and distillation (17.43 vs 16.64), making f-divergence losses a practical alternative to cross-entropy in standard training pipelines.

What carries the argument

The load-bearing mechanism is the Fenchel–Young loss with f-divergence regularization, $\ell_f(\theta,y;q)=\operatorname{softmax}_f(\theta;q)+D_f(y,q)-\langle\theta,y\rangle$, together with Proposition 1, which reduces the defining maximization over the $k$-dimensional simplex to the one-dimensional root equation $\sum_j q_j f'_\ast(\max\{\theta_j-\tau, f'(0)\})=1$. The reduction uses Fenchel duality: enforcing the simplex constraint $\langle p,\mathbf{1}\rangle=1$ with a Lagrange multiplier $\tau$ turns the constrained concave maximization into an unconstrained problem that needs only the convex conjugate $f^\ast$ and its derivative $f'_\ast$ of the chosen divergence generator $f$. Because $f'$ is increasing, the root is unique and bracketed by $\tau_{\min}=\theta_{j^\star}-f'(1/q_{j^\star})$ and $\tau_{\max}=\theta_{j^\star}-f'((\sum_j q_j)^{-1})$, so bisection converges with error $(\tau_{\max}-\tau_{\min})/2^t$ and parallelizes over the class or token dimension; gradients then flow through Danskin's theorem and through implicit differentiation of the scalar root.

What would settle it

Run the $\alpha$-sweep at $\alpha\in\{1.0,1.25,1.5,1.75,2.0\}$ on the 1.2B-parameter decoder-only pretraining pipeline on C4 and on the XSum SFT task with several random seeds: if next-token accuracy does not peak near $\alpha=1.5$, or if the 0.04-point pretraining gain over KL at $\alpha=1.5$ disappears within seed noise, the cross-task transfer claim fails. Independently, for random logits and a random strictly convex $f$ with closed-form conjugate, compare the bisection solution of the root equation with a direct numerical solution of the simplex maximization defining the f-softargmax, which would expose any error in the Fenchel-duality reduction.

Watch

Extended reading notes

Core claim

The paper's central claim is that the pair $\operatorname{softmax}_f(\theta;q)=\max_{p\in\Delta_k}\langle p,\theta\rangle - D_f(p,q)$ and its argmax $\operatorname{softargmax}_f(\theta;q)$ form a valid generalization of softmax and softargmax for any strictly convex differentiable $f$ with $(0,\infty)\subseteq\operatorname{dom} f'$, and that the Fenchel–Young loss $\ell_f(\theta,y;q)=\operatorname{softmax}_f(\theta;q)+D_f(y,q)-\langle\theta,y\rangle$, where $D_f(p,q)=\sum_j f(p_j/q_j)q_j$ is the f-divergence, is convex in $\theta$ with gradient $\operatorname{softargmax}_f(\theta;q)-y$. Proposition 1 reduces both operators to one-dimensional root finding: with $\tau^\star$ the unique solution of $\sum_j q_j f'_\ast(\max\{\theta_j-\tau, f'(0)\})=1$ on the bracket $[\tau_{\min},\tau_{\max}]$, the outputs are $\operatorname{softargmax}_f(\theta;q)_j = q_j f'_\ast(\max\{\theta_j-\tau^\star, f'(0)\})$ and $\operatorname{softmax}_f(\theta;q)=\tau^\star+\sum_j q_j f^\ast(\max\{\theta_j-\tau^\star, f'(0)\})$. This unifies KL (logistic loss), Chi-square (sparsemax), and α-divergence/Tsallis (entmax) as instances of one construction with non-uniform class priors, and the bisection solver used in practice halves the error each iteration. The paper further reports that decoding with the classical softargmax instead of the matching f-softargmax changes downstream results only by sampling noise, so the observed gains come from the training loss itself.

Load-bearing premise

The paper's headline empirical claim assumes that $\alpha=1.5$, chosen by sweeping validation accuracy on ImageNet, transfers to language-modeling tasks without retuning; the theory also assumes a strictly positive reference measure $q$, which the paper's notation $q\in\mathbb{R}_+^k$ does not explicitly guarantee.

Editorial extensions

If this is right

  • The α=1.5 divergence loss, equivalent to Tsallis entropy regularization with unit reference measure, becomes a drop-in replacement for cross-entropy in classification and next-token prediction, where the paper reports consistent gains over KL across four settings.
  • Any f-divergence with a closed-form conjugate can be turned into a trainable convex loss plus an inference operator, giving a recipe for designing new losses, including sparse-output ones such as Chi-square and α>1, rather than hand-crafting each loss.
  • Because decoding with the standard softargmax performs about as well as with the matching f-softargmax, models fine-tuned with f-divergence losses can be served with unchanged inference code.
  • Non-uniform reference measures q let prior class weights be folded into the loss, and q can be learned by gradient descent, though the paper reports that learning q did not beat q=1 in its NanoDO experiments.
  • The bisection algorithm's overhead is comparable to a standard softmax (the paper profiles ResNet18 runtimes), so the generalized losses can replace cross-entropy in standard training pipelines without a runtime penalty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own tables show the α=1.5 wins are unequal in size — 0.69 accuracy points on ImageNet but only 0.04 on pretraining — so a per-task α-sweep would settle whether 1.5 is a universal optimum or an ImageNet-tuned one; this is a natural follow-up the authors do not run.
  • Because Proposition 1 computes the f-softargmax from a scalar root equation, the same computation applies wherever KL-regularized policies appear, such as preference optimization, where the f-softargmax is exactly the optimal policy; testing f-divergence losses there is a next step the paper leaves open.
  • The sparse outputs of Chi-square and α>1 divergences suggest using f-softargmax as a sparse attention mechanism; the paper derives implicit differentiation through the operator for exactly that use but does not measure its effect on attention quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes to generate convex loss functions and associated prediction operators by regularizing with f-divergences. For a reference measure q, it defines f-softmax and f-softargmax through the variational problem max_p <p,theta> - D_f(p,q), and the corresponding Fenchel-Young loss l_f(theta,y;q)=softmax_f(theta;q)+D_f(y,q)-<theta,y>. Proposition 1 reduces the computation of these operators to a one-dimensional root-finding problem in tau, and Algorithm 1 solves it by bisection with a guaranteed exponential error decay. The framework recovers the logistic loss (KL divergence) and sparsemax/entmax losses (Chi-square/alpha-divergence) and extends them to non-uniform reference measures. Experiments compare these losses on ImageNet classification, language model pretraining on C4, SFT and distillation on XSum, and report that the alpha-divergence with alpha=1.5 outperforms the KL-based cross-entropy loss.

Significance. The theoretical contribution is solid and useful: it connects f-divergence regularization to Fenchel-Young losses, generalizes existing softmax/sparsemax/entmax constructions, and the proof of Proposition 1 via Fenchel duality is a genuine extension of prior work, with careful treatment of the domain of f'. The bisection algorithm is simple, parallelizable, and comes with a clean convergence rate, and the paper provides computational-cost experiments supporting its practical overhead. The empirical study is commendably broad, including 5-seed statistics for ImageNet, SFT, and distillation, and a useful ablation separating the effect of the training loss from the decoding operator. However, the headline cross-task claim for alpha=1.5 is weakened by the fact that this value is selected on ImageNet validation and then transferred to language tasks without per-task validation, and by the pretraining result being within noise. If the authors address the selection issue and the pretraining error bars, the contribution is a strong candidate for publication.

major comments (3)
  1. [Section 4.1, Figure 5, Section 4.3] The value alpha=1.5 is selected by sweeping 11 values on ImageNet validation accuracy (Figure 5) and then reused without retuning for language model pretraining, SFT, and distillation. The abstract and Section 4.3 claim that this loss 'performs well across several tasks,' but the cross-task evidence rests on a single validation curve. If the peak near alpha=1.5 is dataset-specific or a selection artifact, the headline generalization collapses. Please provide per-task sensitivity to alpha (e.g., a sweep over alpha on a held-out validation split for at least one language task) or an explicit pre-registered choice of alpha; at minimum, soften the cross-task claim to reflect that alpha was tuned only on ImageNet.
  2. [Table 3, Appendix A.2] The pretraining result in Table 3 reports 48.70% next-token accuracy for alpha-divergence (alpha=1.5) versus 48.66% for KL, a 0.04 percentage-point difference, with no seed statistics or error bars. With 250B tokens and a single run, this gap is indistinguishable from noise. The text in Section 4.2 says alpha=1.5 'slightly outperformed' KL, but the evidence does not support that conclusion. Please report multiple seeds and standard deviations for the pretraining experiments, or remove this result from the empirical claims and rely on the SFT/distillation results, which have 5-seed statistics and larger gaps.
  3. [Section 3.1, Proposition 1, Algorithm 1] The paper repeatedly defines q in R_+^k, including in Equations (8), (12)-(14), and Algorithm 1, but Proposition 1 and Algorithm 1 require q>0: they evaluate f'(1/q_j) and f'(1/(sum_j q_j)), which are undefined when any q_j=0 or when all q_j=0. The statement of Proposition 1 should explicitly assume q in R_{++}^k, or the zero-boundary case should be handled separately. This is a real domain issue, since the text elsewhere says q in R_+^k may contain class priors with zero entries.
minor comments (3)
  1. [Section 3.2, 'Effective domain'] The text says some f-entropies are only well-defined on the relative interior of the simplex 'if lim_{u->0} f(u) = -infinity' and gives reverse KL and Jeffreys as examples. For reverse KL, f(u)=-log u, and for Jeffreys, f(u)=(u-1) log u; in both cases the limit at 0 is +infinity, not -infinity. Please correct the sign.
  2. [Section 2.3, Eq. (6)] For a differentiable and strictly convex Omega, the Fenchel-Young loss in Eq. (5) equals the Bregman divergence B_Omega(y, softargmax_Omega(theta)), not merely a lower bound. If a lower bound is intended for a non-differentiable or non-strict case, please clarify the definition of B_Omega; otherwise the inequality should be an equality.
  3. [Figure 5, Table 4] Figure 5 shows the alpha sweep on ImageNet with no error bars, while Table 4 gives 5-seed statistics only for the selected alpha=1.5 and the other divergences, not for the other values of alpha in the sweep. Adding error bars to Figure 5, or at least reporting how many seeds the sweep used, would better justify the statement that alpha=1.5 is 'optimal' among the 11 values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical derivation is self-contained and the empirical α=1.5 selection is a tuning/transfer concern, not an equation-level circularity.

full rationale

The paper's theoretical chain is self-contained. Proposition 1 is proven in Appendix C.4 from the variational definition of f-softmax (Eq. 9) via Fenchel duality (Lemmas 1 and 2), giving the root equation (14) and formulas (12)-(13). The construction of the loss in Eq. (11) as a Fenchel-Young loss is explicit, and its convexity and gradient follow from standard Fenchel-Young properties rather than from assuming the conclusion. Citations to Blondel et al. (2020) are to an independent published framework; the paper explicitly states that its proof technique differs (Fenchel duality instead of Lagrange duality), so the self-citation is not load-bearing. The recovery of logistic, sparsemax, and entmax losses is acknowledged as a special case of the framework, not presented as an independent prediction. On the empirical side, the choice α=1.5 is selected from the ImageNet sweep in Figure 5 and then transferred to language tasks. This is a hyperparameter-selection and external-validity concern: the ImageNet number for α=1.5 is the best of a validation sweep, and the small pretraining gain over KL (48.70 vs 48.66) lacks error bars. However, the language-model, SFT, and distillation results are new data, not statistically forced by the ImageNet fit, so this does not constitute circularity under the standards of quoted-equation reduction. No step in the manuscript reduces a claimed result to its own input by construction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The math rests on standard convex analysis and the cited Fenchel-Young framework; no new particles, forces, or named entities are introduced. The main hand-set quantity is α=1.5, chosen by validation.

free parameters (1)
  • α (alpha) in α-divergence = 1.5
    Selected by sweeping 11 values of α∈[1,2] on ImageNet validation accuracy (Figure 5), then used for LM pretraining, SFT, and distillation. This selection is not corrected for multiple comparisons.
assumptions (3)
  • standard math Fenchel-Young losses with strictly convex Ω give convex losses with gradient softargmax_Ω(θ)-y.
    Invoked in Section 2.3 and used to define ℓ_f; theorem from Blondel et al. 2020.
  • standard math Strong duality for convex separable problems over the simplex (Lemma 1).
    Central to the proof of Proposition 1 in Appendix C.5; requires Slater-type strict feasibility, satisfied when (0,+∞)⊆dom Ω_j.
  • domain assumption All listed f are strictly convex and differentiable on (0,+∞), f'(0) is defined (possibly -∞), and q>0.
    Proposition 1 and Algorithm 1 require q_j>0 and finite f'(1/q_j); the main text writes q∈R_+^k, which could include zero entries, but no zero-q case is handled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Loss Functions and Operators Generated by f-Divergences." pith.science (2026). https://pith.science/paper/2UXNT4UC

@misc{pith2026250118537,
  author       = {Pith},
  title        = {Pith review of: Loss Functions and Operators Generated by f-Divergences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UXNT4UC}},
  note         = {Machine review of arXiv:2501.18537}
}
abstract

The logistic loss (a.k.a. cross-entropy loss) is one of the most popular loss functions used for multiclass classification. It is also the loss function of choice for next-token prediction in language modeling. It is associated with the Kullback--Leibler (KL) divergence and the softargmax operator. In this work, we propose to construct new convex loss functions based on $f$-divergences. Our loss functions generalize the logistic loss in two directions: i) by replacing the KL divergence with $f$-divergences and ii) by allowing non-uniform reference measures. We instantiate our framework for numerous $f$-divergences, recovering existing losses and creating new ones. By analogy with the logistic loss, the loss function generated by an $f$-divergence is associated with an operator, that we dub $f$-softargmax. We derive a novel parallelizable bisection algorithm for computing the $f$-softargmax associated with any $f$-divergence. On the empirical side, one of the goals of this paper is to determine the effectiveness of loss functions beyond the classical cross-entropy in a language model setting, including on pre-training, post-training (SFT) and distillation. We show that the loss function generated by the $\alpha$-divergence (which is equivalent to Tsallis $\alpha$-negentropy in the case of unit reference measures) with $\alpha=1.5$ performs well across several tasks.

Figures

Figures reproduced from arXiv: 2501.18537 by the authors.

Figure 1
Figure 1. Illustration of f-entropies −Ωf (p) for p = (1 − π, π) and varying π ∈ [0, 1]. We add a constant f(0) to ensure non￾negativity of the f-entropies. 3.3. f-softmax and f-softargmax Overloading the notation, we define the f-softmax as softmaxf (θ; q) := max p∈△k ⟨p, θ⟩ − Df (p, q) ∈ R (9) and the f-softargmax as softargmaxf (θ; q) := argmax p∈△k ⟨p, θ⟩ − Df (p, q) ∈ △k . (10) Compared to a classical softmax and softarg… view at source ↗
Figure 2
Figure 2. Illustration of (p1, p2, p3) = softargmaxf (θ1, θ2, 0; q) when varying θ1, θ2 ∈ R for three possible divergences and with q = (1, 1, 1). More illustrations are given in Appendix B. and the f-sigmoid sigmoidf (θ; q) := argmax π∈[0,1] πθ−Df ((1−π, π), q) ∈ [0, 1]. The effect of the prior class weights q ∈ R 2 + on the shape of the sigmoid is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. shows results for SFT (unhatched) and distilla￾js rcs kl cs alpha1.5 0 5 10 15 Rouge-2 Scores SFT Distillation [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Decoding with the classical (KL-based) softargmax per￾forms similarly to decoding with f-softargmax associated with the loss used for training. This suggests that the choice of f-divergence loss during training, not the decoding method, primarily drives performance dif…
Figure 5
Figure 5. Figure 5: Validation accuracy on ImageNet when using the α-divergence generated loss, with α ∈ [1., 2.]. Accuracy is maximized near α = 1.5. A.2. Pretraining NanoDO models The 1.2B NanoDO model’s architecture and hyperparameters follow from the setup in Wortsman et al. (2024). T…
Figure 6
Figure 6. Figure 6: Error for computing the root in Proposition 1 using our bisection based Algorithm 1 as a function of the number of iterations. We use the output with 30 iterations as a proxy for the true root. We compare the measured error (dark purple) with the theoretical 2 −t error…
Figure 7
Figure 7. Figure 7: Profiled time for an input of shape (b, 224, 224, 3) to be processed by a ResNet18 followed by f-softargmax corresponding the 1.5-divergence (dark purple) or the standard Kullback–Leibler divergence (light purple, in which case we use the standard JAX implementation) f…
Figure 8
Figure 8. Figure 8: Illustration of θ 7→ sigmoidf (θ; (1 − q1, q1)) for q1 ∈ {0.1, 0.25, 0.5, 0.75, 0.9} and for f(u) = u log u, the generating function of the KL divergence. We see that sigmoidf (θ; (1 − q1, q1)) at θ = 0 is equal to q1. Intuitively, if a model is uncertain and produces …
Figure 9
Figure 9. Figure 9: Illustration of (p1, p2, p3) = softargmaxf (θ1, θ2, 0; q) when varying θ1, θ2 ∈ R for diverse f-divergences and with q = (1, 1, 1). 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Illustration of (p1, p2, p3) = softargmaxf (θ1, θ2, 0; q) when varying θ1, θ2 ∈ R for diverse f-divergences and priors q. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 35 canonical work pages

  1. [1]

    R., Geist, M., and Bachem, O

    Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: learning from self-generated mistakes. In Proceedings of the International Conference on Learning Representations (ICLR), 2024

    work page 2024

  2. [2]

    Ali, S. M. and Silvey, S. D. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28 0 (1): 0 131--142, 1966

    work page 1966

  3. [3]

    Robust semi-supervised learning via f-divergence and -r \'e nyi divergence

    Aminian, G., Bagheri, A., JafariNodeh, M., Karimian, R., and Yassaee, M.-H. Robust semi-supervised learning via f-divergence and -r \'e nyi divergence. In Proceedings of the IEEE International Symposium on Information Theory (ISIT), pp.\ 1842--1847. IEEE, 2024

    work page 2024

  4. [4]

    Bregman divergence of alpha-divergence, 2017

    Belousov, B. Bregman divergence of alpha-divergence, 2017. URL http://www.boris-belousov.net/2017/04/16/bregman-divergence/

    work page 2017

  5. [5]

    and Peters, J

    Belousov, B. and Peters, J. f -divergence constrained policy improvement. arXiv preprint arXiv:1801.00056, 2017

    arXiv 2017

  6. [6]

    f-entropies, probability of error, and feature selection

    Ben-Bassat, M. f-entropies, probability of error, and feature selection. Information and Control, 39 0 (3): 0 227--242, 1978

    work page 1978

  7. [7]

    and Roulet, V

    Blondel, M. and Roulet, V. The elements of differentiable programming. arXiv preprint arXiv:2403.14606, 2024

    arXiv 2024

  8. [8]

    F., and Niculae, V

    Blondel, M., Martins, A. F., and Niculae, V. Learning with Fenchel--Young losses. Journal of Machine Learning Research, 21 0 (35): 0 1--69, 2020

    work page 2020

Show all 44 references
  1. [9]

    Efficient and modular implicit differentiation

    Blondel, M., Berthet, Q., Cuturi, M., Frostig, R., Hoyer, S., Llinares-Lopez, F., Pedregosa, F., and Vert, J.-P. Efficient and modular implicit differentiation. In Advances in Neural Information Processing Systems, 2022

    2022

  2. [10]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  3. [11]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. PaLM : Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  4. [12]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017

    2017

  5. [13]

    and Amari, S.-i

    Cichocki, A. and Amari, S.-i. Families of alpha- beta- and gamma-divergences: Flexible and robust measures of similarities. Entropy, 12 0 (6): 0 1532--1568, 2010

    2010

  6. [14]

    On information-type measure of difference of probability distributions and indirect observations

    Csisz \'a r, I. On information-type measure of difference of probability distributions and indirect observations. Studia Sci. Math. Hungar., 2: 0 299--318, 1967

    1967

  7. [15]

    Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Riquelme Ruiz, C., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., Steenkist...

    2023

  8. [16]

    Multiclass classification, information, divergence and surrogate risk

    Duchi, J., Khosravi, K., and Ruan, F. Multiclass classification, information, divergence and surrogate risk . The Annals of Statistics, 46 0 (6B): 0 3246 -- 3275, 2018

    2018

  9. [17]

    Universal uncertainty relations

    Friedland, S., Gheorghiu, V., and Gour, G. Universal uncertainty relations. Physical review letters, 111 0 (23), 2013

    2013

  10. [18]

    and Williamson, R

    Garcia-Garcia, D. and Williamson, R. C. Divergences and risks for multiclass experiments. In Proceedings of the Annual Conference on Learning Theory (COLT), pp.\ 28--1. JMLR Workshop and Conference Proceedings, 2012

    2012

  11. [19]

    Aligning language models with preferences through f -divergence minimization

    Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with preferences through f -divergence minimization. In Proceedings of the International Conference on Machine Learning (ICML), 2023

    2023

  12. [20]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 770--778, 2016

    2016

  13. [21]

    Distilling the knowledge in a neural network

    Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  14. [22]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Ra...

    2022

  15. [23]

    Krantz, S. G. and Parks, H. R. The implicit function theorem: history, theory, and applications. Springer Science & Business Media, 2002

    2002

  16. [24]

    and Turner, R

    Li, Y. and Turner, R. E. R \'e nyi divergence variational inference. Advances in neural information processing systems, 2016

    2016

  17. [25]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp.\ 74--81, 2004

    2004

  18. [26]

    J., Novak, R., Lee, J., Wortsman, M., Xiao, L., Everett, K., Alemi, A

    Liu, P. J., Novak, R., Lee, J., Wortsman, M., Xiao, L., Everett, K., Alemi, A. A., Kurzeja, M., Marcenac, P., Gur, I., Kornblith, S., Xu, K., Elsayed, G., Fischer, I., Pennington, J., Adlam, B., and Dickstein, J.-S. Nanodo: A minimal transformer decoder-only language model imp...

    2024

  19. [27]

    and Astudillo, R

    Martins, A. and Astudillo, R. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of the International Conference on Machine Learning (ICML), 2016

    2016

  20. [28]

    B., and Lapata, M

    Narayan, S., Cohen, S. B., and Lapata, M. Don`t give me the details, just the summary! Topic -aware convolutional neural networks for extreme summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 1797--1807. Associati...

    2018

  21. [29]

    J., and Jordan, M

    Nguyen, X., Wainwright, M. J., and Jordan, M. I. On surrogate loss functions and f -divergences . The Annals of Statistics, 37 0 (2): 0 876 -- 904, 2009

    2009

  22. [30]

    and Tonello, A

    Novello, N. and Tonello, A. M. f-divergence based classification: beyond the use of cross-entropy. In Proceedings of the International Conference on Machine Learning (ICML), 2024

    2024

  23. [31]

    f-gan: Training generative neural samplers using variational divergence minimization

    Nowozin, S., Cseke, B., and Tomioka, R. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 2016

    2016

  24. [32]

    Peters, B., Niculae, V., and Martins, A. F. Sparse sequence-to-sequence models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  25. [33]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  26. [34]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2024

    2024

  27. [35]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020

    2020

  28. [36]

    On measures of entropy and information

    R \'e nyi, A. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pp.\ 547--562. University of California Press, 1961

    1961

  29. [37]

    C., and Fei-Fei, L

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115: 0 211--252, 2015

    2015

  30. [38]

    Entropies from f-divergences

    Sbert, M., Chen, M., Poch, J., Feixas, M., and Chen, S. Entropies from f-divergences. In Proceedings of the International Conference on Modeling Decisions for Artificial Intelligence (MDAI), pp.\ 47--60. Springer, 2024

    2024

  31. [39]

    and Sazim, S

    Sharma, G. and Sazim, S. A geometric way to find the measures of uncertainty from statistical divergences for discrete and finite probability distributions. arXiv preprint arXiv:2106.14874, 2021

    2021 arXiv

  32. [40]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  33. [41]

    Moreau--Yosida f -divergences

    Terj \'e k, D. Moreau--Yosida f -divergences. In Proceedings of the International Conference on Machine Learning (ICML), 2021

    2021

  34. [42]

    Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints

    Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints. In Proceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  35. [43]

    J., Xiao, L., Everett, K

    Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilitie...

    2024

  36. [44]

    Label distributionally robust losses for multi-class classification: Consistency, robustness and adaptivity

    Zhu, D., Ying, Y., and Yang, T. Label distributionally robust losses for multi-class classification: Consistency, robustness and adaptivity. In Proceedings of the International Conference on Machine Learning (ICML), 2023

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.