Pith. sign in

REVIEW 4 major objections 5 minor 37 references

Harmonic Loss Trains Interpretable AI Models

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that replacing cross-entropy with a distance-based, scale-invariant harmonic loss makes trained weights readable as class centers and empirically cuts grokking while improving data efficiency.

desk verdict A clean loss-function idea with strong empirical moments on algorithmic tasks, but the interpretability theorem is mis-derived and the headline claims outrun the evidence. read the letter →

arxiv 2502.01628 v2 pith:SDCWQGR3 submitted 2025-02-03 cs.LG

David D. Baek , Ziming Liu , Riya Tyagi , Max Tegmark This is my paper · ORCID
classification cs.LG
keywords harmoniclossinterpretabilitycross-entropysoftmaxgrokkingdataefficiencyclasscenterHarMax
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single change to the loss function can make neural networks more interpretable and more data-efficient. It replaces cross-entropy's SoftMax normalization and dot-product logits with harmonic loss, which measures how close a representation is to each class weight by Euclidean distance and turns those distances into probabilities with a scale-invariant HarMax function. Because the loss reaches its minimum at finite weight vectors rather than at infinitely large logits, the authors argue that each class weight converges to a readable class center. Their experiments across algorithmic tasks, MNIST, ImageNet, and a GPT-2 language model report less grokking, faster generalization from less data, and cleaner geometric structure in embeddings. If this holds, harmonic loss is a practical drop-in replacement that buys interpretability and reliability without sacrificing accuracy.

What carries the argument

The central object is the harmonic layer, which replaces the final linear-plus-SoftMax layer of a classifier. Logits are Euclidean distances $d_i = \|\mathbf{w}_i - \mathbf{x}\|_2$ between the penultimate representation $\mathbf{x}$ and the class weight vectors $\mathbf{w}_i$, and probabilities are $p_i = (1/d_i^n)/\sum_j (1/d_j^n)$, the HarMax function. Two properties carry the argument: scale invariance, because multiplying all distances by a constant leaves $p_i$ unchanged, and a finite convergence point, because $p_i \to 1$ can be achieved with $\mathbf{w}_i = \mathbf{x}$ instead of infinitely large logits. The finite convergence point is what lets weights become class centers, and the paper traces its faster convergence, lower data requirements, and reduced grokking to that property.

What would settle it

Train a harmonic classifier on two Gaussian classes with small overlap and inspect the converged weight vectors. If a weight vector deviates materially from the mean of its class's training examples, or if the off-class terms in the stationarity equation are not small compared with the on-class terms, the class-center interpretability claim fails; a concrete comparison is $\|\mathbf{w}_k - \frac{1}{N_k}\sum_{i:y_i=k}\mathbf{x}_i\|$ for harmonic versus cross-entropy models.

Watch

Extended reading notes

Core claim

The central claim is that harmonic loss changes where a trained network's final weights land, and that this location is what makes the model interpretable. Under cross-entropy, driving the probability of the correct class to one requires logits to diverge to infinity, so weights grow without bound and carry no fixed meaning. Under harmonic loss, the same goal is met by driving the Euclidean distance between the representation and the class weight to zero, so the optimum sits at finite weights. At a critical point, each weight vector $\mathbf{w}_k$ is a convex combination of training examples, dominated by examples of class $k$: $\mathbf{w}_k = \sum_{i:y_i=k} \alpha_i \mathbf{x}_i$ with $\alpha_i \ge 0$ and $\sum \alpha_i = 1$. The paper reports that this finite, example-weighted geometry yields faster convergence, reduced grokking, greater data efficiency, and more compact, interpretable representations in both small models and GPT-2.

Load-bearing premise

The weakest link is the claim that, at the converged weights, examples from other classes contribute negligibly to each weight vector; if classes overlap enough that off-class examples matter, the learned weights are not simply class centers and the interpretability story weakens accordingly.

Editorial extensions

If this is right

  • In standard classification pipelines, swapping the final SoftMax layer for a harmonic layer gives weight vectors that can be read as class prototypes, since each weight is a weighted average of that class's training examples.
  • On algorithmic tasks such as modular addition and permutation composition, harmonic models reach test accuracy much closer in time to train accuracy, meaning the grokking delay is substantially shortened.
  • Data-efficiency plots show harmonic models generalizing from smaller training fractions, suggesting the loss lowers the amount of labeled data needed for a given accuracy in the tested settings.
  • A GPT-2 pretrained with harmonic loss reaches a comparable validation loss while producing token embeddings that form more rectangular parallelograms on function-vector tasks, supporting the interpretability claim at language-model scale.
  • The recommended harmonic exponent scales as $n \sim \sqrt{D}$ with embedding dimension, so the construction is designed to remain well-behaved as model width grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If class-center convergence proves robust, harmonic loss offers a natural confidence score for out-of-distribution detection: the distance from a new input to its nearest class weight, which is not available from cross-entropy's diverging weights.
  • The authors' observation that harmonic GPT-2 has both more perfect parallelograms and a heavier tail in intermediate layers suggests an untested question: whether the heavy tail degrades or helps downstream tasks, and whether a different harmonic exponent can control it.
  • A testable extension is to combine harmonic loss with weight decay or normalization in low-data vision benchmarks beyond MNIST, where the finite convergence point may interact differently than cross-entropy's unbounded weights.
  • Because harmonic loss only needs distances to vanish, it may be suited to metric-learning and few-shot settings where class anchors are known a priori; this is not explored in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes replacing the softmax/cross-entropy classification head with a scale-invariant 'HarMax' over Euclidean distances to class prototypes, calling this the harmonic loss. It claims that models trained with this loss are more interpretable, more data-efficient, faster to converge, and less prone to grokking. The evidence includes toy problems, five algorithmic tasks (in-context learning, modular addition, equivalence classes, genealogy, permutation groups), an MNIST linear model, and a GPT-2 pretraining comparison. Theoretical support is provided in Appendix G as three theorems on finite convergence, PAC-Bayes bounds, and class-center interpretability.

Significance. If the empirical results hold, harmonic loss would be a simple and potentially useful alternative to cross-entropy for classification and language modeling, especially in low-data or interpretability-critical settings. The experimental breadth is a real strength, and the modular-addition and grokking results are suggestive. However, the theoretical guarantee behind the central interpretability claim is currently invalid, and two empirical pillars, the equivalence-class explained-variance results and the GPT-2 comparison, have serious problems. The contribution is promising but not yet established at the level the abstract claims.

major comments (4)
  1. [Appendix G, Theorem 3] The stationarity calculation in the proof of Theorem 3 is incorrect. For a sample i with y_i=k, the derivative of the per-sample harmonic loss with respect to w_k is proportional to (1-p_i^k)(w_k-x_i)/||x_i-w_k||^2, and for y_i≠k it is proportional to -p_i^k(w_k-x_i)/||x_i-w_k||^2, where p_i^k is the harmonic probability of class k. The proof drops the factors (1-p_i^k) and p_i^k and uses 1/||x_i-w_k||^2 for both contributions. This is not a harmless approximation: near a perfectly classifying solution, (1-p_i^k)→0 and p_i^k→0, so the stationarity equation becomes vacuous and does not constrain w_k to be a convex combination of class-k points. The assertion that off-class repulsions 'tend to balance out on average' is not proved. Consequently, Theorem 3 does not support the abstract's statement that the finite convergence point 'can be interpreted as a class center.'
  2. [Section 3.2 / Figure 3(a) / Appendix A] The compression-based interpretability metric, cumulative explained variance, does not support the claimed advantage on Equivalence Classes: the harmonic MLP has EV 64–92% across the 20 seeds in Appendix A while the standard MLP has EV 86–98%, and the harmonic transformer is also lower than the standard transformer. This contradicts the blanket caption claim in Figure 3(a) that 'harmonic representations are more compact than standard counterparts.' The informal statement that standard representations are 'elongated' in Section 3.2 is not a quantitative resolution of this contradiction. The compactness/interpretability claim should be restricted to tasks where the metric is favorable, or the metric should be revised.
  3. [Section 5 / Figure 5] The GPT-2 comparison is confounded by different training schedules: the harmonic model uses maximum learning rate 6e-3 with 1k warmup while the standard model uses 6e-4 with 2k warmup. Thus the faster initial loss decrease and the final validation losses (3.146 vs 3.159) cannot be attributed to the loss function. The authors should compare both losses under matched optimization schedules and report multiple seeds or error bars before claiming benefits at scale.
  4. [Appendix G, Theorem 1] The theorem that harmonic loss has a finite global minimizer is not proved. Scale invariance is stated for simultaneous scaling of all w_k and x_i, but the argument that increasing the norm of W does not reduce the loss refers to scaling W alone; since ||x_i - t w_k|| does not scale uniformly in t when the w_k have different norms, correct classification can be lost. The existence of a finite minimizer and the 'finite convergence point by design' property used in the abstract therefore require a different argument.
minor comments (5)
  1. [Section 3.3 / Appendix F] The main text says Figure 3(b) is a mean over 3 seeds, whereas Appendix F Figure 14(b) is described as a fixed seed; please make the reporting consistent.
  2. [Appendix D] The statement that contrastive learning methods are 'not inherently supervised' is inaccurate given that supervised contrastive learning (Ref. [35]) is cited elsewhere in the paper; please qualify.
  3. [Figure 4] The caption mentions yellow cells for values below 0.01, but the MNIST panels appear in grayscale; please add a visible color legend or relabel the threshold.
  4. [Section 2] The heuristic for choosing n ~ sqrt(D) is not justified by the displayed limit lim(1+x^{-1})^x=e; please give the derivation or label the choice as a heuristic.
  5. [Table 1] The ImageNet comparison would be more informative if the harmonic pipeline were also evaluated with the same augmentation and schedule as the published cross-entropy baseline, rather than only against the authors' own cross-entropy run.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: harmonic loss is defined independently of the evaluation metrics, and the reported benefits are empirically tested or immediate mathematical consequences rather than fitted predictions.

full rationale

The paper's central claims are not circular. Harmonic loss is defined in Eq. (2) independently of the interpretability metrics used later (explained variance, parallelogram loss, probing F1), and the empirical evaluations use external benchmarks such as MNIST, ImageNet, OpenWebText, SST-2, and CoLA. No fitted parameter is renamed as a prediction: the harmonic exponent n is chosen by an explicit heuristic (n ~ sqrt(D)) and then swept in ablations, while the reported improvements in grokking, data efficiency, and interpretability are measured directly against standard cross-entropy baselines. Self-citations in the related-work sections, such as [7], [8], [17], [18], [20], and [21], provide background on grokking and representation geometry; none of them is load-bearing for the harmonic loss derivation, and the linear representation hypothesis is attributed to external work [33]. The strongest potential concern is Appendix G, Theorem 3, where the proof's stationarity calculation omits the probability factors and assumes off-class contributions are negligible. That is a proof-correctness issue, not a circular reduction: the theorem's conclusion does not feed back into the definition of harmonic loss, and the paper explicitly describes the finite convergence point and class-center interpretability as design goals rather than as fitted predictions. Overall, the derivation chain is self-contained with respect to its inputs.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central results depend on the hyperparameter n and on two unproven assumptions in the theoretical appendix: that PAC-Bayes margin bounds transfer to the distance-based classifier, and that off-class examples vanish from the weight formula. No new physical or conceptual entities are introduced.

free parameters (1)
  • harmonic exponent n = n=1 for MLPs; n≈28 for GPT-2 (sqrt(768)); swept in Appendix E
    Controls heavy-tailedness of the probability distribution; the choice affects representation quality and is set by heuristic or tuning, not derived.
assumptions (3)
  • standard math Standard PAC-Bayes margin bounds apply to the harmonic classifier
    Theorem 2 invokes a PAC-Bayes bound from [32] without derivation; assumes a distance margin and a bounded input ball.
  • ad hoc to paper Off-class contributions to the stationarity condition are negligible
    Theorem 3 proof asserts other-class repulsive forces balance out on average, turning a weighted average over all examples into a convex combination of class-k examples.
  • domain assumption Training data lies within a ball of radius R
    Used in Theorem 2 for the PAC-Bayes bound; typical but unstated for the empirical tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Harmonic Loss Trains Interpretable AI Models." pith.science (2026). https://pith.science/paper/SDCWQGR3

@misc{pith2026250201628,
  author       = {Pith},
  title        = {Pith review of: Harmonic Loss Trains Interpretable AI Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SDCWQGR3}},
  note         = {Machine review of arXiv:2502.01628}
}
read the original abstract

In this paper, we introduce harmonic loss as an alternative supervisory signal for training neural networks and large language models (LLMs). Harmonic loss differs from standard cross-entropy loss by (a) replacing the usual SoftMax normalization with a scale-invariant HarMax function and (b) computing logits via Euclidean distance rather than a dot product. Harmonic loss enables improved interpretability and faster convergence, owing to its scale invariance and finite convergence point by design, which can be interpreted as a class center. We first validate the performance of harmonic models across algorithmic, vision, and language datasets. Through extensive experiments, we demonstrate that models trained with harmonic loss perform better than standard models by: (a) enhancing interpretability, (b) requiring less data for generalization, and (c) reducing grokking. Moreover, we compare a GPT-2 model trained with harmonic loss to the standard GPT-2, illustrating that the harmonic model develops more interpretable representations. Looking forward, we believe harmonic loss may become a valuable tool in domains with limited data availability or in high-stakes applications where interpretability and reliability are paramount, paving the way for more robust and efficient neural network models.

Figures

Figures reproduced from arXiv: 2502.01628 by the authors.

Figure 1
Figure 1. Cross-entropy loss versus harmonic loss (ours). (a) Definitions. Cross-entropy loss leverages [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the top two principal components of the embeddings in synthetic experi [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) Cumulative explained variance as a function of principal components (mean over 20 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Left: Case study on modular addition. Standard MLP trained for modular addition without weight decay often fails to generalize. Generalization is only achieved with the addition of strong weight decay; however, (a) significant grokking occurs, and (b) while the first t…
Figure 5
Figure 5. Figure 5: GPT2 experiments: (Top left) loss curves. Harmonic GPT achieves a slightly lower loss compared to standard GPT. (Top right) cumulative distribution function with respect to parallelogram loss, for twelve function-vector tasks. Harmonic GPT consistently shows lower para…
Figure 6
Figure 6. Figure 6: Learned embeddings on the lattice and modular addition tasks. Each pane shows the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Visualization of the top two principal components of the embeddings in synthetic experi [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Harmonic loss (harmonic) and cross-entropy loss (standard) induce qualitatively different [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Visualization of layer 6 representations projected onto the first two principal components, [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Results for MLP models. Rows show harmonic, DotProd+HarMax, [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Results for Transformer models. Same ordering as Fig. 10. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Effect of the harmonic exponent n on lattice in-context learning. We sweep n ∈ {1, . . . , 10}. Columns 1–4: Harmonic–MLP, columns 5–8: Harmonic Transformer. The learned 5 × 5 lattice is remarkably stable; n=1 already provides crisp and interpretable geometry. E Sweep…
Figure 13
Figure 13. Figure 13: Effect of the harmonic exponent n on modular addition. Columns 1–4: Harmonic–MLP, columns 5–8: Harmonic Transformer. MLPs remain stable across seeds, whereas Transformers are more sensitive yet form tighter circles at higher n; n=1 works well for MLPs, while a larger …
Figure 14
Figure 14. Figure 14: (a) Cumulative explained variance vs. principal components (mean over 20 seeds). [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 13 canonical work pages

  1. [1]

    Sensitivity and generalization in neural networks: an empirical study

    Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08760, 2018

    arXiv 2018

  2. [2]

    Mechanistic interpretability for ai safety–a review

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082, 2024

    arXiv 2024

  3. [3]

    Deepspeed data efficiency: Improving deep learning model quality and training efficiency via efficient data sampling and routing

    Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Connor Holmes, Cheng Li, and Yuxiong He. Deepspeed data efficiency: Improving deep learning model quality and training efficiency via efficient data sampling and routing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18490–18498, 2024

    work page 2024

  4. [4]

    Patch diffusion: Faster and more data-efficient training of diffusion models

    Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, Mingyuan Zhou, et al. Patch diffusion: Faster and more data-efficient training of diffusion models. Advances in neural information processing systems, 36, 2024

    work page 2024

  5. [5]

    Grokking: Gen- eralization beyond overfitting on small algorithmic datasets

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022

    arXiv 2022

  6. [6]

    Towards out-of-distribution generalization: A survey

    Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021

    arXiv 2021

  7. [7]

    The clock and the pizza: Two stories in mechanistic explanation of neural networks

    Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  8. [8]

    Omnigrok: Grokking beyond algorithmic data

    Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117, 2022

    arXiv 2022

Show all 37 references
  1. [9]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in

    2020 doi

  2. [10]

    Function vectors in large language models

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023

    2023 arXiv

  3. [11]

    Language models represent space and time

    Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023

    2023 arXiv

  4. [12]

    Implicit representations of meaning in neural language models

    Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737, 2021

    2021 arXiv

  5. [13]

    Can language models encode perceptual structure without grounding? a case study in color

    Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color. arXiv preprint arXiv:2109.06129, 2021

    2021 arXiv

  6. [14]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023

    2023 arXiv

  7. [15]

    Monotonic representation of numeric properties in language models

    Benjamin Heinzerling and Kentaro Inui. Monotonic representation of numeric properties in language models. arXiv preprint arXiv:2403.10381, 2024

    2024 arXiv

  8. [16]

    The geometry of categorical and hierarchical concepts in large language models

    Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. arXiv preprint arXiv:2406.01506, 2024. 10

    2024 arXiv

  9. [17]

    Opening the ai black box: program synthesis via mechanistic interpretability

    Eric J Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukeli´c, and Max Tegmark. Opening the ai black box: program synthesis via mechanistic interpretability. arXiv preprint arXiv:2402.05110, 2024

    2024 arXiv

  10. [18]

    The geometry of concepts: Sparse autoencoder feature structure

    Yuxiao Li, Eric J Michaud, David D Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. arXiv preprint arXiv:2410.19750, 2024

    2024 arXiv

  11. [19]

    Iclr: In-context learning of representations

    Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. Iclr: In-context learning of representations. arXiv preprint arXiv:2501.00070, 2024

    2024 arXiv

  12. [20]

    Towards understanding grokking: An effective theory of representation learning

    Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651–34663, 2022

    2022

  13. [21]

    Not all language model features are linear

    Joshua Engels, Isaac Liao, Eric J Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. arXiv preprint arXiv:2405.14860, 2024

    2024 arXiv

  14. [22]

    Enhancing hydrological extremes prediction accuracy: Integrating diverse loss functions in transformer models

    Xue Li, Qi-Liang Sun, Yanfei Zhang, Jian Sha, and Man Zhang. Enhancing hydrological extremes prediction accuracy: Integrating diverse loss functions in transformer models. Envi- ronmental Modelling & Software, 177:106042, 2024

    2024

  15. [23]

    Echocardiographic image segmenta- tion with vision transformers: A comparative analysis of different loss functions

    Edoardo Bosco, Giovanni Magenes, and Giulia Matrone. Echocardiographic image segmenta- tion with vision transformers: A comparative analysis of different loss functions. In 2024 IEEE International Symposium on Medical Measurements and Applications (MeMeA) , pages 1–6. IEEE, 2024

    2024

  16. [24]

    Gener- alised dice overlap as a deep learning loss function for highly unbalanced segmentations

    Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Gener- alised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support...

    2017

  17. [25]

    Topology-aware focal loss for 3d image segmentation

    Andac Demir, Elie Massaad, and Bulent Kiziltan. Topology-aware focal loss for 3d image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 580–589, 2023

    2023

  18. [26]

    Tversky loss function for image segmentation using 3d fully convolutional deep networks

    Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour. Tversky loss function for image segmentation using 3d fully convolutional deep networks. In International workshop on machine learning in medical imaging, pages 379–387. Springer, 2017

    2017

  19. [27]

    Hybrid wind speed forecasting using iceemdan and transformer model with novel loss function

    Bala Saibabu Bommidi, Kiran Teeparthi, and Vishalteja Kosana. Hybrid wind speed forecasting using iceemdan and transformer model with novel loss function. Energy, 265:126383, 2023

    2023

  20. [28]

    Predicting o-glcnacylation sites in mammalian proteins with transformers and rnns trained with a new loss function

    Pedro Seber. Predicting o-glcnacylation sites in mammalian proteins with transformers and rnns trained with a new loss function. arXiv preprint arXiv:2402.17131, 2024

    2024

  21. [29]

    I-con: A unifying framework for representation learning

    Shaden Alshammari, John Hershey, Axel Feldmann, William T Freeman, and Mark Hamilton. I-con: A unifying framework for representation learning. arXiv preprint arXiv:2504.16929, 2025

    2025 arXiv

  22. [30]

    A comprehensive survey of loss functions in machine learning

    Qi Wang, Yue Ma, Kun Zhao, and Yingjie Tian. A comprehensive survey of loss functions in machine learning. Annals of Data Science, 9(2):187–212, 2022

    2022

  23. [31]

    Contrastive learning models for sentence representations

    Lingling Xu, Haoran Xie, Zongxi Li, Fu Lee Wang, Weiming Wang, and Qing Li. Contrastive learning models for sentence representations. ACM Transactions on Intelligent Systems and Technology, 14(4):1–34, 2023

    2023

  24. [32]

    A pac-bayesian approach to spectrally-normalized margin bounds for neural networks

    Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564, 2017. 11

    2017 arXiv

  25. [33]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023

    2023 arXiv

  26. [34]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  27. [35]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673, 2020

    2020

  28. [36]

    Neural network acceptability judgments

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018

    2018 arXiv

  29. [37]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language proce...

    2013

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.