REVIEW 3 major objections 4 minor 34 references
Dist Loss: Enhancing Regression in Few-Shot Region through Distribution Distance Constraint
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Dist Loss adds a differentiable constraint that aligns the distribution of a regressor's predictions with the label distribution, and the paper reports that this term improves accuracy on rare target values in imbalanced regression…
desk verdict Neat loss, one clean comparison, but the SOTA claim mixes protocol and objective. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a differentiable surrogate for the distance between two distributions in one dimension. The label side is summarized by pseudo-labels: the kernel-density-estimated label density is discretized into bins, expected frequencies per bin are computed for a batch-sized sample, and the bin values are repeated to form an ordered sequence. The prediction side is summarized by pseudo-predictions, the sorted values of the model's outputs within the batch. Sorting is made differentiable through the projection-based fast differentiable sorting algorithm of Blondel et al. (2020), so the whole pipeline can be trained end-to-end. The distance between the two ordered sequences—computed with an inverse-probability-weighted L2 loss in the reported experiments—is what carries the distribution-aligning signal.
What would settle it
Run the identical linear-layer fine-tuning from the same pretrained checkpoints on IMDB-WIKI-DIR, AgeDB-DIR, and ECG-K-DIR using only the inverse-probability-weighted MSE loss, with no distribution term. If the few-shot MAE lands near 22.55, 9.12, and 1.33 respectively, the paper's attribution of the gains to the distribution term is not supported.
Extended reading notes
Core claim
The central claim is that aligning the model's output distribution with the label distribution during training reduces error in sparse-data regions without sacrificing overall accuracy. Dist Loss does this by building two one-dimensional sequences: pseudo-labels, sampled from the kernel-density-estimated label distribution at a size matching the batch, and pseudo-predictions, obtained by sorting the batch's predictions. The distance between these sorted sequences is treated as a differentiable surrogate for the distribution distance, using fast differentiable sorting to keep the operation trainable. Optimizing that distance together with inverse-probability-weighted MSE is reported to yield state-of-the-art few-shot MAE of 22.550 on IMDB-WIKI-DIR, 9.122 on AgeDB-DIR, and 1.329 on ECG-K-DIR, and to improve baselines in 26 of 30 metrics.
Load-bearing premise
Every result attributed to Dist Loss also changes the training protocol to linear-layer-only fine-tuning from a pretrained model, and the paper never runs the same fine-tuning with only the standard loss; if that protocol change alone produces the few-shot gains, the distribution term is not responsible.
Editorial extensions
If this is right
- Adding Dist Loss to a vanilla regressor lowers few-shot MAE by about 4.4 years on IMDB-WIKI-DIR, 3.8 years on AgeDB-DIR, and 0.44 mmol/L on ECG-K-DIR.
- Dist Loss can be stacked on existing imbalance methods such as LDS, FDS, RankSim, ConR, and Balanced MSE, improving their few-shot metrics in 26 of 30 reported cases.
- The gains extend beyond the sparse tail: supplementary results show Dist Loss also improves median-shot MAE on the two age datasets and median-shot GM on the ECG dataset.
- The method adds little training cost, with per-epoch time comparable to Balanced MSE and far below feature-level methods like FDS, RankSim, and ConR.
- Batch size has little effect on the result between 256 and 768, indicating the distribution estimate does not require a large batch.
Reading between the lines
- Because the pseudo-labels are generated from the training label distribution, Dist Loss is tied to the training marginal; if the test distribution shifts away from it, the alignment term may actively hurt, a regime the paper does not test.
- The sorted-prediction construction effectively performs a rank-based matching between predictions and labels, so Dist Loss can be seen as a differentiable one-dimensional Wasserstein-style distance; replacing the KDE sampling with exact quantile matching is a natural further simplification.
- The same mechanism should transfer to other long-tailed regression problems, such as forecasting rare weather extremes or estimating drug doses, where the same distribution-collapse phenomenon appears.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Dist Loss, a loss term for imbalanced regression that augments a sample-wise regression loss with a differentiable distribution-distance term. The label distribution is estimated with KDE, pseudo-labels are sampled to match that distribution, predictions are sorted with a differentiable sorting operator, and the distance between the pseudo-labels and sorted predictions is minimized. Experiments on IMDB-WIKI-DIR, AgeDB-DIR, and ECG-K-DIR report few-shot MAE improvements over several baselines and show that Dist Loss can be added to existing methods, with SOTA few-shot MAE values of 22.550, 9.122, and 1.329 on the three datasets.
Significance. If the reported gains are attributable to the loss term, Dist Loss is a valuable, simple, and cheap contribution: it requires no architectural change, adds modest computational overhead, and is compatible with existing imbalanced-regression methods. The paper ships code, includes extensive ablations (loss-function choice, batch size, imbalance ratio), and studies a clinically relevant potassium-regression task. The main caveat is that the experimental protocol for Dist Loss differs from most baselines, and the specific contribution of the distribution-distance term is not isolated; these issues are fixable with additional control experiments.
major comments (3)
- [Appendix A.2, Table 1] The comparison in Table 1 conflates the proposed loss term with a change in training protocol. Appendix A.2 states that Dist Loss runs 'used the same strategy as Balanced MSE, fine-tuning the linear layer based on pre-trained model (vanilla model) parameters,' while LDS, FDS, Ranksim, and ConR are trained end-to-end. Table 3 confirms that Balanced MSE and Dist Loss are much faster precisely because they only fine-tune linear layers. There is no control run that fine-tunes a linear layer using only the standard (or inverse-weighted) loss without the distribution-distance term. Consequently, the reported gains over the end-to-end baselines in Table 1 could be due to linear probing on frozen features rather than to the Dist Loss objective. This is load-bearing because the paper's central claim attributes the few-shot improvements to Dist Loss. Please add a linear-probe control with the vanilla loss, and report the comparison to each baseline under a matched protocol.
- [Section 3.2.1, Table 4] The contribution of the distribution-distance component is not isolated from the sample-wise reweighting. The sequence-difference function used for Dist Loss is inverse-probability-weighted MSE (INV-L2), as stated in Appendix A.2, so the sample-level term already upweights rare labels. Table 4 only varies the underlying metric (INV-L1 vs INV-L2); it does not ablate the distribution-distance mechanism (sorted pseudo-labels and pseudo-predictions) while keeping the inverse weighting. A control using plain per-sample inverse-weighted MSE without sorting would identify how much of the gain comes from distribution matching rather than from the reweighting that is already embedded in the loss. Without this ablation, the mechanism behind the few-shot improvements is ambiguous.
- [Tables 1 and 2] The results are reported from single runs without error bars or multiple seeds. Several of the headline margins are very small (e.g., 0.203 MAE over LDS on IMDB-WIKI-DIR, 0.088 MAE over Balanced MSE on ECG-K-DIR, and 0.491 MAE over Balanced MSE on AgeDB-DIR). Given the stochasticity of deep learning training, these differences may lie within run-to-run variation. The 'state-of-the-art' claims in Section 4.3 require mean and standard deviation over at least three seeds, or at least a statement of variance. This is especially important because the protocol is not fully matched for the non-linear-probe baselines.
minor comments (4)
- [Table 8] In Table 8, the ConR + Dist Loss entry for ECG-K-DIR Few MAE is reported as 11.336; this appears to be a typo for 1.336, since Tables 2 and 12 report 1.336 for the same configuration.
- [Figure 1c] Figure 1c is presented as demonstrating the effectiveness of Dist Loss, but since the loss explicitly minimizes the distribution distance between predictions and labels, the reduced discrepancy shown in the figure is expected by construction. The actual evidence for the method should rest on the accuracy metrics in the tables rather than on this distribution-alignment visualization.
- [Table 3, Appendix A.2] There is an inconsistency between the batch sizes reported in Table 3 (64, 64, and 256 for IMDB-WIKI-DIR, AgeDB-DIR, and ECG-K-DIR) and those stated in Appendix A.2 (512 for Dist Loss and Balanced MSE on all datasets, and 512 for all methods on ECG-K-DIR). Please clarify which batch sizes were used for the timing experiments and for training.
- [Appendix A.5] The numerical example in Appendix A.5 contains an apparent formatting error: the inequality '40 × 1.015 > 42 × 105' should presumably compare products such as 40·(10.1)^5 and 42·(10)^5. Please correct the notation.
Circularity Check
No significant circularity: the few-shot MAE/GM results are empirical benchmark outcomes, not consequences of the loss definition.
full rationale
Dist Loss is explicitly defined as jointly minimizing a differentiable approximation of the distance between prediction and label distributions (pseudo-predictions vs. KDE-derived pseudo-labels) and sample-wise error. Thus the reduced distribution discrepancy shown in Figure 1c is a property of the training objective rather than an independent finding, but the paper's load-bearing claims—SOTA few-shot MAE of 22.550/9.122/1.329 and the 26/30 or 28/30 improvement counts—are held-out test-set measurements against baselines; no fitted parameter is renamed as a prediction and no reported result is derived from the loss definition alone. The only author self-citation is to Net1D (Hong et al., 2020) as the backbone for ECG-K-DIR, an external architecture choice that does not support the central claim. There is no uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result. The linear-layer-only fine-tuning confound is an experimental-attribution concern, not a circularity, and therefore does not affect this score.
Assumptions & free parameters
free parameters (3)
- KDE bandwidth
- Number of bins B
- Distribution loss weight =
1
assumptions (3)
- domain assumption KDE with the chosen but unreported bandwidth gives a reliable estimate of the true label distribution.
- domain assumption Sorted predictions in one batch approximate the overall prediction distribution.
- standard math Fast differentiable sorting (Blondel et al., 2020) returns the correct sorted order with usable gradients.
Cite this review
Pith. "Pith review of Dist Loss: Enhancing Regression in Few-Shot Region through Distribution Distance Constraint." pith.science (2026). https://pith.science/paper/RF324LVY
@misc{pith2026241115216,
author = {Pith},
title = {Pith review of: Dist Loss: Enhancing Regression in Few-Shot Region through Distribution Distance Constraint},
year = {2026},
howpublished = {\url{https://pith.science/paper/RF324LVY}},
note = {Machine review of arXiv:2411.15216}
}
read the original abstract
Imbalanced data distributions are prevalent in real-world scenarios, posing significant challenges in both imbalanced classification and imbalanced regression tasks. They often cause deep learning models to overfit in areas of high sample density (many-shot regions) while underperforming in areas of low sample density (few-shot regions). This characteristic restricts the utility of deep learning models in various sectors, notably healthcare, where areas with few-shot data hold greater clinical relevance. While recent studies have shown the benefits of incorporating distribution information in imbalanced classification tasks, such strategies are rarely explored in imbalanced regression. In this paper, we address this issue by introducing a novel loss function, termed Dist Loss, designed to minimize the distribution distance between the model's predictions and the target labels in a differentiable manner, effectively integrating distribution information into model training. Dist Loss enables deep learning models to regularize their output distribution during training, effectively enhancing their focus on few-shot regions. We have conducted extensive experiments across three datasets spanning computer vision and healthcare: IMDB-WIKI-DIR, AgeDB-DIR, and ECG-Ka-DIR. The results demonstrate that Dist Loss effectively mitigates the negative impact of imbalanced data distribution on model performance, achieving state-of-the-art results in sparse data regions. Furthermore, Dist Loss is easy to integrate, complementing existing methods.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Fast differentiable sorting and ranking
Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning, pp.\ 950--959. PMLR, 2020
work page 2020
-
[2]
Smogn: a pre-processing approach for imbalanced regression
Paula Branco, Lu \' s Torgo, and Rita P Ribeiro. Smogn: a pre-processing approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications, pp.\ 36--50. PMLR, 2017
2017
-
[3]
Paula Branco, Luis Torgo, and Rita P. Ribeiro. Rebagg: Resampled bagging for imbalanced regression. In Luís Torgo, Stan Matwin, Nathalie Japkowicz, Bartosz Krawczyk, Nuno Moniz, and Paula Branco (eds.), Proceedings of the Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, volume 94 of Proceedings of Machine Learnin...
work page 2018
-
[4]
A systematic study of the class imbalance problem in convolutional neural networks
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106: 0 249--259, 2018
2018
-
[5]
Smote: synthetic minority over-sampling technique
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16: 0 321--357, 2002
2002
-
[6]
Heritable arrhythmias associated with abnormal function of cardiac potassium channels
Lia Crotti, Katja E Odening, and Michael C Sanguinetti. Heritable arrhythmias associated with abnormal function of cardiac potassium channels. Cardiovascular research, 116 0 (9): 0 1542--1556, 2020
work page 2020
-
[7]
Learning a distance metric by balancing kl-divergence for imbalanced datasets
Lin Feng, Huibing Wang, Bo Jin, Haohao Li, Mingliang Xue, and Le Wang. Learning a distance metric by balancing kl-divergence for imbalanced datasets. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 49 0 (12): 0 2384--2395, 2018
work page 2018
-
[8]
Abnormalities of potassium in heart failure: Jacc state-of-the-art review
Jo \ a o Pedro Ferreira, Javed Butler, Patrick Rossignol, Bertram Pitt, Stefan D Anker, Mikhail Kosiborod, Lars H Lund, George L Bakris, Matthew R Weir, and Faiez Zannad. Abnormalities of potassium in heart failure: Jacc state-of-the-art review. Journal of the American College of Cardiology, 75 0 (22): 0 2836--2850, 2020
work page 2020
Show all 34 references
-
[9]
Development and validation of a deep-learning model to screen for hyperkalemia from the electrocardiogram
Conner D Galloway, Alexander V Valys, Jacqueline B Shreibati, Daniel L Treiman, Frank L Petterson, Vivek P Gundotra, David E Albert, Zachi I Attia, Rickey E Carter, Samuel J Asirvatham, et al. Development and validation of a deep-learning model to screen for hyperkalemia from ...
2019
-
[10]
R ank S im: Ranking similarity regularization for deep imbalanced regression
Yu Gong, Greg Mori, and Fred Tung. R ank S im: Ranking similarity regularization for deep imbalanced regression. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine L...
2022
-
[11]
Mortality risk stratification utilizing artificial intelligence electrocardiogram for hyperkalemia in cardiac intensive care unit patients
David M Harmon, Chris K Heinrich, John J Dillon, Rickey E Carter, Kianoush B Kashani, Zachi I Attia, Paul A Friedman, and Jacob C Jentzer. Mortality risk stratification utilizing artificial intelligence electrocardiogram for hyperkalemia in cardiac intensive care unit patients...
2024
-
[12]
Maher, Alaa Aljiffry, Jimeng Sun, and Alexey Tumanov
Shenda Hong, Yanbo Xu, Alind Khare, Satria Priambada, Kevin O. Maher, Alaa Aljiffry, Jimeng Sun, and Alexey Tumanov. HOLMES: health online model ensemble serving for deep learning models in intensive care units. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (ed...
2020
-
[13]
Mimic-iv
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pp.\ 49--55, 2020
2021
-
[14]
David Evans
Mahsa Keramati, Lili Meng, and R. David Evans. Conr: Contrastive regularizer for deep imbalanced regression. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024
2024
-
[15]
Potassium disorders: hypokalemia and hyperkalemia
Michael J Kim, Christina Valerio, and Glynnis K Knobloch. Potassium disorders: hypokalemia and hyperkalemia. American Family Physician, 107 0 (1): 0 59--70A, 2023
2023
-
[16]
Logistic regression in rare events data
Gary King and Langche Zeng. Logistic regression in rare events data. Political analysis, 9 0 (2): 0 137--163, 2001
2001
-
[17]
Focal loss for dense object detection
T Lin. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017
2017 arXiv
-
[18]
Large-scale long-tailed recognition in an open world
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 2537--2546, 2019
2019
-
[19]
Agedb: the first manually collected, in-the-wild age database
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, volume...
2017
-
[20]
On estimation of a probability density function and mode
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathematical statistics, 33 0 (3): 0 1065--1076, 1962
1962
-
[21]
Balanced mse for imbalanced visual regression
Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced mse for imbalanced visual regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7926--7935, 2022
2022
-
[22]
Deep expectation of real and apparent age from a single image without facial landmarks
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126 0 (2-4): 0 144--157, 2018
2018
-
[23]
Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019
1911 arXiv
-
[24]
Rusboost: A hybrid approach to alleviating class imbalance
Chris Seiffert, Taghi M Khoshgoftaar, Jason Van Hulse, and Amri Napolitano. Rusboost: A hybrid approach to alleviating class imbalance. IEEE transactions on systems, man, and cybernetics-part A: systems and humans, 40 0 (1): 0 185--197, 2009
2009
-
[25]
Density-based weighting for imbalanced regression
Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. Density-based weighting for imbalanced regression. Machine Learning, 110: 0 2187--2211, 2021
2021
-
[26]
Posterior re-calibration for imbalanced datasets
Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re-calibration for imbalanced datasets. Advances in neural information processing systems, 33: 0 8101--8113, 2020
2020
-
[27]
Smote for regression
Lu \' s Torgo, Rita P Ribeiro, Bernhard Pfahringer, and Paula Branco. Smote for regression. In Portuguese conference on artificial intelligence, pp.\ 378--389. Springer, 2013
2013
-
[28]
Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing
Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[29]
Delving into deep imbalanced regression
Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ ...
2021
-
[30]
Conditional wasserstein generative adversarial network-gradient penalty-based approach to alleviating imbalanced data classification
Ming Zheng, Tong Li, Rui Zhu, Yahui Tang, Mingjing Tang, Leilei Lin, and Zifei Ma. Conditional wasserstein generative adversarial network-gradient penalty-based approach to alleviating imbalanced data classification. Information Sciences, 512: 0 1009--1023, 2020
2020
-
[31]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
-
[32]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[33]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[34]
o/^1c ͛7QcE F [nєb | iٲ F ˱vZٳG' :T mÃT <8p w
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
2000
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.