REVIEW 2 major objections 5 minor 71 references
Noisy ranks are treated as probabilistic associations with several neighboring ranks, and an embedding is learned that respects those associations.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Stochastic Order Learning associates each instance with multiple plausible ranks and trains embeddings via complementary discriminative and stochastic-order losses that remain robust to ordinal label noise.
T0 review reviewed 2026-07-10 challenge →
load-bearing objection Clean, usable extension of geometric order learning to noisy ordinal labels; the stochastic reformulation is real and the empirics are broad enough to trust the gains. the 2 major comments →
Stochastic Order Learning: An Approach to Rank Estimation Using Noisy Data
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
When ordinal labels are noisy, each instance should be treated as stochastically associated with multiple neighboring ranks; an embedding that minimizes the expected squared distance to the corresponding rank centroids, together with pairwise probabilistic order constraints, yields rank estimates that stay reliable across noise types and severities.
What carries the argument
Stochastic dissimilarity Dh(x,r) = sum_s ps ||h(x) - mu_{r+s}||^2, whose minimization under a discrete-Gaussian noise model produces the monotonicity constraint that the discriminative and stochastic-order losses jointly enforce.
Load-bearing premise
The method derives its losses and centroid rule from a discrete Gaussian model of label errors; if real annotation noise is strongly asymmetric or multi-modal, those derived objectives may no longer match the data.
What would settle it
On a large ordinal dataset whose annotation errors are known to be strongly skewed or multi-modal, replace the Gaussian ps with the true error frequencies; if SOL then loses its advantage over ordinary order-learning and noise-robust regression baselines, the Gaussian modeling assumption is essential rather than merely convenient.
If this is right
- Noise-robust rank estimators can be obtained without requiring clean reference labels or explicit noise-rate estimation.
- The same stochastic-order losses apply unchanged to vision, medical imaging and text regression once an appropriate encoder is chosen.
- Outlier detection and mild relabeling become natural by-products of the learned embedding rather than separate heuristics.
- Inference reduces to finding the rank whose expected squared distance is smallest, avoiding expensive nearest-neighbor search at test time.
Where Pith is reading between the lines
- If annotator-specific or input-dependent noise distributions can be estimated on the fly, the same Dh formulation could absorb them without redesigning the losses.
- The monotonicity constraint may transfer to continuous ordinal regression by replacing discrete centroids with a smooth rank manifold.
- Because the method already models structured uncertainty, it is a natural candidate for active learning of the most ambiguous ranks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reformulates rank estimation under noisy ordinal labels as a stochastic ordering problem: each instance is associated with a distribution over plausible ranks rather than a single deterministic label. It introduces Stochastic Order Learning (SOL), which learns an embedding via a discriminative loss that attracts/repels instances relative to rank centroids under a stochastic dissimilarity Dh (Eqs. 3–10) and a stochastic order loss that enforces probabilistic pairwise ordering relations (Eqs. 12–17). Centroids are updated in closed form (Eq. 18); an optional outlier detection and relabeling step (Eqs. 19–21) is also provided. Inference selects the rank minimizing expected distance (Eq. 6). Experiments on MORPH II, CLAP2015, AADB, RSNA, and WMT2020 under Gaussian, Laplacian, uniform, skewed, and real-world noise show consistent gains over noise-robust classification/regression and order-learning baselines.
Significance. If the results hold, SOL supplies a clean, generalizable framework for ordinal prediction under structured label noise—an important practical setting that standard classification noise methods and clean-label order learning both handle poorly. Strengths include: (i) explicit derivations of the monotonicity constraint (Appendix A) and closed-form centroid rule (Appendix B); (ii) complementary losses that are not tautological restatements of the evaluation metric; (iii) broad empirical coverage across five datasets, four synthetic noise families, multiple κ levels, real-world WMT2020 noise, ablations of each loss, multi-seed stability, and alternative relabeling schemes; and (iv) released source code. The free parameters (T, τ, γ, β, σ_test) are documented and analyzed. The contribution is incremental relative to prior order learning (GOL, MWR) but the stochastic reformulation and the resulting robustness gains are concrete and useful.
major comments (2)
- Main-result Tables 1–4 report only point estimates of MAE/CS with no standard deviations or multi-seed statistics. Appendix D.12 later supplies five-seed means±std on MORPH II only. Because the central claim is consistent outperformance under diverse noise, the main tables should either include error bars (or multi-seed means) for all five datasets or explicitly state that the Appendix D.12 stability analysis is representative; otherwise the magnitude of the reported gains (often 0.02–0.1 MAE) cannot be assessed for statistical reliability.
- Section 3.1 and Eq. (2) introduce a discrete Gaussian for the noise model, and the desideratum, losses, and centroid rule are written with general {ps}. The paper correctly stress-tests Laplacian/uniform/skewed/input-dependent noise and real WMT2020 annotations (Tables 1–5, Appendix D.2). However, the main text still presents the Gaussian as the default generative model without a short, explicit statement that the algorithmic objects remain well-defined for any unimodal discrete {ps}. Adding one clarifying sentence (and, if space permits, a one-line general-ps derivation note) would remove residual ambiguity about how load-bearing the parametric form is.
minor comments (5)
- Figure 1 caption and surrounding text correctly contrast nominal vs. ordinal noise severity; a brief pointer to the later quantitative tables would help readers who skip the figure.
- Hyperparameter defaults (Table 9) and sensitivity analyses (Appendix D.1) are thorough; moving a one-sentence summary of recommended defaults into the main Implementation subsection would improve reproducibility for readers who do not open the appendix.
- Notation: the same symbol e is used both for the random noise variable and for individual errors ex; a minor typographic distinction would avoid momentary confusion in Section 3.1.
- Impact Statement correctly flags demographic bias risk on facial datasets; a parallel one-sentence note for the medical (RSNA) setting would be consistent.
- A few typographical slips remain (e.g., “order-identitiy” in Related Work; occasional missing spaces around citations). A final proofreading pass is warranted.
Circularity Check
No significant circularity: SOL's losses and centroid rule are derived from an explicit desideratum under a stated noise model, then evaluated on external benchmarks against independent baselines.
full rationale
The paper proposes a stochastic reformulation of noisy rank estimation (Eqs. 1–4), derives a sufficient monotonicity condition (Eq. 5, Appendix A), constructs a discriminative surrogate loss (Eqs. 8–10) and a probabilistic pairwise order loss (Eqs. 12–17) that encourage the desideratum, and obtains the centroid update by direct minimization of the expected squared distance (Eq. 18, Appendix B). Inference simply re-uses the same Dh (Eq. 6). None of these steps is self-definitional: the desideratum is an explicit design goal, not a restatement of the evaluation metrics (MAE/CS/PCC/SRCC). The discrete-Gaussian form of ps is an assumption, not a fitted quantity later re-presented as a prediction; the same algebraic forms hold for any discrete {ps}, and the paper empirically stress-tests Laplacian, uniform, skewed and input-dependent noise plus real WMT2020 annotations. Self-citations to prior order-learning papers (Lim et al., Lee et al.) supply only the clean-label baseline architecture; the stochastic losses, centroid rule and outlier module are new and are not justified by any uniqueness theorem or load-bearing self-citation. No parameter is fitted on a subset and then called a prediction of a related quantity. The method is therefore self-contained against external public benchmarks and independent baselines; circularity score is zero.
Axiom & Free-Parameter Ledger
free parameters (5)
- T (summation range in discriminative loss)
- τ (order-case threshold)
- γ (margin in order losses)
- β (outlier detection fraction)
- σ_test (fixed noise scale at inference)
axioms (3)
- domain assumption Label noise e follows a discrete Gaussian (symmetric, unimodal) distribution whose probabilities weight both Dh and the pairwise order probabilities.
- standard math The monotonicity constraint d(hx, μ_rx+s) ≤ d(hx, μ_rx+t) for |s|≤|t| is a sufficient condition for the stochastic-dissimilarity desideratum.
- domain assumption Centroids that minimize the expected squared distance under the same noise model are the optimal representatives of each rank.
invented entities (2)
-
Stochastic dissimilarity Dh(x,r)
no independent evidence
-
Discriminative loss ℓ_disc and stochastic order loss ℓ_order
no independent evidence
Cite this review
Pith. "Pith review of Stochastic Order Learning: An Approach to Rank Estimation Using Noisy Data." pith.science (2026). https://pith.science/paper/Z6V5UZQJ
@misc{pith2026260708103,
author = {Pith},
title = {Pith review of: Stochastic Order Learning: An Approach to Rank Estimation Using Noisy Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z6V5UZQJ}},
note = {Machine review of arXiv:2607.08103}
}
read the original abstract
Rank estimation under label noise poses a fundamental challenge, as ordinal annotations often exhibit structured uncertainty rather than simple label corruption. In this paper, we reformulate rank estimation with noisy ordinal labels as a stochastic ordering problem, in which each instance is inherently associated with multiple plausible ranks instead of a single deterministic label. Based on this view, we propose stochastic order learning (SOL), a learning framework that captures ordinal label uncertainty and learns an embedding space through two complementary objectives: a discriminative loss that structures instance--centroid interactions and a stochastic order loss that enforces probabilistic ordering relations between instances. Extensive experiments across diverse datasets demonstrate that SOL enables reliable rank estimation under various types and levels of label noise. The source code is available at https://github.com/cwlee00/SOL.
Figures
Reference graph
Works this paper leans on
-
[1]
Scaling Learning Algorithms Towards
Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards
-
[2]
and Osindero, Simon and Teh, Yee Whye , journal =
Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =
- [3]
-
[4]
Maaten, Laurens van der and Hinton, Geoffrey , journal=. Visualizing data using t-
- [5]
-
[6]
Semi-supervised semantic segmentation under label noise via diverse learning groups , author=. ICCV , year=
-
[7]
Hwang, Seong-Hyeon and Kim, Minsu and Whang, Steven Euijong , booktitle=
-
[8]
Castells, Thibault and Weinzaepfel, Philippe and Revaud, Jerome , booktitle=
-
[9]
Scalable penalized regression for noise detection in learning with noisy labels , author=. CVPR , year=
-
[10]
Very deep convolutional networks for large-scale image recognition , author=
- [11]
-
[12]
Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li , booktitle=CVPR, year=. Image
-
[13]
A constrained deep neural network for ordinal regression , author=
-
[14]
Soft labels for ordinal regression , author=
- [15]
-
[16]
Robust deep ordinal regression under label noise , author=. 2020 , booktitle=
work page 2020
-
[17]
Robust loss functions under label noise for deep neural networks , author=. AAAI , year=
-
[18]
Training deep neural-networks using a noise adaptation layer , author=
- [19]
-
[20]
Han, Bo and Yao, Quanming and Yu, Xingrui and Niu, Gang and Xu, Miao and Hu, Weihua and Tsang, Ivor and Sugiyama, Masashi , booktitle=NIPS, year=. Co-teaching:
-
[21]
Jiang, Lu and Zhou, Zhengyuan and Leung, Thomas and Li, Li-Jia and Fei-Fei, Li , booktitle=ICLR, year=. Mentor
-
[22]
Photo aesthetics ranking network with attributes and content adaptation , author=
-
[23]
Order learning using partially ordered data via chainization , author=
-
[24]
Geometric order learning for rank estimation , author=
-
[25]
Unsupervised order learning , author=
-
[26]
Shin, Nyeong-Ho and Lee, Seon-Ho and Kim, Chang-Su , booktitle=CVPR, year=. Moving window regression:
-
[27]
Deep Repulsive Clustering of Ordered Data Based on Order-Identitiy Decomposition , author=
-
[28]
Ordinal regression by extended binary classification , author=
-
[29]
Li, Wanhua and Lu, Jiwen and Feng, Jianjiang and Xu, Chunjing and Zhou, Jie and Tian, Qi , booktitle=CVPR, year=. Bridge
-
[30]
Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression , author=
-
[31]
Order learning and its application to age estimation , author=
-
[32]
Normalized loss functions for deep learning with noisy labels , author=
- [33]
-
[34]
Active Negative Loss Functions for Learning with Noisy Labels , author =. NIPS , year =
-
[35]
Learning with noisy labels , author=
-
[36]
Mean-variance loss for deep age estimation from a face , author=
-
[37]
Escalera, Sergio and Fabian, Junior and Pardo, Pablo and Bar\'. ICCV Workshops , year=
-
[38]
Ricanek, Karl and Tesafaye, Tamirat , booktitle=
-
[39]
Selfie: Refurbishing unclean samples for robust deep learning , author=
-
[40]
Learning from noisy labels by regularized estimation of annotator confusion , author=. CVPR , year=
-
[41]
Robust early-learning: Hindering the memorization of noisy labels , author=. ICLR , year=
-
[42]
Learning with noisy class labels for instance segmentation , author=. ECCV , year=
-
[43]
Yao, Huaxiu and Wang, Yiping and Zhang, Linjun and Zou, James and Finn, Chelsea , booktitle=
-
[44]
mixup: Beyond empirical risk minimization , author=
-
[45]
Generalized cross entropy loss for training deep neural networks with noisy labels , author=
-
[46]
Ordinal hyperplanes ranker with cost sensitivities for age estimation , author=
- [47]
-
[48]
Sample selection via contrastive fragmentation for noisy label regression , author=
-
[49]
Proceedings of the 29th International Conference on Computational Linguistics , year=
Noisy label regularisation for textual regression , author=. Proceedings of the 29th International Conference on Computational Linguistics , year=
-
[50]
Specia, Lucia and Blain, Fr \'e d \'e ric and Fomicheva, Marina and Fonseca, Erick and Chaudhary, Vishrav and Guzm \'a n, Francisco and Martins, Andr \'e F. T. Findings of the WMT 2020 Shared Task on Quality Estimation. Proceedings of the Fifth Conference on Machine Translation
work page 2020
- [51]
- [52]
-
[53]
Proceedings of the 2008 International Conference on Web Search and Data Mining , year=
Softrank: optimizing non-smooth rank metrics , author=. Proceedings of the 2008 International Conference on Web Search and Data Mining , year=
work page 2008
-
[54]
Selective-supervised contrastive learning with noisy labels , author=. CVPR , year=
-
[55]
Peer loss functions: Learning from noisy labels without knowing noise rates , author=. ICML , year=
- [56]
-
[57]
Pitawela, Dileepa and Carneiro, Gustavo and Chen, Hsiang-Ting , booktitle=
-
[58]
Distribution Highlighted Reference-based Label Distribution Learning for Facial Age Estimation , author=. WACV , year=
-
[59]
Age estimation using expectation of label distribution learning , author=. IJCAI , year=
-
[60]
Unimodal-concentrated loss: Fully adaptive label distribution learning for ordinal regression , author=. CVPR , year=
-
[61]
Halabi, Safwan S and Prevedello, Luciano M and Kalpathy-Cramer, Jayashree and Mamonov, Artem B and Bilbily, Alexander and Cicero, Mark and Pan, Ian and Pereira, Lucas Ara. The. Radiology , volume=
-
[62]
Unimodal regularized neuron stick-breaking for ordinal classification , author=. Neurocomputing , volume=
-
[63]
Distributed robust support vector ordinal regression under label noise , author=. Neurocomputing , pages=
-
[64]
Curriculum Loss: Robust Learning and Generalization against Label Corruption
Curriculum loss: Robust learning and generalization against label corruption , author=. arXiv preprint arXiv:1905.10045 , year=
work page internal anchor Pith review Pith/arXiv arXiv 1905
-
[65]
Van der Maaten, Laurens and Hinton, Geoffrey , journal=. Visualizing data using t-
-
[66]
IEEE Transactions on pattern analysis and machine intelligence , volume=
Automatic age estimation based on facial aging patterns , author=. IEEE Transactions on pattern analysis and machine intelligence , volume=
-
[67]
Deep label distribution learning with label ambiguity , author=. TIP , volume=
-
[68]
Journal of the Royal Statistical Society: Series B (Methodological) , volume=
Regression models for ordinal data , author=. Journal of the Royal Statistical Society: Series B (Methodological) , volume=
-
[69]
Journal of the Royal Statistical Society: Series B (Methodological) , volume=
Regression and ordered categorical variables , author=. Journal of the Royal Statistical Society: Series B (Methodological) , volume=
-
[70]
INFORMS Journal on Computing , volume=
Isotonic separation , author=. INFORMS Journal on Computing , volume=
-
[71]
Rule learning with monotonicity constraints , author=
This paper was first reviewed by grok-4.5 on July 10, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.