REVIEW 2 major objections 5 minor 57 references
Estimation of perceptual scales using ordinal embedding
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Ordinal embedding from machine learning, applied to triplet comparisons, estimates perceptual scaling functions as accurately as MLDS in the standard one-dimensional monotonic case and also handles non-monotonic and multi-dimensional…
desk verdict Solid methods-transfer paper: the 1D monotonic and non-monotonic results hold up, but the multi-dimensional 'desirable scaling function' claim is under-validated and should be tempered or better anchored. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ordinal embedding problem: given triplet answers, find points $y_1,\dots,y_n$ in $d$-dimensional Euclidean space whose distances are consistent with the answers. The workhorse is stochastic triplet embedding (STE), which models the probability that stimulus $i$ is judged closer to $j$ than to $k$ as $p_{ijk} = \frac{\exp(-\lVert y_i - y_j \rVert^2)}{\exp(-\lVert y_i - y_j \rVert^2) + \exp(-\lVert y_i - y_k \rVert^2)}$, then maximizes the likelihood of the observed answers; the t-STE variant replaces the Gaussian kernel with a heavy-tailed Student-t kernel for robustness to noise. This machinery carries the argument because it imposes neither monotonicity nor a one-dimensional target, and it is paired with the cross-validated triplet error as a ground-truth-free way to judge whether a recovered scale is any good.
What would settle it
Conduct a triplet experiment on a stimulus set known to violate the Euclidean metric (e.g., similarity judgments that fail the triangle inequality); if t-STE's cross-validated triplet error remains far above the repeat-answer baseline while a non-Euclidean model fits well, the Euclidean assumption is the load-bearing weakness.
Extended reading notes
Core claim
The discovery is that the classic scaling problem can be re-cast as ordinal embedding: treat the stimuli as abstract items, the triplet answers as ordinal constraints, and the recovered Euclidean coordinates as the perceptual scale. The paper shows that stochastic triplet embedding, in particular the t-distributed variant t-STE, produces scaling functions whose cross-validated triplet error is on par with MLDS in the one-dimensional monotonic setting, while MLDS fails on non-monotonic functions and cannot produce multi-dimensional scales at all. In a simulation built from color-similarity data, the embedding methods reconstruct the two-dimensional color circle from a fraction of the triplets, where NMDS loses the structure to noise. In a real slant-from-texture experiment with eight observers, the ordinal embedding methods match MLDS's triplet error and, unlike MLDS, reveal non-monotonic perceptual patterns in two observers. The paper presents this as evidence that ordinal embedding methods are promising default scaling algorithms.
Load-bearing premise
The approach assumes perceived dissimilarity is Euclidean and low-dimensional, so if the true perceptual geometry is not Euclidean the recovered embedding can be a best-fit artifact rather than the actual perception.
Editorial extensions
If this is right
- If the claim holds, t-STE can serve as a default scaling method in the standard one-dimensional monotonic setting, matching MLDS accuracy when enough triplets are collected.
- Non-monotonic perceptual functions such as U-shaped or sinusoidal scales become estimable from triplet data, a case where MLDS systematically produces the wrong function.
- Multi-dimensional perceptual spaces such as color or pitch become recoverable from triplet judgments, extending scaling to settings MLDS cannot address.
- Because about $d\,n\log n$ randomly sampled triplets suffice, experiments are feasible where collecting a full dissimilarity ordering would be impossible.
- The cross-validated triplet error supplies a ground-truth-free criterion for judging when an embedding is good enough, usable for MLDS and NMDS as well.
Reading between the lines
- Editorial: if the Euclidean model is wrong, a t-STE embedding could fit triplet answers well while misrepresenting the true geometry, so testing on known non-Euclidean similarity spaces would delimit when the method is trustworthy.
- Editorial: because embeddings are only defined up to rotation, reflection, and scaling, comparing observers requires an alignment step; adding such an alignment could make the method a tool for studying inter-observer agreement in perceptual geometry.
- Editorial: the $d\,n\log n$ sample bound invites adaptive triplet selection that targets comparisons near estimated equality boundaries, potentially reducing data requirements below random sampling.
- Editorial: the same triplet-embedding formulation should transfer to conjoint measurement and multimodal stimuli, offering a way to map combined perceptual spaces without additivity or independence assumptions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes using ordinal embedding methods (STE, t-STE, LOE) to estimate perceptual scaling functions from triplet comparison data, in contrast to the traditional psychophysical methods NMDS and MLDS. It argues that ordinal embedding requires fewer triplets than NMDS, does not impose monotonicity as MLDS does, and can handle multi-dimensional perceptual spaces. The authors support this with simulations covering one-dimensional monotonic and non-monotonic scaling functions and a two-dimensional perceptual space, plus two real experiments (slant-from-texture and Eidolon image distortion). They conclude that in the one-dimensional monotonic case ordinal embedding performs comparably to MLDS, while in non-monotonic and multi-dimensional cases it is the only method among those considered that yields a desirable scaling function.
Significance. If the central claim holds, the paper provides psychophysics with a practical 'default' scaling algorithm based on triplet comparisons, with weaker assumptions than MLDS and lower data requirements than NMDS. The simulation study is extensive and well controlled, covering four noise levels and five triplet fractions, and the real experiments use cross-validated triplet error with a useful human-baseline calibration for the Eidolon data. The comparison with MLDS in the monotonic one-dimensional case is informative, and the non-monotonic one-dimensional simulation makes a clear point. The main limitation is that the multi-dimensional part of the central claim is only validated by a simulation whose ground truth is itself an NMDS Euclidean embedding, and by a real experiment that checks predictive consistency but not the interpretability of the recovered dimensions.
major comments (2)
- [Section 3.3, Figure 6(a)] The only multi-dimensional simulation uses as ground truth the two-dimensional NMDS embedding of Ekman's color-similarity data, with the authors explicitly stating that 'we do not argue that this embedding is "correct" in any way; we just use it as a ground truth.' Because NMDS and the tested ordinal-embedding methods both model dissimilarity with Euclidean distances, this simulation demonstrates that t-STE can recover an NMDS-style Euclidean configuration from triplet answers, but it does not validate that the recovered coordinates equal the true perceptual scale. The abstract's claim that in higher dimensions 'only our ordinal embedding methods can produce a desirable scaling function' therefore rests on this simulation plus the Eidolon experiment, and the latter provides only predictive consistency, not perceptual interpretability.
- [Section 4.1, Figure 9] The evidence that the Eidolon perceptual space is two-dimensional is the decrease in cross-validated triplet error from d=1 to d=2. Cross-validated triplet error measures agreement with the same type of triplet answers used for training; an embedding that fits triplets well need not have coordinate axes that correspond to any meaningful perceptual attribute. The paper itself lists 'Interpreting the embedding' as an open issue in Section 6.1, and no analysis is reported relating the recovered coordinates to reach, grain, or coherence. To support the multi-dimensional scaling claim, the authors should either provide an external validation (for example, correlation of embedding coordinates with the generation parameters, or a separate judgment task) or explicitly limit the claim to predictive consistency of triplet answers.
minor comments (5)
- [Section 3.2.1] The text says the average MSE and triplet errors for the monotonic simulation are depicted in 'Figure 5 (d) and (e)', but the relevant panels are in Figure 4; this cross-reference should be corrected.
- [Figure 6 caption] The caption describes the comparison as 'the ordinal embedding methods (MLDS, STE and TSTE)' but MLDS is not an ordinal embedding method; the caption should list LOE, STE, and t-STE.
- [Section 2.3.2] There is a typo in the text: 'called chroma and height by Shapard' should read 'by Shepard'.
- [Section 4.1] The side experiment on triplet difficulty should clarify the relationship between the measured percentage of inconsistent repeated answers (9.2%, 9.8%, 11%) and the statement that 'we would expect at least 0.10 triplet error'; as written, the step from the observed inconsistency rate to the 0.10 error floor is not immediate.
- [Section 3.1] The description of choosing the best of 10 restarts by 'the least triplet error' should specify whether this is the training-set triplet error, since the definition in Equation (5) allows both training and validation variants.
Circularity Check
No significant circularity: evaluation uses held-out triplet error and ground-truth MSE, with no fitted parameter or self-citation chain forcing the central claims.
full rationale
The derivation chain is self-contained against independent benchmarks. In simulations, the authors generate triplet answers from explicitly specified ground-truth scaling functions and evaluate with MSE against those functions: “we can compute the mean-squared-error (MSE) between the estimated scales ŷ and the true perceptual function values y” (Section 3.1), so the quality measure is not the training objective. For real data, the paper consistently uses cross-validated triplet error and warns that the training-set version “can be highly biased and typically underestimates the true triplet error (overfitting)” (Section 5.3), so no training-set prediction is relabeled as a result. The only multi-dimensional simulation uses an NMDS embedding of Ekman's color data as ground truth and explicitly disclaims it: “we do not argue that this embedding is ‘correct’ in any way; we just use it as a ground truth to generate further simulations” (Section 3.3). That is a same-model benchmark (both NMDS and ordinal embedding are Euclidean), but it is an honest simulation limitation, not a circular derivation, because the NMDS configuration is not an input to t-STE and the triplets are generated with added noise. Likewise, the Eidolon experiment is evaluated on held-out triplet prediction, and the paper itself flags the absence of an external anchor: “Interpreting the embedding ... What does each perceptual dimension mean? How are the perceptual dimensions related to the parameters of the stimulus (in this case reach, coherence and grain)?” (Section 6.1). Self-citations (LOE, uncertainty estimates, comparison-based nearest neighbors) are background or baseline algorithms and are not load-bearing; no uniqueness theorem is invoked to forbid alternatives, and t-STE is taken from the external implementation of Van Der Maaten and Weinberger (2012) with fixed default parameters. The central 1D monotonic claim is additionally checked against MLDS with the same triplet inputs and against NMDS, so the paper's main comparisons do not reduce to its own definitions.
Assumptions & free parameters
free parameters (1)
- embedding dimension d =
1 to 8 in Eidolon; set to true dimension in simulations
assumptions (3)
- domain assumption Perceptual dissimilarity can be represented by Euclidean distances in a low-dimensional space.
- domain assumption Triplet responses follow a probabilistic model based on embedding distances (STE) or are generated by a noisy observer model.
- domain assumption Observers answer triplet questions independently.
Cite this review
Pith. "Pith review of Estimation of perceptual scales using ordinal embedding." pith.science (2026). https://pith.science/paper/UPTZR5JX
@misc{pith2026190807962,
author = {Pith},
title = {Pith review of: Estimation of perceptual scales using ordinal embedding},
year = {2026},
howpublished = {\url{https://pith.science/paper/UPTZR5JX}},
note = {Machine review of arXiv:1908.07962}
}
read the original abstract
In this paper, we address the problem of measuring and analysing sensation, the subjective magnitude of one's experience. We do this in the context of the method of triads: the sensation of the stimulus is evaluated via relative judgments of the form: "Is stimulus S_i more similar to stimulus S_j or to stimulus S_k?". We propose to use ordinal embedding methods from machine learning to estimate the scaling function from the relative judgments. We review two relevant and well-known methods in psychophysics which are partially applicable in our setting: non-metric multi-dimensional scaling (NMDS) and the method of maximum likelihood difference scaling (MLDS). We perform an extensive set of simulations, considering various scaling functions, to demonstrate the performance of the ordinal embedding methods. We show that in contrast to existing approaches our ordinal embedding approach allows, first, to obtain reasonable scaling function from comparatively few relative judgments, second, the estimation of non-monotonous scaling functions, and, third, multi-dimensional perceptual scales. In addition to the simulations, we analyse data from two real psychophysics experiments using ordinal embedding methods. Our results show that in the one-dimensional, monotonically increasing perceptual scale our ordinal embedding approach works as well as MLDS, while in higher dimensions, only our ordinal embedding methods can produce a desirable scaling function. To make our methods widely accessible, we provide an R-implementation and general rules of thumb on how to use ordinal embedding in the context of psychophysics.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Generalized non-metric multidimensional scaling
Agarwal, S., Wills, J., Cayton, L., Lanckriet, G., Kriegman, D., and Belongie, S. Generalized non-metric multidimensional scaling. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 11--18, 2007
work page 2007
-
[2]
Aguilar, G., Wichmann, F. A., and Maertens, M. Comparing sensitivity estimates from MLDS and forced-choice methods in a slant-from-texture experiment. Journal of Vision, 17 0 (1): 0 37, 1--18, 2017
work page 2017
-
[3]
Active learning ranking from pairwise preferences with almost optimal query complexity
Ailon, N. Active learning ranking from pairwise preferences with almost optimal query complexity. In Advances in Neural Information Processing Systems ( NIPS ) , pages 810--818, 2011
work page 2011
-
[4]
Amid, E. and Ukkonen, A. Multiview triplet embedding: Learning attributes in multiple maps. In International Conference on Machine Learning (ICML) , pages 1472--1480, 2015
work page 2015
-
[5]
Some theory for ordinal embedding
Arias-Castro , E. Some theory for ordinal embedding. Preprint available at Arxiv, abs/1501.02861, 2015
work page Pith review arXiv 2015
-
[6]
Learning combinatorial functions from pairwise comparisons
Balcan, M., Vitercik, E., and White, C. Learning combinatorial functions from pairwise comparisons. In Conference on Learning Theory (COLT) , 2016
work page 2016
-
[7]
Barsalou, L. W. Cognitive psychology: An overview for cognitive scientists. Psychology Press, 2014
work page 2014
-
[8]
Brainard, D. H. The psychophysics toolbox. Spatial Vision, 10: 0 433--436, 1997
work page 1997
Show all 57 references
-
[9]
H., Dawes, R
Coombs, C. H., Dawes, R. M., and Tversky, A. Mathematical Psychology. Prentice-Hall, New Jersey, 1970
1970
-
[10]
O., Wagemans, J., and Vogels, R
de Beeck , H. O., Wagemans, J., and Vogels, R. Inferotemporal neurons represent low-dimensional configurations of parameterized shapes. Nature neuroscience, 4 0 (12): 0 1244, 2001
2001
-
[11]
S., and Heer, J
Demiralp, C ., Bernstein, M. S., and Heer, J. Learning perceptual kernels for visualization design. IEEE transactions on visualization and computer graphics, 20 0 (12): 0 1933--1942, 2014
1933
-
[12]
Dimensions of color vision
Ekman, G. Dimensions of color vision. The Journal of Psychology, 38 0 (2): 0 467--474, 1954
1954
-
[13]
Fechner, G. T. Elemente der Psychophysik. Breitkopf und Härtel, Leipzig, 1860
-
[14]
Gescheider, G. A. Psychophysical scaling. Annual review of psychology, 39 0 (1): 0 169--200, 1988
1988
-
[15]
Comparison-based nearest neighbor search
Haghiri, S., Ghoshdastidar, D., and von Luxburg, U. Comparison-based nearest neighbor search. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 851--859, 2017
2017
-
[16]
Houtsma, A. J. M. Pitch perception . Hearing, 6: 0 262, 1995
1995
-
[17]
G., and Nowak, R
Jain, L., Jamieson, K. G., and Nowak, R. Finite sample prediction and recovery bounds for ordinal embedding. In Advances in Neural Information Processing Systems (NIPS) , pages 2711--2719, 2016
2016
-
[18]
Jamieson, K. G. and Nowak, R. D. Low-dimensional embedding using adaptively selected ordinal data. In Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1077--1084, 2011
2011
-
[19]
M., and Suppes, P
Kaneshiro, B., Perreau Guimaraes , M., Kim, H.-S., Norcia, A. M., and Suppes, P. A representational similarity analysis of the dynamics of object processing using single-trial eeg classification. PLoS One, 10 0 (8): 0 e0135697, 2015
2015
-
[20]
P., and Vogels, R
Kayaert, G., Biederman, I., Op de Beeck, H. P., and Vogels, R. Tuning for shape dimensions in macaque inferior temporal cortex. European Journal of Neuroscience, 22 0 (1): 0 212--224, 2005
2005
-
[21]
and von Luxburg , U
Kleindessner, M. and von Luxburg , U. Uniqueness of ordinal embedding. In Conference on Learning Theory (COLT) , pages 40--67, 2014
2014
-
[22]
and von Luxburg , U
Kleindessner, M. and von Luxburg , U. Dimensionality estimation without distances. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 471--479, 2015
2015
-
[23]
What’s new in Psychtoolbox-3
Kleiner, M., Brainard, D., Pelli, D., Ingling, A., Murray, R., and Broussard, C. What’s new in Psychtoolbox-3 . Perception, 36 0 (14): 0 1, 2007
2007
-
[24]
Difference scaling of image quality in compression-degraded images
Knoblauch, K., Charrier, C., Cherifi, H., Yang, J., and Maloney, L. Difference scaling of image quality in compression-degraded images. Perception ECVP abstract, 27, 1998
1998
-
[25]
and Maloney, L
Knoblauch, K. and Maloney, L. T. MLDS : Maximum likelihood difference scaling in R . Journal of Statistical Software, 25: 0 1--26, 2010
2010
-
[26]
Eidolons: Novel stimuli for vision research
Koenderink, J., Valsecchi, M., van Doorn, A., Wagemans, J., and Gegenfurtner, K. Eidolons: Novel stimuli for vision research. Journal of Vision, 17 0 (2): 0 7, 2017
2017
-
[27]
Krantz, D. H. Visual scaling. In Visual psychophysics, pages 660--689. Springer, 1972
1972
-
[28]
H., Luce, R
Krantz, D. H., Luce, R. D., Suppes, P., and Tversky, A. Foundations of measurement: Geometrical, threshold, and probabilistic representations, volume 1. Courier Corporation, 2007
2007
-
[29]
Kruskal, J. B. Nonmetric multidimensional scaling: A numerical method. Psychometrika, 29 0 (2): 0 115--129, 1964 a
1964
-
[30]
Kruskal, J. B. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29 0 (1): 0 1--27, 1964 b
1964
-
[31]
L., Song, A., and Yu, A
Li, L., Malave, V. L., Song, A., and Yu, A. Extracting human face similarity judgments: Pairs or triplets? In CogSci , 2016
2016
-
[32]
Euclidean distance geometry and applications
Liberti, L., Lavor, C., Maculan, N., and Mucherino, A. Euclidean distance geometry and applications. Siam Review, 56 0 (1): 0 3--69, 2014
2014
-
[33]
Uncertainty estimates for ordinal embeddings
Lohaus, M., Hennig, P., and von Luxburg , U. Uncertainty estimates for ordinal embeddings. Preprint available at Arxiv, abs/1906.11655, 2019
1906 arXiv
-
[34]
Luce, R. D. and Edwards, W. The derivation of subjective scales from just noticeable differences. Psychological review, 65 0 (4): 0 222, 1958
1958
-
[35]
Fractional state space analysis of economic systems
Machado, J., Mata, M., and Lopes, A. Fractional state space analysis of economic systems. Entropy, 17 0 (8): 0 5402--5421, 2015
2015
-
[36]
Maloney, L. T. and Yang, J. N. Maximum likelihood difference scaling. Journal of Vision, 3 0 (8): 0 5, 2003
2003
-
[37]
Marks, L. E. and Gescheider, G. A. Psychophysical scaling. In Stevens' Handbook of Experimental Psychology, volume IV, Methodology in Experimental Psychology, chapter 3, pages 91--138. John Wiley and Sons, 2002
2002
-
[38]
Norris, W. F. and Oliver, C. A. System of Diseases of the Eye, volume 3. JB Lippincott, 1898
-
[39]
Reed, S. K. Pattern recognition and categorization. Cognitive psychology, 3 0 (3): 0 382--407, 1972
1972
-
[40]
A., and Wagemans, J
Rosas, P., Wichmann, F. A., and Wagemans, J. Some observations on the effects of slant and texture type on slant-from-texture. Vision Research, 44 0 (13): 0 1511--1535, 2004
2004
-
[41]
O., Wagemans, J., and Wichmann, F
Rosas, P., Ernst, M. O., Wagemans, J., and Wichmann, F. A. Texture and haptic cues in slant discrimination: Reliability-based cue weighting without statistically optimal cue combination. Journal of the Optical Society of America A, 22 0 (5): 0 801--809, 2005
2005
-
[42]
A., and Wagemans, J
Rosas, P., Wichmann, F. A., and Wagemans, J. Texture and object motion in slant discrimination: Failure of reliability-based weighting of cues may be evidence for strong fusion. Journal of Vision, 7 0 (6:3): 0 1--21, 2007
2007
-
[43]
and Joachims, T
Schultz, M. and Joachims, T. Learning a distance metric from relative comparisons. In Advances in Neural Information Processing Systems (NIPS) , pages 41--48, 2003
2003
-
[44]
Shepard, R. N. The analysis of proximities: multidimensional scaling with an unknown distance function. i. Psychometrika, 27 0 (2): 0 125--140, 1962
1962
-
[45]
Shepard, R. N. Psychological relations and psychophysical scales: On the status of “direct” psychophysical measurement. Journal of Mathematical Psychology, 24 0 (1): 0 21--57, 1981
1981
-
[46]
Shepard, R. N. Geometrical approximations to the structure of musical pitch. Psychological review, 89 0 (4): 0 305, 1982
1982
-
[47]
Smith, C. A. and Ellsworth, P. C. Patterns of cognitive appraisal in emotion. Journal of personality and social psychology, 48 0 (4): 0 813, 1985
1985
-
[48]
Stevens, S. S. On the psychophysical law. Psychological review, 64 0 (3): 0 153, 1957
1957
-
[49]
Stevens, S. S. To honor fechner and repeal his law. Science, 133 0 (3446): 0 80--86, 1961
1961
-
[50]
Adaptively learning the crowd kernel
Tamuz, O., Liu, C., Belongie, S., Shamir, O., and Kalai, A. Adaptively learning the crowd kernel. In International Conference on Machine Learning (ICML) , pages 673--680, 2011
2011
-
[51]
and von Luxburg , U
Terada, Y. and von Luxburg , U. Local ordinal embedding. In International Conference on Machine Learning (ICML) , pages 847--855, 2014
2014
-
[52]
Thurstone, L. L. A law of comparative judgment. Psychological review, 34 0 (4): 0 273, 1927
1927
-
[53]
Torgerson, W. S. Theory and Methods of Scaling. John Wiley and Sons, New York, 1958
1958
-
[54]
Crowdsourced nonparametric density estimation using relative distances
Ukkonen, A., Derakhshan, B., and Heikinheimo, H. Crowdsourced nonparametric density estimation using relative distances. In Conference on Human Computation and Crowdsourcing (HCOMP) , 2015
2015
-
[55]
and Weinberger, K
Van Der Maaten, L. and Weinberger, K. Stochastic triplet embedding. In International Workshop on Machine Learning for Signal Processing (MLSP) , pages 1--6, 2012
2012
-
[56]
Wichmann, F. A. and J\"akel, F. Methods in psychophysics. In The S tevens' H andbook of E xperimental P sychology and C ognitive N euroscience. , volume V. Methodology. Wiley, 4th edition, 2018
2018
-
[57]
A., Janssen, D
Wichmann, F. A., Janssen, D. H. J., Geirhos, R., Aguilar, G., Sch\"utt, H. H., Maertens, M., and Bethge, M. Methods and measurements to compare men against machines. Electronic Imaging, Human Vision and Electronic Imaging 2017, pages 36--45, 2017
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.