REVIEW 4 major objections 4 minor 30 references
Joint Manifold Learning and Optimal Transport for Dynamic Imaging
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Combining a low-dimensional manifold prior with a temporal optimal-transport prior improves interpolation and reconstruction of time-evolving images beyond either prior alone.
desk verdict Original combination of latent manifold learning and dynamic OT, but the barycenter definition is wrong as written and the quantitative support is modest; worth a referee after a fix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the scaled Wasserstein barycentric interpolation: for consecutive normalized frames, the decoded trajectory is forced to track the geodesic $B(\mu_{t_j}, \mu_{t_{j+1}}, (t-t_j)/(t_{j+1}-t_j))$, multiplied by a linear interpolation of the frame masses $s^i_j(t)$. This couples a Riemannian dynamical optimal-transport prior with a learned latent manifold: the encoder and decoder supply the manifold, the neural ODE supplies the latent trajectory, and the barycenter term supplies temporal consistency. Entropy-regularized solvers make the Wasserstein term differentiable, so the whole loss can be trained end-to-end.
What would settle it
Train the same model on a time series with substantial monotonic mass growth, such as cells whose total intensity roughly doubles between frames, and compare against an unbalanced-OT counterpart: if the balanced-OT model systematically underestimates intensity growth or smears the transition while the unbalanced model tracks it, the load-bearing geodesic assumption is violated.
Extended reading notes
Core claim
The central claim is that the two priors reinforce each other: OT geodesic regularization makes manifold learning respect temporal image structure, while the learned manifold steers OT-style interpolation along data-consistent paths rather than raw shortest paths. Concretely, the model encodes each frame, evolves the first latent code through a neural ODE, decodes the trajectory, and penalizes each decoded intermediate against the Wasserstein barycenter of its two neighboring frames, rescaled by a linear interpolation of the frame masses. On synthetic Gaussians and cell-division data, the OT-regularized model is the only one among the tested regularizers that keeps Gaussian shapes consistent and splits a cell cleanly; the manifold trajectory also avoids the premature split produced by raw Wasserstein interpolation.
Load-bearing premise
The OT regularizer assumes that, after normalizing each frame to unit mass, the true dynamics between consecutive frames are approximately a Wasserstein-2 geodesic, with mass changing only linearly between endpoints.
Editorial extensions
If this is right
- On sparse time series, using the OT prior yields sharper intermediate frames than penalizing the $\ell^2$ derivative of latent codes or decoded images, as shown for moving Gaussians and cell division.
- The learned manifold path gives a data-consistent alternative to plain Euclidean and Wasserstein interpolation, avoiding artifacts such as prematurely split cells.
- The combined loss is trainable end-to-end because the OT regularizer is evaluated with differentiable, entropy-regularized barycenters.
- Dynamic reconstruction metrics (MSE and SSIM) improve with OT regularization on both tested datasets, while static reconstructions are at least as good as with derivative regularizers.
- Because the manifold is learned rather than predefined, the approach removes the need for hand-designed static priors such as total variation or segmentation masks used in earlier dynamic-OT imaging.
Reading between the lines
- If the OT prior is the active ingredient for shape-preserving interpolation, then moving to unbalanced optimal transport should extend the method to genuinely growing or shrinking objects, where the linear mass interpolation is a biased constraint; the paper flags this as future work.
- The same architecture could be tested on inverse problems with a forward operator, such as sparse-view tomography or undersampled MRI, where the differentiable OT term should be compatible with a measurement consistency loss.
- A direct ablation that anneals the OT weight $\lambda$ during training could reveal whether the manifold alone eventually learns the same dynamics or whether the temporal prior is required throughout.
- The comparison against $\ell^2$ regularizers suggests a broader test: on smooth, mass-conserving dynamics OT regularization should dominate, while on strongly non-mass-conserving dynamics derivative regularizers may prove more robust.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a joint manifold-learning and optimal-transport (OT) framework for dynamic imaging. An autoencoder maps images to a low-dimensional latent space; a neural ODE evolves the latent code over time; and the decoder produces image trajectories. A dynamical OT regularizer penalizes the squared L2 distance between the decoded, mass-normalized, and mass-rescaled trajectory and the Wasserstein barycentric interpolation between consecutive observed frames, with an additional linear mass interpolation factor. Experiments on synthetic Gaussian and HeLa time-lapse data compare the OT regularizer against L2 penalties on latent or image velocities, and compare the learned model against direct L2 and Wasserstein interpolations. The paper concludes that the OT prior improves manifold learning and that the manifold improves OT interpolation.
Significance. If the stated mechanism is correct, the work offers a principled way to inject temporal OT structure into latent manifold models, backed by public code and falsifiable comparisons. The idea of combining a learned static image manifold with a dynamical OT prior is timely and relevant for scarce-data dynamic imaging. However, the paper's central mathematical definition of the barycentric interpolation is incorrect as written, and the quantitative evidence is weak; the significance of the contribution therefore hinges on correcting Definition 2 and on substantially strengthening the experimental support.
major comments (4)
- [Definition 2 (Section 3) and OT regularizer (Section 4)] Definition 2 defines the barycentric interpolation B(µ1,µ2,t) as the minimizer of (1−t)W2(µ1,µ)+tW2(µ2,µ) and asserts that this equals the W2 geodesic µt from Theorem 1. This is mathematically incorrect: with unsquared W2 distances, for t∈(0,1/2) the functional is minimized at µ=µ0 (value tW2(µ0,µ1)), while the geodesic has value 2t(1−t)W2(µ0,µ1), which is larger; the geodesic is not the minimizer. The standard Wasserstein barycenter requires squared W2 distances. Since the regularizer in Section 4 penalizes the squared L2 distance between D(z_i(t)) and the scaled barycentric interpolation B(...), the regularizer as written would pull trajectories toward boundary frames rather than along an OT geodesic. The reported experiments do not show this collapse, which indicates that the implementation necessarily uses a different definition (likely squared-W2 or Sinkhorn barycenters). This discrepancy is load-bearing: the central claim that the OT prior imposes a dynamical-geodesic prior is not supported by the equations as written. Please correct Definition 2, define the geodesic explicitly via McCann interpolation, and state precisely which numerical barycenter is used in the reported loss.
- [Section 5.2, Tables 1 and 2] The quantitative support for the central claim is weak. In Table 2 (HeLa dynamic SSIM), the OT regularizer gives 0.773 with standard deviation 0.189, while the best L2 baseline gives 0.762 with standard deviation 0.176; the difference is far below one standard deviation. In Table 1 (HeLa static SSIM), all regularizers are statistically tied (0.905 to 0.907), and the Gaussian improvements are also small. No multiple-seed variability, confidence intervals, or significance tests are reported. These results do not support the conclusion that the OT prior 'enhances manifold learning'; they are at best suggestive. The authors should report per-seed results, effect sizes, and a statistical test for the differences.
- [Section 5.3 and Conclusion] The conclusion that 'the manifold improves OT' is not supported by the comparison in Figure 4. That figure compares the full model (neural ODE latent dynamics plus OT regularizer) against direct L2 and W2 interpolations, which does not isolate the contribution of the learned manifold from the contributions of the OT prior and the neural ODE architecture. To support the attribution, the authors need an ablation of the latent neural ODE model without the OT regularizer, compared against the same model with the OT regularizer and against W2 interpolation. As presented, the improved cell-division shapes could be due to the OT prior, the latent dynamics, or the combination, rather than to the manifold prior per se.
- [Section 4 (mass interpolation) and Section 6] The OT regularizer relies on a hand-crafted linear interpolation of image masses, s_i^j(t), combined with a normalized Wasserstein barycenter. Because the datasets involve growth and division (mass creation and destruction), this balanced-OT model with a linear mass factor is a heuristic; the paper defers unbalanced OT to future work (Section 6). The reported experiments do not ablate the mass term, so it is unclear whether the observed benefit comes from the Wasserstein shape prior or from the linear mass interpolation itself. A concrete test would be to compare with a model that uses only the mass interpolation without the OT shape term, or with an unbalanced-OT formulation. As written, the contribution of the OT prior is not cleanly isolated.
minor comments (4)
- [Definition 1] The marginal conditions in the definition of Π(µ,ν) are typeset incorrectly: the expressions π(dx,y) and π(x,dy) are not well-formed; they should be written as π(dx,dy) with the appropriate integration over one variable.
- [Section 4] The notation for the mass term is inconsistent: the definition of s_i^j(t) uses parentheses in the numerator, while the regularizer uses square brackets; please reformat for clarity.
- [Section 5.2 and Section 5.3] The manuscript does not specify the numerical values of the hyperparameters (γ1, γ2, λ, latent dimension, Sinkhorn entropy) or the training details (architectures, number of epochs, number of seeds); even with code provided, the paper should state these for reproducibility.
- [Tables 1 and 2] The phrase 'average and standard deviation (between brackets)' should read 'in parentheses', since standard deviations are shown in parentheses.
Circularity Check
No circularity found: the method's central claim is empirically tested against ground truth; the sole self-citation is in related work and is not load-bearing.
full rationale
The paper's central claim is an empirical comparison: models trained with an OT regularizer are evaluated against ground-truth frames using MSE and SSIM, and manifold interpolation is compared against l2 and W2 interpolation. The OT regularizer is a soft prior computed from the model's own decoded boundary frames, but this is a standard self-consistency regularizer, not a prediction fitted to the evaluation data. No parameter is fit to the target and then renamed a prediction. The only self-citation [11] appears in Section 2.1 as related work describing a different model (RDA-INR) and is not used to justify the method or rule out alternatives. The admitted limitation in Section 6 that balanced OT ignores mass variation is a scope caveat, not a circular step. A mathematical concern exists: Definition 2's barycentric interpolation with unsquared W2 is degenerate and does not equal the W2 geodesic of Theorem 1, but this is a correctness issue about the stated regularizer, not a circularity of the derivation chain.
Assumptions & free parameters
free parameters (6)
- gamma_1 (static reconstruction weight) =
not reported
- gamma_2 (latent consistency weight) =
not reported
- lambda (OT regularizer weight) =
not reported
- latent dimension d =
not reported
- subsampling factor =
5
- Sinkhorn entropy parameter =
not reported
assumptions (5)
- standard math The 2-Wasserstein metric derivative characterization (Theorem 1) holds for the image measures used.
- domain assumption Normalizing images to unit mass and comparing via W2 is a valid image prior; temporal evolution of biological images is close to OT geodesics.
- ad hoc to paper The barycentric interpolation in Definition 2 is the correct geodesic target (requires squared W2; paper writes W2).
- domain assumption The neural ODE value function v_phi with a single initial condition captures the full per-series temporal dynamics.
- domain assumption Sinkhorn-based approximations of barycenters are accurate enough for the OT regularizer to be meaningful.
Cite this review
Pith. "Pith review of Joint Manifold Learning and Optimal Transport for Dynamic Imaging." pith.science (2026). https://pith.science/paper/ZUOPBL63
@misc{pith2026250511913,
author = {Pith},
title = {Pith review of: Joint Manifold Learning and Optimal Transport for Dynamic Imaging},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZUOPBL63}},
note = {Machine review of arXiv:2505.11913}
}
read the original abstract
Dynamic imaging is critical for understanding and visualizing dynamic biological processes in medicine and cell biology. These applications often encounter the challenge of a limited amount of time series data and time points, which hinders learning meaningful patterns. Regularization methods provide valuable prior knowledge to address this challenge, enabling the extraction of relevant information despite the scarcity of time-series data and time points. In particular, low-dimensionality assumptions on the image manifold address sample scarcity, while time progression models, such as optimal transport (OT), provide priors on image development to mitigate the lack of time points. Existing approaches using low-dimensionality assumptions disregard a temporal prior but leverage information from multiple time series. OT-prior methods, however, incorporate the temporal prior but regularize only individual time series, ignoring information from other time series of the same image modality. In this work, we investigate the effect of integrating a low-dimensionality assumption of the underlying image manifold with an OT regularizer for time-evolving images. In particular, we propose a latent model representation of the underlying image manifold and promote consistency between this representation, the time series data, and the OT prior on the time-evolving images. We discuss the advantages of enriching OT interpolations with latent models and integrating OT priors into latent models.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[11]
SIAM Journal on Imaging Sciences 17(4), 2302–2330 (2024)
Dummer, S., Strisciuglio, N., Brune, C.: Rda-inr: Riemannian diffeomorphic au- toencoding via implicit neural representations. SIAM Journal on Imaging Sciences 17(4), 2302–2330 (2024). https://doi.org/10.1137/24M1644730
-
[1]
Ambrosio, L., Gigli, N.: A User’s Guide to Optimal Transport, pp. 1–155. Springer Berlin Heidelberg, Berlin, Heidelberg (2013). https://doi.org/10.1007/978-3-642- 32160-3_1
-
[2]
Springer Science & Business Media (2005)
Ambrosio, L., Gigli, N., Savaré, G.: Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media (2005)
work page 2005
-
[3]
Master’s thesis, University of Twente (2016)
Boink, Y.E.: Combined modelling of optimal transport and segmentation revealing vascular properties. Master’s thesis, University of Twente (2016)
work page 2016
-
[4]
Optimal Transport on the Lie Group of Roto-translations
Bon, D., Pai, G., Bellaard, G., Mula, O., Duits, R.: Optimal transport on the lie group of roto-translations. arXiv preprint arXiv:2402.15322 (2024)
work page Pith review arXiv 2024
-
[5]
Foundations of Computational Mathematics23(3), 833–898 (2023)
Bredies, K., Carioni, M., Fanzon, S., Romero, F.: A generalized conditional gradi- ent method for dynamic inverse problems with optimal transport regularization. Foundations of Computational Mathematics23(3), 833–898 (2023)
work page 2023
-
[6]
Brune,C.:4Dimagingintomographyandopticalnanoscopy.Ph.D.thesis,Münster (Westfalen), Univ., Diss., 2010 (2010)
work page 2010
-
[7]
SIAM Journal on Imaging Sciences11(1), 94–128 (2018)
Burger, M., Dirks, H., Schonlieb, C.B.: A variational model for joint motion estima- tion and image reconstruction. SIAM Journal on Imaging Sciences11(1), 94–128 (2018)
work page 2018
Show all 30 references
-
[8]
In- verse Problems37(11), 115004 (2021)
Chen, C.: Spatiotemporal imaging with diffeomorphic optimal transportation. In- verse Problems37(11), 115004 (2021)
2021
-
[9]
SIAM Journal on Imaging Sciences12(4), 1686–1719 (2019)
Chen, C., Gris, B., Oktem, O.: A new variational model for joint image reconstruc- tion and motion estimation in spatiotemporal imaging. SIAM Journal on Imaging Sciences12(4), 1686–1719 (2019)
2019
-
[10]
SIAM Journal on Imaging Sciences11(1), 575–617 (2018)
Chen, C., Oktem, O.: Indirect image registration with large diffeomorphic defor- mations. SIAM Journal on Imaging Sciences11(1), 575–617 (2018)
2018
-
[12]
arXiv preprint arXiv:2404.08900 (2024)
Feng, X., Strohmer, T.: Improving autoencoder image interpolation via dynamic optimal transport. arXiv preprint arXiv:2404.08900 (2024)
2024 arXiv
-
[13]
Inverse Problems39(9), 095007 (2023)
Gao, Y., Jin, Z., Li, X.: Template-based ct reconstruction with optimal transport and total generalized variation. Inverse Problems39(9), 095007 (2023)
2023
-
[14]
In- verse Problems36(2), 025001 (2020)
Gris, B., Chen, C., Öktem, O.: Image reconstruction through metamorphosis. In- verse Problems36(2), 025001 (2020)
2020
-
[15]
SIAM Journal on Mathematics of Data Science5(2), 475–501 (2023) 12 S
Hamm, K., Henscheid, N., Kang, S.: Wassmap: Wasserstein isometric mapping for image manifold learning. SIAM Journal on Mathematics of Data Science5(2), 475–501 (2023) 12 S. Dummer, P. Vaish, and C. Brune
2023
-
[16]
arXiv preprint arXiv:2311.08549 (2023)
Hamm, K., Moosmüller, C., Schmitzer, B., Thorpe, M.: Manifold learning in wasserstein space. arXiv preprint arXiv:2311.08549 (2023)
2023 arXiv
-
[17]
Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging: Mathematical Imaging and Vision pp
Hauptmann, A., Öktem, O., Schönlieb, C.: Image reconstruction in dynamic in- verse problems with temporal models. Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging: Mathematical Imaging and Vision pp. 1–31 (2021)
2021
-
[18]
IEEE Transactions on Computational Imaging8, 693–704 (2022)
Hertrich, J., Houdard, A., Redenbach, C.: Wasserstein patch prior for image su- perresolution. IEEE Transactions on Computational Imaging8, 693–704 (2022). https://doi.org/10.1109/TCI.2022.3199600
2022
-
[19]
Applied Mathematics & Optimization 82, 1081–1109 (2020)
Lang, L.F., Neumayer, S., Öktem, O., Schönlieb, C.B.: Template-based image re- construction from sparse tomographic data. Applied Mathematics & Optimization 82, 1081–1109 (2020)
2020
-
[20]
ESAIM: Math- ematical Modelling and Numerical Analysis49(6), 1745–1769 (2015)
Maas, J., Rumpf, M., Schönlieb, C., Simon, S.: A generalized model for optimal transport of images including dissipation and density modulation. ESAIM: Math- ematical Modelling and Numerical Analysis49(6), 1745–1769 (2015)
2015
-
[21]
Nature Methods 20(7), 1010–1020 (2023)
Maška, M., Ulman, V., Delgado-Rodriguez, P., Gómez-de Mariscal, E., Nečasová, T., Guerrero Peña, F.A., Ren, T.I., Meyerowitz, E.M., Scherr, T., Löffler, K., et al.: The cell tracking challenge: 10 years of objective benchmarking. Nature Methods 20(7), 1010–1020 (2023)
2023
-
[22]
Inverse Problems35(5), 055005 (2019)
Neumayer, S., Persch, J., Steidl, G.: Regularization of inverse problems via time discrete geodesics in image spaces. Inverse Problems35(5), 055005 (2019)
2019
-
[23]
Journal of Scientific Computing98(3), 57 (2024)
Neumayer, S., Topalovic, A.: Template-based image reconstruction facing different topologies. Journal of Scientific Computing98(3), 57 (2024)
2024
-
[24]
Otto, F.: The geometry of dissipative evolution equations: the porous medium equation (2001)
2001
-
[25]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
2022
-
[26]
SIAM Journal on Imaging Sciences11(1), 643–678 (2018)
Schmitz, M.A., Heitz, M., Bonneel, N., Ngole, F., Coeurjolly, D., Cuturi, M., Peyré, G., Starck, J.L.: Wasserstein dictionary learning: Optimal transport-based unsu- pervised nonlinear dictionary learning. SIAM Journal on Imaging Sciences11(1), 643–678 (2018)
2018
-
[27]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Simon, D., Aberdam, A.: Barycenters of natural images constrained wasserstein barycenters for image morphing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7910–7919 (2020)
2020
-
[28]
In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=j8hdRqOUhN
Song, B., Kwon, S.M., Zhang, Z., Hu, X., Qu, Q., Shen, L.: Solving in- verse problems with latent diffusion models via hard data consistency. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=j8hdRqOUhN
2024
-
[29]
Nature methods14(12), 1141–1152 (2017)
Ulman, V., Maška, M., Magnusson, K.E., Ronneberger, O., Haubold, C., Harder, N., Matula, P., Matula, P., Svoboda, D., Radojevic, M., et al.: An objective com- parison of cell-tracking algorithms. Nature methods14(12), 1141–1152 (2017)
2017
-
[30]
Villani, C., et al.: Optimal transport: old and new, vol. 338. Springer (2009)
2009
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.