Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Sampling from Boltzmann densities with physics informed low-rank formats

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read A velocity field in functional tensor train format, trained on the continuity-equation residual and interleaved with Langevin and resampling steps, samples unnormalized Boltzmann densities efficiently.

desk verdict Novel TT-based Boltzmann sampler with promising numbers, but the PDE loss is not fully specified because C_t is never defined. read the letter →

arxiv 2412.07637 v1 pith:K5CGAMBX submitted 2024-12-10 cs.LG math.OC

classification cs.LGmath.OC MSC 15A6965C0565N35
keywords BoltzmannsamplingtensortrainfunctionalcontinuityequationannealingpathLangevindynamicssequentialMonteCarloresamplingalternatinglinearscheme
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to draw samples from a target density proportional to $\exp(-f_1)$, where the normalization constant is unknown. It treats sampling as a transport problem: along the annealing path $f_t = t f_1 + (1-t) f_0$, one learns a velocity field that pushes an easy latent density $p_0$ onto the target. The velocity field is stored in a functional tensor train format, a low-rank tensor-product representation with a Fourier basis, and the continuity equation becomes a residual minimized by an alternating linear scheme. To keep all modes populated, the authors interleave deterministic transport steps with stochastic Langevin and resampling steps borrowed from sequential Monte Carlo. If the method works as reported, it gives a parameter-light, PDE-based sampler that avoids the mode collapse of reverse-KL normalizing flows; their best many-well experiment reaches energy distance $1.9\times 10^{-3}\pm 2\times 10^{-4}$.

What carries the argument

The central object is the functional tensor train (FTT), a tensor-train coefficient tensor contracted with univariate basis functions, here an $H^2$-orthonormalized Fourier basis. The FTT represents the velocity field $v_t$ with cores that can be optimized one at a time; for each core, the PDE residual (6) is linear in the core entries, so the update reduces to solving a small quadratic program. Two auxiliary mechanisms carry the argument: an exponential moving average of successive TT approximations with TT-SVD truncation makes the ranks adaptive, and the alternating sequence of ODE transport, resampling, and Langevin steps keeps all modes of the target populated during training and evaluation.

What would settle it

Run Algorithm 1 on a two-dimensional Gaussian mixture with two modes of equal weight whose centers are separated by a large energy barrier, starting $p_0$ concentrated on only one mode; compute the fraction of final samples in the neglected mode over many trials. The central claim predicts both modes are populated in roughly equal proportion and a small energy distance, so a reproducible run where the second mode remains empty would disprove the method's mode-coverage claim.

Watch

Extended reading notes

Core claim

The central claim is that the continuity equation governing the annealing path can be solved directly in a functional tensor train (FTT) format, and that the resulting velocity field, when evaluated by numerical ODE integration, yields accurate samples from the Boltzmann density without ever evaluating the normalization constant. Specifically, for $f_t = t f_1 + (1-t) f_0$, the authors show the continuity equation reduces to the scalar condition $f_1 - f_0 + \langle \nabla f_t, v_t\rangle - \nabla\cdot v_t + C_t = 0$, with $C_t = \partial_t \log Z_t$. The paper approximates $v_t$ as a low-rank FTT with $H^2$-orthonormalized Fourier basis functions, and trains it by minimizing the empirical squared residual (6) with ALS, where each core update is a quadratic program. Stochastic steps are added after transport: resampling reweights particle populations to handle mode-weight teleportation near $t=1$, and Langevin steps prevent collapse to point masses. The reported energy distances, for example $1.9\times 10^{-3}\pm 2\times 10^{-4}$ for the many-well $f^{4,8}_1$ with flow+stochastic, support the claim that the combination works.

Load-bearing premise

The load-bearing premise is that samples drawn from the current trained flow adequately cover the regions where the target density has mass, because the PDE residual is only evaluated on those samples; if the flow has not yet discovered a mode, the training signal never penalizes the velocity field there.

Editorial extensions

If this is right

  • A user only needs the unnormalized target energy $f_1$ and an easy latent density $p_0$; no target samples and no normalizing constant are required.
  • The TT ranks are not fixed in advance: the EMA update with TT-SVD truncation adapts them during training.
  • The $H^2$-orthonormalized Fourier basis is what makes the PDE residual tractable; polynomial bases perform poorly, as stated in the conclusions.
  • For high-dimensional many-well targets, the full pipeline flow+stochastic reaches energy distances an order of magnitude smaller than the pure TT flow, showing the stochastic steps are necessary for good mode coverage.
  • Because every core update is a linear least-squares problem, training avoids the backpropagation and stability issues of physics-informed neural network solvers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit in the paper: because the residual is evaluated on samples from the currently trained flow, the velocity field is unconstrained in regions the flow has not yet discovered; a target mode outside the initial latent support could be invisible to the PDE loss.
  • A testable extension would be to add a small number of uniform or adaptive collocation points to residual (6); if this improves performance on separated modes, it would confirm that the residual's sample support, not the FTT expressivity, is the limiting factor.
  • The reported experiments are two-dimensional; whether the FTT ranks remain manageable in higher dimensions is open, since TT ranks for the annealed flow can in principle grow with the complexity of the target.
  • The effect of Langevin steps on the resampling proposal density $q$ is left unassessed in the paper; a correction that accounts for this bias could make the method asymptotically exact rather than approximate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes sampling from an unnormalized Boltzmann density e^{-f_1}/Z_1 by approximating the velocity field of the continuity equation for the annealing path f_t = t f_1 + (1-t) f_0. The velocity field is represented as a functional tensor train (FTT) with an H^2-orthonormalized Fourier basis, and the coefficients are fitted by an alternating linear scheme (ALS) applied to the empirical squared residual of the logarithmic continuity equation. Training is performed on samples transported by the current approximation (ODEsolve), with optional Langevin and resampling steps inspired by sequential Monte Carlo, and TT ranks are adjusted adaptively by an exponential moving average with SVD truncation. The paper reports energy distances for Gaussian mixture and many-well targets, comparing a pure flow, a flow with post-processing, a flow with stochastic steps during training, and a stochastic-only baseline. The best reported result is an energy distance of 1.9e-3 +/- 2e-4 for the many-well problem f^{4,8}_1 with the flow+stochastic method (Table 1).

Significance. If the method works as claimed, it is a useful alternative to neural-network PDE samplers: the FTT parameterization avoids backpropagation through networks, the H^2 Fourier basis is a principled choice for PDE residuals, and the rank-adaptive ALS update is elegant. The ablation study in Table 1 separates the contributions of the flow, resampling, and Langevin steps and includes standard deviations. However, the main algorithmic claim is not yet supported because the normalizing-constant term C_t entering the objective is never specified or estimated, and because the stochastic corrections are explicitly left unanalyzed. The numerical experiments are therefore not reproducible in their current form, and the reported accuracy cannot yet be attributed to a solved continuity equation with certainty.

major comments (3)
  1. [Section 2, Eq. (6); Algorithm 1] The empirical loss J_{t,N} in Eq. (6) contains the term C_t = d/dt log Z_t, but the manuscript never states how C_t is computed or estimated for the unnormalized targets. Algorithm 1 lists no C_t computation or update, and the statement in Section 5 that the ground truth is known refers to validation samples, not to the training objective. This is a load-bearing omission: if C_t is replaced by a wrong constant, the residual in Eq. (3) changes by a spatially constant function, which can be absorbed by a divergent component of div v, so the learned velocity field need not solve the intended continuity equation. Until C_t is explicitly defined, estimated, and shown to enter the implemented objective, the central claim that the method solves the continuity equation in the TT format is unsupported.
  2. [Section 4.1] Sampling the empirical loss from the current trained flow means the PDE residual is only penalized on regions that already carry mass under the current approximation. Section 4.1 adopts this mode-seeking strategy from [19] without discussing its coverage failure mode: a target mode that is not discovered by the transported samples is never seen by the ALS residual, and no mechanism in Algorithm 1 provably discovers it. The Langevin and resampling steps are proposed as a mitigation, but their effect is heuristic. A concrete test would be to initialize from a prior that misses a well-separated mode and report whether flow+stochastic recovers it; without such a test or an analysis, the method's exploration behavior for multimodal Boltzmann densities is not established.
  3. [Section 4.1, Algorithm 1] The resampling step uses importance weights w^{(l)} = p(x^{(l)})/q(x^{(l)}), where q is obtained from the flow transport T_k only. Algorithm 1 nevertheless applies Langevin steps before and after resampling, and the authors explicitly state that the effect of the Langevin layers on the density q is left for future work and that only the flow layers are used. Hence the proposal density used in the weights is not the density of the samples that are actually resampled, so the stochastic correction can introduce an uncontrolled bias. Since the stochastic steps are presented as the mechanism that adjusts the relative mode weights and anneals to the correct path distribution, this limitation is load-bearing and should be addressed either by including the Langevin transition in q or by reporting a sensitivity analysis.
minor comments (5)
  1. [Section 2, Eq. (3)] The inner product notation in Eq. (3) is missing a comma: it should read <nabla f_t, v_t> rather than <nabla f_t v_t>.
  2. [Section 4.2] The UpdateTT procedure depends on an SVD truncation tolerance and an EMA weighting factor, but neither quantity is specified in Section 5, so the rank-adaptivity claim is not reproducible.
  3. [Section 5] The number of Langevin steps, the Langevin step size h, and the exact Fourier basis setup (n, orthonormalization domain) are not reported for each experiment, and no runtime or TT-rank trajectories are given.
  4. [Section 5, stochastic baseline] The description of the stochastic baseline as taking 'the ratio for the resampling step to be p_{k+1}/p_k' is ambiguous, since for Boltzmann densities this ratio contains the unknown normalizing constants Z_k/Z_{k+1}; the baseline should state explicitly whether an unnormalized density ratio is used.
  5. [Section 6] The claim that polynomial bases perform poorly is not supported by a dedicated experiment or a comparison in Table 1, so it reads as speculation rather than a reported finding.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the continuity-equation residual and the energy-distance evaluation are independent; C_t is a completeness gap, not a circular fit.

full rationale

The derivation chain is not circular. The continuity-equation residual in Eq. (3) is derived from the annealing path ft = t f1 + (1-t) f0 and the Boltzmann density definition, and Eq. (6) is an empirical least-squares version of that residual. The reported metric, energy distance between drawn samples and ground-truth samples in Table 1, is external to the training loss and is not a fitted parameter. No equation identifies a predicted quantity with an input: the velocity field is optimized with ALS, samples are transported by ODEsolve and optionally resampled/Langevin-corrected, and the metric is evaluated on those samples. The self-citations ([6], [15], [22], [33]) are used as background, for the annealing-path/teleportation motivation, and for Langevin step convergence; they are not used to define the reported predictions or to forbid alternative constructions. The reviewer's concern that C_t = d/dt log Z_t is never computed is an implementational/completeness gap in the stated objective, not a circular reduction: C_t is a term in the PDE, not a fitted version of the final energy distance, and the paper does not claim to fit C_t from the metric. The acknowledged limitations, such as leaving the effect of Langevin layers on q for future work and lacking ground-truth-free tuning, weaken guarantees but do not make the derivation self-referential.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard existence results from optimal transport, plus several modeling assumptions about the training distribution, basis expressivity, and stochastic-step corrections that the paper does not fully analyze. The free parameters are predominantly hyperparameters of the adaptive scheme and stochastic steps, none of which are reported with concrete values.

free parameters (6)
  • TT rank truncation threshold (SVD tolerance) in UpdateTT = not reported
    Controls the adaptive TT ranks after the EMA update; directly trades off approximation error against rank and cost.
  • EMA weighting factor in UpdateTT = not reported
    Blends the newly trained TT with the previous one; affects training stability and rank growth.
  • Number of Fourier basis coefficients n = n <= 10 for problems 1 and 3; n = 41 for problem 2
    Basis size determines the representable detail of the velocity field.
  • Langevin step size h and number of steps = not reported
    Used in the stochastic steps before and after resampling; values are tuned but not given.
  • Time discretization M = M <= 35
    Number of annealing steps; the vector field is learned on this time grid and the flow is integrated across it.
  • Basis orthonormalization domain = [-5,5]^d for problems 1 and 3; [-50,50]^2 for problem 2
    The H2-orthonormalized Fourier basis is defined on this box; the box must cover the target support to represent the flow accurately.
assumptions (5)
  • standard math Existence of a velocity field satisfying the continuity equation along the annealing path (Eq 3)
    The paper cites [6] and [2, Theorem 8.1.8] for existence of absolutely continuous curves and Borel vector fields satisfying the continuity equation; this is a standard optimal transport result.
  • domain assumption Training samples drawn from the current flow cover the support of the true path distribution
    The empirical loss (6) is minimized on samples pushed by the current learned flow; the method assumes this drives the residual to zero where the true distribution has mass.
  • domain assumption Langevin dynamics converge asymptotically to the target for the given energies
    Section 4.1 invokes [8,22,38], which require strong convexity and Lipschitz gradient assumptions; the many-well target is not strongly convex, so the assumption may be violated in practice.
  • ad hoc to paper The H2 orthonormal Fourier basis accurately represents the velocity field for these targets
    Section 6 says the Fourier basis works well while polynomials perform poorly; this is an empirical choice not theoretically justified.
  • domain assumption Resampling density q is computed from flow layers only, ignoring Langevin modifications
    Section 4.1 explicitly states that the effect of Langevin steps on q is left for future work and only flow layers are used; the implicit assumption is that the flow-only pushforward is an adequate approximation of the sampling distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sampling from Boltzmann densities with physics informed low-rank formats." pith.science (2026). https://pith.science/paper/K5CGAMBX

@misc{pith2026241207637,
  author       = {Pith},
  title        = {Pith review of: Sampling from Boltzmann densities with physics informed low-rank formats},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5CGAMBX}},
  note         = {Machine review of arXiv:2412.07637}
}
read the original abstract

Our method proposes the efficient generation of samples from an unnormalized Boltzmann density by solving the underlying continuity equation in the low-rank tensor train (TT) format. It is based on the annealing path commonly used in MCMC literature, which is given by the linear interpolation in the space of energies. Inspired by Sequential Monte Carlo, we alternate between deterministic time steps from the TT representation of the flow field and stochastic steps, which include Langevin and resampling steps. These adjust the relative weights of the different modes of the target distribution and anneal to the correct path distribution. We showcase the efficiency of our method on multiple numerical examples.

Figures

Figures reproduced from arXiv: 2412.07637 by the authors.

Figure 1
Figure 1. GM density sampled with the different methods. While the methods perform similarly [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Histogram corner plots showing the 1-d marginals on the diagonal and the 2-d joint [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 27 canonical work pages

  1. [19]

    Máté and F

    B. Máté and F. Fleuret. Learning interpolations between boltzmann densities.Transactions on Machine Learning Research, 2023

  2. [1]

    Albergo, G

    M. Albergo, G. Kanwar, and P. Shanahan. Flow-based generative models for Markov chain Monte Carlo in lattice field theory.Physical Review D, 100(3), Aug. 2019

  3. [2]

    Ambrosio, N

    L. Ambrosio, N. Gigli, and G. Savaré.Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2008

  4. [3]

    Arbel, A

    M. Arbel, A. Matthews, and A. Doucet. Annealed flow transport Monte Carlo. InInternational Conference on Machine Learning, pages 318–330. PMLR, 2021

  5. [4]

    Bandeira, A

    A. Bandeira, A. Maillard, R. Nickl, and S. Wang. On free energy barriers in gaussian priors and failure of cold start MCMC for high-dimensional unimodal distributions.Philosophical transactions. Series A, Mathematical, physical, and engineering sciences, 381:20220150, 03 2023

  6. [5]

    Berner, L

    J. Berner, L. Richter, and K. Ullrich. An optimal control perspective on diffusion-based gen- erative modeling. Transactions on Machine Learning Research, 2024

  7. [6]

    Chemseddine, C

    J. Chemseddine, C. Wald, R. Duong, and G. Steidl. Neural sampling from Boltzmann densities: Fisher-Rao curves in the Wasserstein geometry.arXiv preprint arXiv:2410.03282, 2024

  8. [7]

    Chuang and L

    P.-Y. Chuang and L. A. Barba. Experience report of physics-informed neural networks in fluid simulations: pitfalls and frustration.arXiv preprint arXiv:2205.14249, 2022

Show all 38 references
  1. [8]

    A. S. Dalalyan and A. G. Karagulyan. User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient.ArXiv, abs/1710.00095, 2017. 10

  2. [9]

    Del Debbio, J

    L. Del Debbio, J. Marsh Rossney, and M. Wilson. Efficient modeling of trivializing maps for lattice ϕ4 theory using normalizing flows: A first look at scalability.Physical Review D, 104(9), Nov. 2021

  3. [10]

    Del Moral, A

    P. Del Moral, A. Doucet, and A. Jasra. Sequential Monte Carlo samplers.Journal of the Royal Statistical Society Series B: Statist. Method., 68(3):411–436, 2006

  4. [11]

    Dolgov, D

    S. Dolgov, D. Kalise, and K. K. Kunisch. Tensor decomposition methods for high-dimensional Hamilton–Jacobi–Bellman equations. SIAM Journal on Scientific Computing, 43(3):A1625– A1650, 2021

  5. [12]

    Doucet, S

    A. Doucet, S. Godsill, and C. Andrieu. On sequential monte carlo sampling methods for bayesian filtering.Statistics and Computing, 10:197–208, 2000

  6. [13]

    Eigel, J

    M. Eigel, J. Neumann, R. Schneider, and S. Wolf. Non-intrusive tensor reconstruction for high-dimensional random pdes.Computational Methods in Applied Mathematics, 19(1):39–53, 2019

  7. [14]

    Eigel, R

    M. Eigel, R. Schneider, and D. Sommer. Dynamical low-rank approximations of solutions to the Hamilton–Jacobi–Bellman equation.Numerical Linear Algebra with Applications, 30(3):e2463, 2023

  8. [15]

    Hagemann, J

    P. Hagemann, J. Hertrich, and G. Steidl. Stochastic normalizing flows for inverse problems: A Markov chains viewpoint.SIAM/ASA Journal on Uncertainty Quantification, 10(3):1162–1190, 2022

  9. [16]

    Hertrich and R

    J. Hertrich and R. Gruhlke. Importance corrected neural JKO sampling. arXiv preprint arXiv:2407.20444, 2024

  10. [17]

    Thealternatinglinearschemefortensoroptimization in the tensor train format.SIAM Journal on Scientific Computing, 34(2):A683–A713, 2012

    S.Holtz, T.Rohwedder, andR.Schneider. Thealternatinglinearschemefortensoroptimization in the tensor train format.SIAM Journal on Scientific Computing, 34(2):A683–A713, 2012

  11. [18]

    Holtz, T

    S. Holtz, T. Rohwedder, and R. Schneider. On manifolds of tensors of fixed TT-rank.Nu- merische Mathematik, 120(4):701–731, 2012

  12. [20]

    A. G. D. G. Matthews, M. Arbel, D. J. Rezende, and A. Doucet. Continual repeated annealed flow transport Monte Carlo. InProceedings of the 39th Int. Conference on Machine Learning, Proceedings of Machine Learning Research, Jul 2022

  13. [21]

    L. I. Midgley, V. Stimper, G. N. C. Simm, B. Schölkopf, and J. M. Hernández-Lobato. Flow annealed importance sampling bootstrap.arXiv preprint arXiv:2208.01893, 2023

  14. [22]

    Miranda, J

    C. Miranda, J. Schütte, D. Sommer, and M. Eigel. Approximating Langevin Monte Carlo with ResNet-like neural network architectures.arXiv preprint arXiv:2311.03242, 2024

  15. [23]

    R. M. Neal. Annealed importance sampling.Statist. and Comp., 11:125–139, 2001. 11

  16. [24]

    Nouy.Chapter 4: Low-Rank Methods for High-Dimensional Approximation and Model Order Reduction, page 171–226

    A. Nouy.Chapter 4: Low-Rank Methods for High-Dimensional Approximation and Model Order Reduction, page 171–226. Society for Industrial and Applied Mathematics, July 2017

  17. [25]

    F. Noé, S. Olsson, J. Köhler, and H. Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning.Science, 365(6457), 2019

  18. [26]

    I. V. Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing , 33(5):2295–2317, 2011

  19. [27]

    I. V. Oseledets. Constructive representation of functions in low-rank tensor formats.Construc- tive Approximation, 37:1–18, 2013

  20. [28]

    Papamakarios, E

    G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Nor- malizing flows for probabilistic modeling and inference.Journal of Machine Learning Research, 22(57):1–64, 2021

  21. [29]

    Raissi, P

    M. Raissi, P. Perdikaris, and G. Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Comp. Phys., 378:686–707, 2019

  22. [30]

    Richter, L

    L. Richter, L. Sallandt, and N. Nüsken. Solving high-dimensional parabolic PDEs using the ten- sor train format. InInternational Conference on Machine Learning, pages 8998–9009. PMLR, 2021

  23. [31]

    Rohwedder and A

    T. Rohwedder and A. Uschmajew. On local convergence of alternating schemes for optimiza- tion of convex problems in the tensor train format.SIAM Journal on Numerical Analysis, 51(2):1134–1162, 2013

  24. [32]

    Sejdinovic, B

    D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-based and RKHS-based statistics in hypothesis testing.The Annals of Statistics, 41(5):2263 – 2291, 2013

  25. [33]

    Sommer, R

    D. Sommer, R. Gruhlke, M. Kirstein, M. Eigel, and C. Schillings. Generative modelling with tensor train approximations of Hamilton–Jacobi–Bellman equations. arXiv preprint arXiv:2402.15285, 2024

  26. [34]

    J. Sun, J. Berner, L. Richter, M. Zeinhofer, J. Müller, K. Azizzadenesheli, and A. Anand- kumar. Dynamical measure transport and neural PDE solvers for sampling.arXiv preprint arXiv:2407.07873, 2024

  27. [35]

    G. J. Székely and M. L. Rizzo. Energy statistics: A class of statistics based on distances. Journal of Statistical Planning and Inference, 143(8):1249–1272, 2013

  28. [36]

    S. K. Vemuri, T. Büchner, J. Niebling, and J. Denzler. Functional tensor decompositions for physics-informed neural networks.arXiv preprint arXiv:2408.13101, 2024

  29. [37]

    H. Wu, J. Köhler, and F. Noé. Stochastic normalizing flows.Advances in Neural Information Processing Systems, 33:5933–5944, 2020

  30. [38]

    Zhang, Ö

    Y. Zhang, Ö. D. Akyildiz, T. Damoulas, and S. Sabanis. Nonasymptotic estimates for stochas- tic gradient Langevin dynamics under local conditions in nonconvex optimization. Applied Mathematics & Optimization, 87(2):25, 2023. 12

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.