Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

This paper claims that consistency-model instability in scratch training stems from guiding the flow map by conditional rather than marginal velocity, and that a reformulated self-distillation fixes it, making pretrained diffusion precondit

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:28 UTC pith:EWMHUJTT

load-bearing objection Useful training recipe with a shaky theoretical core; don't cite the theory until Prop 4.2 and the LSD-R fixed-point claims are fixed. the 3 major comments →

arxiv 2601.22679 v2 pith:EWMHUJTT submitted 2026-01-30 cs.LG cs.CV

Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation

Youngjoong Kim , Duhoe Kim , Woosung Kim , Jaesik Park This is my paper
classification cs.LG cs.CV MSC 68T07
keywords consistency modelsflow mapsself-distillationtraining instabilitymarginal velocityfew-step generationclassifier-free guidancegenerative modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that consistency models—generators that map any point on a probability-flow trajectory straight to the data—become unstable when trained from scratch because the standard objective guides them with the conditional velocity of a single data sample instead of the marginal velocity of the true flow. That mismatch, the authors argue, creates degenerate optima and explains the large loss variance and seed-to-seed irreproducibility seen in practice. As a remedy, they reformulate self-distillation so the network learns from its own marginal velocity, relax the time condition, and add classifier-free guidance during training. If the claim is right, few-step generative models can be trained from random initialization without a pretrained diffusion backbone, and training is reproducible across seeds. The method is demonstrated on ImageNet-1K and on diffusion-based robot-policy learning.

Core claim

The paper's central claim is that the optimum of 'direct training'—Eulerian distillation under a conditional velocity field—is not the true flow map. An Euler–Lagrange analysis shows the optimum satisfies a first-order condition with a divergence term involving the conditional covariance of the velocity error, ∇·(Σ_{Δv|x} ∇f)=0, so a learned map can satisfy the objective by flattening its gradient rather than by reproducing the true flow. Consistency training, the paper argues, has the Eulerian equation as a fixed point but lacks the second-order structure needed to converge to it, which is why small batches, random weight init, and continued training drift toward degenerate maps. The remedy

What carries the argument

The central object is the flow map f_{t,s}(x_t), the mapping that transports a point x_t along a probability-flow trajectory back to x_s, represented as f_θ(x_t;t,s)=ν^{-1}(A'_{t,s}x_t - A_{t,s}F_θ(...)). The argument hinges on the distinction between the conditional velocity v_t(x_t|x) of an individual sample and the marginal velocity v*_t(x_t) that defines the true flow: the paper shows the former biases the objective (via a conditional-covariance divergence term in the Euler–Lagrange condition), while the latter yields the desired optimum. Fixing the instability then turns on a reformulated self-distillation loss (LSD-R) that regresses the network onto a stop-gradient self-target driven b

Load-bearing premise

The diagnosis that conditional-velocity training is degenerate rests on a non-standard Euler–Lagrange derivation that swaps derivatives and expectations to obtain the condition ∇·(Σ_Δv|x ∇f)=0; if that derivation is not valid, the paper's central explanation of instability collapses.

What would settle it

On a small Gaussian-mixture dataset where marginal and conditional velocities are known in closed form, numerically minimize the direct-training objective LDT with a rich enough network; if the global optimum satisfies the Eulerian equation (zero residual) rather than the paper's predicted condition with a nonzero divergence term, then Proposition 4.2 is refuted, and the instability must have a different cause.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Consistency models for few-step generation can be trained from random initialization, removing the need for pretrained diffusion preconditioners and making results reproducible across seeds.
  • Plain consistency training guided by conditional velocity is predicted to have a degenerate global optimum, meaning the bias is not merely a finite-batch or initialization effect.
  • Small batch sizes shift consistency training toward direct training, so the observed instability and degenerate solutions should worsen as batch size shrinks; large batch sizes mask the problem.
  • The time-condition relaxation s<t reduces the linearization cost and is the key ingredient that lets self-distillation and flow matching take over, explaining why recent scratch-trained methods succeed where fixed-s=0 training fails.
  • Applying self-distillation with Pre-CFG yields flow maps that follow the classifier-free-guidance velocity field; naive Post-CFG extrapolation does not, so training-time guidance is the correct recipe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the marginal-velocity diagnosis is correct, then any consistency-style objective that conditions on per-sample paths—including variants with learned couplings—should exhibit a similar bias, and the paper's stop-gradient self-distillation fix could serve as a general stabilization module for those methods.
  • The theory suggests a sharp, checkable prediction: on a toy distribution with known conditional covariance, direct training should converge to a map with nonzero Eulerian residual; a careful numerical optimization should exhibit the predicted degeneracy rather than the true flow map.
  • The result hints that the instability of consistency models is not primarily an optimization-difficulty phenomenon but an optimum-shift phenomenon; if so, increasing model capacity or training time alone will not fix it, and guidance by marginal velocity is necessary.
  • Extending iSD-style marginal-guidance training to other flow-map learners (e.g., in video, audio, or offline RL) is a natural next step, since the same conditional-vs-marginal gap appears wherever flow maps are learned from paired samples.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper develops a flow-map-based analysis of consistency models, arguing that (i) direct training with conditional velocity has degenerate optima (Proposition 4.2), (ii) consistency training admits fixed points satisfying the Eulerian equation but lacks second-order curvature needed for convergence (Proposition 4.3), and (iii) a reformulated self-distillation objective, iSD/iSD-T, provides stable training from random initialization by aligning with the marginal velocity and optionally with a classifier-free guidance trajectory. The empirical section reports ImageNet-1K 256×256 FID improvements from a consistency-training baseline of 120.5 to 15.20 with iSD-T, lower variance across seeds than baselines, and extensions to CelebA-HQ and diffusion-based policy learning. The paper includes detailed ablations, loss landscapes, and algorithmic descriptions.

Significance. If the theoretical analysis were correct, the paper would make a substantial contribution: it would unify several consistency/self-distillation methods under a flow-map representation and explain why direct conditional-velocity training can fail while marginal-velocity training succeeds. The practical recipe is also valuable: the reported variance reduction and training-from-scratch capability directly address a known reproducibility problem in consistency models. Strengths include controlled comparisons with fixed backbone/data/optimizer, multiple seeds with reported standard deviations, detailed hyperparameters and algorithms, and ablations isolating time-condition relaxation, joint flow matching, and Pre-CFG. The loss-landscape analysis is a useful empirical diagnostic. However, the central theoretical claims rest on a variational derivation in Appendix A.6 that is, as written, not valid, and the fixed-point argument for the final iSD objective is imported from a theorem about a different objective. The empirical results are credible and the gaps appear addressable, so the contribution is currently stronger as an empirical study than as a theory paper.

major comments (3)
  1. [A.6, Proposition 4.2, Eq. (10)] The Euler–Lagrange derivation of the optimality condition is not mathematically valid as written. The stationarity condition for L_DT = E[||∂_t f + v·∇f||^2] is ∂_t E[R] + ∇·E[R v] = 0 with R = ∂_t f + v·∇f. The manuscript replaces E[∂_t R] with ∂_t(L^*f) without accounting for the t-dependence of the conditional distribution p(x|xt), and then writes the adjoint step as L^*L^*f − ∇·(Σ∇f)=0 ⇔ ||L^*f||_2^2 + ∇·(Σ∇f)=0. This conflates a differential operator applied to f with an L^2 norm, drops the adjoint sign, and is dimensionally inconsistent. Consequently Proposition 4.2, which is the paper's central diagnosis of degenerate optima under direct training, is unsupported. The toy experiment in Figure 2 is suggestive but cannot substitute for a correct variational statement.
  2. [A.5/A.7, Prop. 4.3] The claim that the continuous-time consistency objective 'reduces in value to Eulerian distillation' is incorrect as stated. The derivation gives L_CT = E[ f·(L^*f^-) + f·(Δv·∇f^-) ], but Eulerian distillation is E[||L^*f||^2], not E[f·L^*f]. The two have different gradients even with a detached target, since ∇_θ E[f·L^*f^-] = E[∇f·L^*f^-], whereas the Eulerian gradient is E[2 L^*f·∇(L^*f)]. The fixed-point condition E[L^*f]=0 may still be derivable from the linear objective, but the 'reduces to ED' wording and the resulting discussion of second-order structure need to be corrected. This also affects the interpretation of small-batch behavior in Section 4.2 and the summary paragraph calling Theorem 4.2/4.3.
  3. [Section 5.2, Eq. (13)–(15)] The assertion that LSD-R 'admits the desired fixed points (Theorem 4.3)' is not supported by the cited theorem. Proposition 4.3 analyzes the continuous-time consistency objective L_CT with conditional velocity v_t(x_t|x); it does not analyze the self-distillation target F_tgt = F_θ + A''_{t,s} x_t + A'_{t,s}(v_θ−F_θ) − A_{t,s} dF_θ/dt, which uses the network's own instantaneous velocity v_θ = F_θ(x_t;t,t) under joint flow-matching training. A separate fixed-point or stationarity analysis for Eq. (13) is needed, or the theoretical claim should be weakened. The strong empirical results do not depend on this specific theorem, but the paper's theoretical framing of iSD does.
minor comments (5)
  1. [General] The text refers to 'Theorem 4.2' and 'Theorem 4.3' in the Section 4 summary and Section 5.2, but the paper only contains Proposition 4.2 and Proposition 4.3. Please correct the cross-references.
  2. [Algorithm 1] In Algorithm 1, the velocity assignment is written as 'vt ← α'_t x + σ'_t x'; the second term should presumably be σ'_t z. Please fix this typo.
  3. [Eq. (12)] The displayed equation for the reformulated self-distillation loss has malformed norm delimiters. Please clean up the LaTeX/rendering.
  4. [A.6] The dichotomy in the final paragraph of A.6 ('if this scenario is admitted... otherwise...') is not a proof; it is an informal case analysis. If retained, it should be clearly labeled as heuristic, not as part of the proposition proof.
  5. [Table 1 / Section 5.2] The table and text use inconsistent notation for the iSD objective: Table 1 lists 'L_SD-R' while Section 5.2 defines 'LSD-R' and 'LiSD'. Please unify notation.

Circularity Check

0 steps flagged

No circularity: the derivation chain is self-contained; flagged issues are proof gaps and citation-overreach, not fitted parameters renamed as predictions.

full rationale

No circular step meets the quoted-reduction threshold. The paper's central theoretical claims (Prop. 4.2 and Prop. 4.3) are derived from the stated objectives (Eqs. 4, 6, 9) by the paper's own variational arguments in Appendices A.6 and A.7; the correctness of those manipulations is a proof-validity question, not a circularity question. The iSD objective (Eq. 15) is obtained by algebraic reformulation of the consistency-training gradient in Appendix B.2 and by replacing the marginal velocity with the network's own instantaneous velocity while jointly training with flow matching; it is not a fitted value that is then relabeled as a prediction. Empirical claims are checked against external FID/IS benchmarks and controlled baselines (Table 5) with fixed backbone/data/training budget, and the paper reports multiple random initializations and standard deviations. Two support gaps should be flagged for the correctness review but do not constitute circularity: (1) Section 5.2 says LSD-R "admits the desired fixed points (Theorem 4.3)", yet Theorem 4.3 is stated for consistency training with conditional velocity, so the transfer to the self-distillation objective is asserted rather than proved; (2) Appendix A.6's Euler-Lagrange step contains a nonstandard operator manipulation that may be invalid, undermining Proposition 4.2 as written. Neither gap is a case of a prediction being equivalent to its inputs by construction, nor is the central claim carried by a self-citation chain: the cited consistency-model frameworks (Kim et al. 2024; Boffi et al. 2025a,b; Sabour et al. 2025) are used as context, while the load-bearing convergence justification for iSD is given in the paper's own Appendix B.1. Hyperparameters such as the guidance scale and JVP step are tuned on validation FID but are not used to derive the theory nor disguised as theoretical predictions.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central method rests on standard flow-matching assumptions (Lipschitz marginal velocity, interpolation with constant ν) and on two non-standard analytical steps: the Euler-Lagrange optimality condition in A.6 and the fixed-point argument in A.7/Section 5.2. No new entities are invented; the primary free parameters are hyperparameters (ω, ε, loss weights) tuned on validation data.

free parameters (3)
  • Pre-CFG guidance scale ω = 5.0 (ImageNet DiT-B/4), 3.0 (DiT-B/2, XL/2), 5.0 (XL/2†, XL/1†)
    Selected per architecture and dataset via ablation (Table 11); the best value does not transfer across scales and is a key hyperparameter of iSD-T.
  • JVP approximation step ε = 0.005
    Chosen by hand; the finite-difference approximation of the JVP has O(ε²) error and is used in all experiments.
  • Loss weighting scheme and parameters = adaptive (η=0.01, p=1.0) or cosine
    Weighting of L_CFM and L_SD-R is a design choice tuned per setting (SD-VAE vs VA-VAE); adaptive parameters η and p are set by hand.
axioms (5)
  • domain assumption The interpolation functions satisfy α_t σ'_t - σ_t α'_t = ν ≠ 0 for all t, with ν constant (linear and trigonometric interpolations satisfy ν=1).
    Required for the flow map representation f = ν^{-1}(A'x - AF) and for the claim that a time-dependent ν would make the global optimum time-dependent (Appendix A.4).
  • standard math The velocity error Δv = v_t(x_t|x) - v*_t(x_t) satisfies E_{x|x_t}[Δv]=0 (conditional velocity is an unbiased estimator of the marginal velocity).
    Used throughout Section A.6 and A.7 to decompose L_DT and L_CT into an Eulerian part plus a variance/covariance term; standard tower property of conditional expectation.
  • ad hoc to paper The Euler–Lagrange equation provides necessary and sufficient stationary conditions for the direct-training objective L_DT.
    The derivation in A.6 assumes the conditional expectation can be treated as a Lagrangian and that operator manipulations (L*L*f, ∇·(Σ∇f)) are valid; this is not standard and appears erroneous, so it is an ad hoc assumption of the paper's analysis.
  • ad hoc to paper For fixed-point analysis, the stop-gradient objective L_CT can be studied by its Euler-Lagrange equation, and absence of quadratic curvature implies failure to converge.
    Section A.7: the paper states that L_CT is linear in f and therefore has vanishing Hessian at the fixed point, and that this prevents convergence; this is asserted rather than derived rigorously.
  • domain assumption The network has sufficient capacity to approximate the unconditional marginal velocity F_θ(x_t;t,∅) ≈ v*_t(x_t) in the Pre-CFG proof.
    Appendix A.10 uses this to replace F_∅ with v* in the tower-property derivation; the paper states 'if the network has sufficient capacity' but this is not verified.

pith-pipeline@v1.3.0-alltime-deepseek · 40144 in / 18091 out tokens · 183389 ms · 2026-08-03T06:28:44.455718+00:00 · methodology

0 comments
read the original abstract

Consistency models have been proposed for fast generative modeling, achieving results competitive with diffusion and flow models. However, these methods exhibit inherent instability and limited reproducibility when training from scratch, motivating subsequent work to explain and stabilize these issues. While these efforts have provided valuable insights, the explanations remain fragmented, and the theoretical relationships remain unclear. In this work, we provide a theoretical examination of consistency models by analyzing them from a flow map-based perspective. This joint analysis clarifies how training stability and convergence behavior can give rise to degenerate solutions. Building on these insights, we revisit self-distillation as a practical remedy for certain forms of suboptimal convergence and reformulate it to avoid excessive gradient norms for stable optimization. We demonstrate that our strategy extends beyond image generation to diffusion-based policy learning, without reliance on pretrained diffusion models for initialization, illustrating its broader applicability.

Figures

Figures reproduced from arXiv: 2601.22679 by Duhoe Kim, Jaesik Park, Woosung Kim, Youngjoong Kim.

Figure 1
Figure 1. Figure 1: From consistency training to improved Self-Distillation. Consistency training learns a mapping over conditional velocity, often suffering from training instability and reproducibility issues. Relaxing the time condition mitigates this instability, and self-distillation provides a principled target by aligning with the marginal velocity. However, directly applying self-distillation leads to unstable trainin… view at source ↗
Figure 2
Figure 2. Figure 2: Toy experiments with a 5-layer MLP (batch size of 2048). LDT drives flow map training toward a suboptimal solution, while LED leads to a solution close to the ground-truth. LCT LED (a) B = 2048 (b) B = 512 (c) B = 128 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Consistency training (with LCT) is biased toward degen￾erate distributions when the batch size B decreases. convergence and only ensures the existence of such fixed points. As a result, it may fail to converge. 0 200 400 600 800 1000 Training Steps 0.0 0.2 0.4 0.6 ED DT ED CT [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: LED over training steps on a toy dataset. The experiment follows [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Loss landscapes of four methods. α and β denote the top-2 eigenvectors of the Hessian on ImageNet-1K with DiT-B/4. σ denotes the standard deviations of each landscape, and N the number of samples outside each method’s own 95% confidence bound. Values in parentheses report the number of samples exceeding the 95% bound defined by iSD, as a common reference (details in Section C.6) [PITH_FULL_IMAGE:figures/f… view at source ↗
Figure 6
Figure 6. Figure 6: Design choices. (a) FIDs over training steps. Solid lines indicate the JVP approximation, and dash-dot lines indicate the exact JVP. (b) FIDs of Post-CFG over guidance scales. (c) FIDs of Pre-CFG over training steps. Solid lines indicate trigonometric interpolation, and dash-dot lines indicate linear [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Gradient norms of each method. We estimate gradient norms under the same setting as [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Training curves of consistency training. (a) Training loss over training steps. (b) Two-step FIDs over training steps. (c) Rescaled Y-axis for the consistency model with a preconditioner. Without Preconditioner. We train a DiT-XL/2 architecture from randomly initialized weights without any preconditioner. In this case, training consistently failed, with the loss diverging and no meaningful samples being g… view at source ↗
Figure 11
Figure 11. Figure 11: Variants of Pre-CFG. Dash, dash-dotted, and solid lines indicate iSD-U, iSD-C, and iSD-T, respectively. Each color represents a different Pre-CFG scale. Trigonometric interpolation and the JVP approximation are used by default on DiT-B/4. ( on DiT-B/4a) FIDs of Pre-CFG variants over training steps. (b) FIDs of Post-CFG over guidance scales. (c) FIDs over multiplied guidance scales. For DiT (Peebles & Xie,… view at source ↗
Figure 12
Figure 12. Figure 12: 4-step samples from iSD on CelebA-HQ 256×256. CelebA-HQ. We conduct additional comparisons on CelebA￾HQ to validate our method in an unconditional setting. For fair comparison, all methods in [PITH_FULL_IMAGE:figures/full_fig_p031_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Results of policy learning. The left two figures show average success rates (solid lines) and action mean-squared errors (dashed-dot lines) over training steps. Numeric entries indicate the optimal values across training steps. The rightmost figure shows the simulation results of iSD with NFE=2. reflect the success rate, and iSD can achieve higher success rates despite yielding higher action errors than o… view at source ↗
Figure 14
Figure 14. Figure 14: Loss landscapes of four methods. α and β denote the top-2 eigenvectors of the Hessian on ImageNet-1K with DiT-B/4. σ denotes the standard deviations of the landscape fields, and N the number of samples outside each method’s own 95% confidence bound. Values in parentheses report the number of samples exceeding the 95% bound defined by iSD, as a common reference. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Class-level samples generated by iSD-T with two-step sampling on ImageNet 256×256. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Two-step samples from the iSD-T on ImageNet 256×256. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficient Image Synthesis with Sphere Latent Encoder

    cs.CV 2026-05 unverdicted novelty 6.0

    Decouples Sphere Encoder into fixed pretrained encoder and spherical latent denoiser, yielding higher quality and faster inference than the joint original on Animal-Faces, Oxford-Flowers and ImageNet-1K.

Reference graph

Works this paper leans on

3 extracted references · 1 linked inside Pith · cited by 1 Pith paper

  1. [2]

    Alternatively, since Ft,t =F t,s +O(d) , both terms can be represented in terms ofF t,t

    The firstv ∗ t term corresponds to Ft,t, while the second vt corresponds to Ft,s. Alternatively, since Ft,t =F t,s +O(d) , both terms can be represented in terms ofF t,t. In the case of Ft,t, the model learns F θ t,t(xt)≈v ∗ t (xt) due to the loss term of∥F θ t,t(xt)−v t(xt|x)∥2

  2. [3]

    Therefore, the Shortcut Model can be seen as Eulerian self-distillation under anO((t−s) 3)bound

    This can be interpreted as the model learning a flow map corresponding to the trajectory induced by an approximated marginal velocity. Therefore, the Shortcut Model can be seen as Eulerian self-distillation under anO((t−s) 3)bound. Consistency Trajectory Model.Consistency Trajectory Model (CTM) trains a network to satisfy the semigroup condition of flow m...

    arXiv 2024

  3. [6851]

    vt(xt|x)−F θ(xt;t, s) + (s−t)· d dt Fθ− (xt;t, s) 2 2 # =∇ θE

    Curran Associates, Inc., 2020. Issenhuth, T., Lee, S., Santos, L. D., Franceschi, J.-Y ., Kim, C., and Rakotomamonjy, A. Improving consistency mod- els with generator-augmented flows, 2025. Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progres- sive growing of GANs for improved quality, stability, and variation. InInternational Conference on Learning ...

    Pith/arXiv arXiv 2020