Pith. sign in

REVIEW 2 major objections 7 minor 14 references

Accelerate TarFlow Sampling with GS-Jacobi Iteration

T0 review · 2 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read TarFlow sampling can be solved as a parallel fixed-point system, speeding it up 2.5–5.3× without hurting image quality.

desk verdict Solid speedup for TarFlow sampling, but the 'no FID degradation' claim needs error bars before it is published as a blanket statement. read the letter →

arxiv 2505.12849 v1 pith:CYJIYR5U submitted 2025-05-19 cs.CV

classification cs.CV
keywords TarFlownormalizingflowsGauss-Seidel-Jacobiiterationparallelsamplingfixed-pointcausalattentionimagegenerationFID
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TarFlow is a state-of-the-art image generator whose sampling phase is slow because each autoregressive block must be decoded token by token, like an RNN. This paper claims that sampling can instead be written as a nonlinear fixed-point system and solved with a hybrid Gauss-Seidel-Jacobi iteration, in which many image patches are updated in parallel. The catch is that TarFlow blocks are not equally demanding: a few 'tough' blocks need many iterations and careful initial guesses, while most converge in a few. The paper introduces two metrics (CRM and IGM) to identify those blocks and pick good starting values, and reports 2.51–5.32× speed-ups on four TarFlow models with FID error below 1%.

What carries the argument

The load-bearing object is the diagonalized nonlinear system $X^{(k+1)} = \Sigma(X^{(k)})Z + \mu(X^{(k)})$ derived from the TarFlow inverse block, together with its modular GS-Jacobi refinement. Two cheap metrics steer it: CRM = $\|\Sigma^{-1}(X)X\|_2\,\|W_s\|_2 + \|W_u\|_2$ ranks blocks by expected convergence difficulty, and IGM = $\|\Sigma(X^{(0)})Z + \mu(X^{(0)}) - X^*\|_2$ scores candidate initial guesses. Proposition 1 supplies the error-propagation structure: the iteration converges strictly within $T-1$ steps and error recurses through a strictly lower-triangular matrix $\Gamma^{(k)}$, so later positions see residual error from earlier ones until the diagonal fills in.

What would settle it

Measure the Jacobian norm $\|J_{\text{attn}}^{(l)}\|$ of the attention stack in each block of one of the four models and compare the per-block values; if they vary by more than a small factor, the CRM ranking should change once this term is included, and the reported strategies would no longer keep FID within 1% relative error on that model. A cheaper check: recompute the dominant-block selection from a different random batch of 128 training images and see whether the same 'tough' blocks are chosen.

Watch

Extended reading notes

Core claim

On the paper's own terms: the inverse pass of TarFlow, $x_t = \exp(s(x_{<t}))z_t + u(x_{<t})$, is a causal nonlinear system that is normally solved serially over the sequence length $T$. The authors rewrite it as the fixed-point problem $X = \Sigma(X)Z + \mu(X)$, so that the diagonal Newton / nonlinear Jacobi iteration $X^{(k+1)} = \Sigma(X^{(k)})Z + \mu(X^{(k)})$ updates all positions in parallel and provably converges in at most $T-1$ iterations. Naive parallel Jacobi fails in practice, which they trace to two block-dependent effects: a small number of blocks dominate the transformation and converge slowly, and some blocks are sensitive to the initial guess and overflow numerically. They therefore define the Convergence Ranking Metric (CRM), a cheap proxy based on weight norms and the non-volume-preserving term, to classify blocks as simple or tough, and the Initial Guessing Metric (IGM) to choose between $Z$ and zero-padded initial values. Tough blocks are segmented into modules and solved by Gauss-Seidel-Jacobi iteration with more per-module Jacobi steps, while simple blocks get few steps; across four TarFlow models this preserves FID (relative error <1%) while accelerating sampling by 2.51–5.32×.

Load-bearing premise

The approach stands on the premise that attention layers in different TarFlow blocks have similar-sized derivatives (Jacobian norms), so the Convergence Ranking Metric can ignore them; the paper calls this 'a very strong assumption' and notes it holds in its experiments but is hard to verify strictly.

Editorial extensions

If this is right

  • TarFlow sampling no longer needs to be serial: with the reported strategies, Img128cond runs 4.53× faster, AFHQ 5.32×, Img64uncond 2.96×, and Img64cond 2.51×, all at FID relative error below 1%.
  • The speed-up grows when fewer blocks have dominant CRM values and when the patched sequence length $T$ is longer, since those conditions give the parallel Jacobi phase more work to hide.
  • Simple blocks can be solved with as few as 8–10 Jacobi iterations, while tough blocks need roughly 30 module-local iterations after segmentation, giving a practical recipe for setting iteration budgets.
  • The acceleration is a preprocessing change plus a sampling-time loop change: the same trained weights are used, so no retraining or distillation is needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The CRM/IGM recipe suggests a general design principle for autoregressive generative models: redundancy across blocks means sequential decoding can often be replaced by parallel fixed-point solving, with the per-block effort set by a cheap norm-based ranking; testing this on other causal-transformer flows or autoregressive decoders is a natural next step the paper does not take.
  • The toughness distribution may reflect an information bottleneck: the one or two dominant blocks per model likely perform the bulk of the non-volume-preserving deformation, so splitting them into more modules could trade a little extra latency for a larger safe speed-up on harder images.
  • A direct extension would be to compute IGM/CRM adaptively per batch or per image during sampling rather than once from a fixed training batch; if the dominant blocks shift with image content, static strategies could misallocate iterations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This manuscript targets the sampling bottleneck of TarFlow, a transformer-based autoregressive normalizing flow, where the inverse pass through each block is a nonlinear triangular (causal) map that forces sequential computation. The authors reformulate the inverse pass as a fixed-point system and solve it with Jacobi iteration, then with a modular Gauss-Seidel-Jacobi variant that updates blocks of time steps in parallel and passes serially between modules. To configure the solver they propose two quantities computed in a preprocessing pass over 128 training images: the Initial Guessing Metric (IGM), which chooses between the feature-map initialization Z and a zero-padded initialization Z0 to avoid numerical overflow, and the Convergence Ranking Metric (CRM), which predicts which TarFlow blocks converge slowly and therefore need modular GS treatment with more iterations. Experiments on four TarFlow models (Img128cond, AFHQ, Img64uncond, Img64cond) report wall-clock speedups of 4.53x, 5.32x, 2.96x, and 2.51x, respectively, with FID relative errors below 1% for the selected configurations, plus a convergence and error-propagation analysis (Prop. 1, Appendix A), strategy tables (Tables 3a-3d), convergence curves (Figures 2 and 5), and visual comparisons (Figure 7).

Significance. The result, if statistically supported, is practically significant: TarFlow achieves state-of-the-art generative quality but its sampling is notoriously slow, and a factor 2.5-5.3 speedup at near-constant FID would make the model considerably more usable. The paper has several genuine strengths: it reports many failing configurations alongside the successful ones rather than hiding them in the tables; code and checkpoints are promised publicly; and the authors explicitly and honestly flag the 'very strong assumption' underlying CRM in Appendix B. The underlying fixed-point observation (Prop. 1) is essentially correct for the triangular map, and the empirical convergence curves corroborate the tough-block identification on the four tested models. The two weaknesses that need work before the central claim is acceptable are the absence of any uncertainty quantification for the FID comparisons and the under-specified, partly self-referential selection rule for the CRM strategy; neither undermines the plausibility of the speedups, but both affect whether the headline 'without degrading FID' claim is established.

major comments (2)
  1. [Section 4.3, Tables 3a-3d] The claim that GS-Jacobi sampling 'maintains FID within 1% relative error' rests on single-run FID evaluations with no reported seeds, no repeated runs, and no confidence intervals. The bolded entries differ from their baselines by only 0.03-0.13 FID (Img128cond 5.06 to 5.09; AFHQ 13.60 to 13.73; Img64uncond 14.67 to 14.80), which is the same magnitude as the typical noise of a 50k-sample FID estimate, so the 1% criterion is not statistically established. The post-hoc selection compounds this: Tables 3a-3d scan many configurations and bold only those that landed under 1%, while closely related configurations fail the same criterion (e.g., [6-4-32-10] at 3.20% in Table 3a). I recommend repeating the FID evaluations over several seeds (reporting means and standard deviations, or bootstrap intervals) and stating whether the final strategies were fixed independently of the FID outcomes.
  2. [Section 3.3 and Appendix B] The CRM is derived by dropping a term involving the attention Jacobians under an assumption the authors call 'very strong' (that these norms behave similarly across TarFlow blocks), and it is validated only on the same four models used to develop the method, so the general claim that CRM identifies simple vs. tough blocks for TarFlow models in general is not yet supported. On the tested models, Figure 2 independently corroborates the chosen tough blocks, so the empirical results themselves appear to stand; the issue is rather that the selection rule is not reproducible: 'dominant' is never defined quantitatively, and Table 2 bolds Block7 of Img64cond at 10.46% while not bolding Block3 of Img128cond at 10.93%. Please give an explicit, pre-specified rule for choosing the stacked blocks and test it on at least one held-out TarFlow configuration, or clearly scope the claim to the four reported models.
minor comments (7)
  1. [Section 3.3] The sentence after Figure 2 lists the tough blocks as 'Block6 of Img128cond, Block7 of AFHQ, Block0 of Img64cond, Block6 of Img64uncond,' but the stacking rule in Section 3.4 includes Block0 of Img64uncond, whose CRM (50.71%) is the largest in that model; please make the two lists consistent.
  2. [Appendix A] The statement that the diagonal-Newton/fixed-point iteration is 'superlinear convergence' is not justified by the preceding argument; the strictly triangular error recursion (7) gives finite-step convergence in T-1 iterations in exact arithmetic, which is a stronger property, and the standard superlinear-order claim needs a separate argument or should be removed.
  3. [Section 4.3] The relative-error column appears to truncate or floor negative values to 0.00% (e.g., rows with FID 5.04 vs. 5.06 and 4.38 vs. 4.42 are both listed as 0.00%); please state the exact formula, including the treatment of improvements, so that the 1% criterion is unambiguous.
  4. [Section 4.3 and Abstract] The timing units are inconsistent between the abstract (213 s per 100 images on a single A800) and Table 3a (133.19 in units of '100 s' for 50k images on 8 GPUs); please state hardware, batch size, and the exact unit for all timing measurements.
  5. [Table 4] In the Img64cond block, Block7 reads '0.12 .26 1.95' with a missing leading zero in the Wu entry; please correct the typo and add a caption defining the three component norms.
  6. [Section 3.4 and Algorithm 1] Proposition 1 analyzes the pure Jacobi iteration (4), whereas Algorithm 1 uses modular GS-Jacobi updates; please add an explicit statement of how the convergence and error-propagation analysis transfers to the modular scheme.
  7. [Throughout] Typos: 'Non-V olume' (Section 1), 'Guass-Seidel' (Section 3.4 title), 'vaild' (Appendix B), and 'simpe' (Section 4.2) should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the speedup and FID claims rest on external benchmarks, and the proposed metrics are heuristics checked against independent convergence data.

full rationale

The derivations are self-contained rather than circular. The GS-Jacobi iteration is a standard numerical solver applied to the explicit inverse fixed-point equation X = Σ(X)Z + μ(X); Proposition 1's finite-time convergence follows from the triangular causality of the attention computation, not from any fitted value. IGM (Eq. 5) and CRM (Eq. 6) are defined directly from model weights and a forward pass on training patches; they are not defined in terms of the FID outcomes they are used to explain. The paper validates the tough-block identification against independently computed convergence curves (Figure 2), and it explicitly flags the limitation: 'Now we make a very strong assumption: the norm of Jacobi matrix of attention layers between TarFlow blocks behave similar, or in coordination with the previous item, which can be dropped out. It's hard to strictly verify this but it holds in the experiments.' That is a generalizability caveat about the heuristic, not a circular reduction. The central speedup claim is an external, wall-clock/FID evaluation against the original TarFlow sampler, and the selected strategies are reported transparently in Tables 3a–3d. The tuning of strategy hyperparameters and the absence of FID confidence intervals are statistical selection/uncertainty concerns, not evidence that any prediction is forced by construction. No load-bearing step relies on an author self-citation or on redefining a fitted parameter as a prediction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. The load-bearing assumptions are the uniform attention-Jacobian premise behind CRM and the representativeness of the 128-image calibration batch. The per-model iteration counts and segment sizes are free parameters tuned to make the reported FID numbers work.

free parameters (3)
  • Jacobi iteration counts J_l = Varies per model, e.g., J=32 for the tough block and J=10 for others in Img128cond strategy [6-8-32-10]
    Chosen by scanning strategies per model and selecting those with FID relative error below 1%; not predicted from the metrics.
  • GS module count = 8 for most tough blocks, sometimes 16 or 32 (e.g., Img64cond uses GS=16 for Block0 and 8 for Block7)
    Selected experimentally; finer segmentation trades speed against quality.
  • CRM dominance criterion = None: the rule is 'select blocks with large CRM until no dominant block remains', with 'dominant' left undefined
    The set of stacked blocks is chosen per model and validated by Figure 2, so the threshold is effectively hand-tuned.
assumptions (3)
  • ad hoc to paper Attention-layer Jacobian norms are approximately equal across TarFlow blocks
    Stated in Appendix B; the paper says it is a 'very strong assumption' that is hard to verify and only supported by experiments. CRM's ability to rank convergence depends on it.
  • domain assumption IGM and CRM computed from 128 training images are representative of the test distribution
    Sections 3.2 and 3.3 use a small batch of training images to decide initial guesses and convergence rankings for all future samples; if the training statistics are not representative, the precomputed choices could be suboptimal at test time.
  • standard math First-order Taylor expansion accurately tracks the error propagation for the number of iterations used
    Appendix A uses a linearized error recursion; the practical method runs few iterations and does not verify that the linearization remains accurate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerate TarFlow Sampling with GS-Jacobi Iteration." pith.science (2026). https://pith.science/paper/CYJIYR5U

@misc{pith2026250512849,
  author       = {Pith},
  title        = {Pith review of: Accelerate TarFlow Sampling with GS-Jacobi Iteration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYJIYR5U}},
  note         = {Machine review of arXiv:2505.12849}
}
read the original abstract

Image generation models have achieved widespread applications. As an instance, the TarFlow model combines the transformer architecture with Normalizing Flow models, achieving state-of-the-art results on multiple benchmarks. However, due to the causal form of attention requiring sequential computation, TarFlow's sampling process is extremely slow. In this paper, we demonstrate that through a series of optimization strategies, TarFlow sampling can be greatly accelerated by using the Gauss-Seidel-Jacobi (abbreviated as GS-Jacobi) iteration method. Specifically, we find that blocks in the TarFlow model have varying importance: a small number of blocks play a major role in image generation tasks, while other blocks contribute relatively little; some blocks are sensitive to initial values and prone to numerical overflow, while others are relatively robust. Based on these two characteristics, we propose the Convergence Ranking Metric (CRM) and the Initial Guessing Metric (IGM): CRM is used to identify whether a TarFlow block is "simple" (converges in few iterations) or "tough" (requires more iterations); IGM is used to evaluate whether the initial value of the iteration is good. Experiments on four TarFlow models demonstrate that GS-Jacobi sampling can significantly enhance sampling efficiency while maintaining the quality of generated images (measured by FID), achieving speed-ups of 4.53x in Img128cond, 5.32x in AFHQ, 2.96x in Img64uncond, and 2.51x in Img64cond without degrading FID scores or sample quality. Code and checkpoints are accessible on https://github.com/encoreus/GS-Jacobi_for_TarFlow

Figures

Figures reproduced from arXiv: 2505.12849 by the authors.

Figure 1
Figure 1. Simple intuition diagram of GS-Jacobi sampling. First pass forward a small batch of images [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The distance between X(k) (up to 150 times) and target X∗ of all 8 blocks in four models. Most blocks converge within iteration times << T, with each model exhibiting only one or two slowly descending curves. with σ1 = 1, u1 = 0, then the transform from X to Z can be seen as an non-linear system: Forward: Z = Σ−1 (X)(X − µ(X)), Inverse: X = Σ(X)Z + µ(X). (3) For the inverse process, we can view the target X∗ as the … view at source ↗
Figure 3
Figure 3. The trace of the sampling in four models. From top to bottom: Img128cond, Img64cond, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The influence of different initial value and iteration times of an Img64cond sample. From [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The distance between GS-Jacobi iteration and target [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Intuition diagram of Gauss-Seidel-Jacobi sampling in single block. The horizontal long [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Visual Comparison of Different Methods 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 5 canonical work pages

  1. [1]

    Iterative sparse triangular solves for precondi- tioning

    Hartwig Anzt, Edmond Chow, and Jack Dongarra. Iterative sparse triangular solves for precondi- tioning. In Euro-Par 2015: Parallel Processing: 21st International Conference on Parallel and Distributed Computing, Vienna, Austria, August 24-28, 2015, Proceedings 21 , pages 650–661. Springer,

  2. [4]

    Imagenet: A large-scale hier- archical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255,

  3. [8]

    Neural autoregressive flows

    Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International conference on machine learning, pages 2078–2087. PMLR,

  4. [13]

    Conditional density estimation with bayesian normalising flows

    Brian L Trippe and Richard E Turner. Conditional density estimation with bayesian normalising flows. arXiv preprint arXiv:1802.04908,

  5. [14]

    Normalizing flows are capable generative models

    Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, and Josh Susskind. Normalizing flows are capable generative models. arXiv preprint arXiv:2412.06329,

  6. [2003]

    Accelerating transformer inference for translation via parallel decoding

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427,

  7. [2009]

    Laurent Dinh, David Krueger, and Yoshua Bengio

    doi: 10.1109/CVPR.2009.5206848. Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516,

  8. [2014]

    Density estimation using real nvp

    Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803,

Show all 14 references
  1. [2015]

    Domain overlap for iterative sparse triangular solves on gpus

    Hartwig Anzt, Edmond Chow, Daniel B Szyld, and Jack Dongarra. Domain overlap for iterative sparse triangular solves on gpus. In Software for Exascale Computing-SPPEXA 2013-2015, pages 527–545. Springer,

  2. [2016]

    Parallel complexity of forward and backward propagation

    Maxim Naumov. Parallel complexity of forward and backward propagation. arXiv preprint arXiv:1712.06577,

  3. [2017]

    Transformer neural autoregressive flows

    Massimiliano Patacchiola, Aliaksandra Shysheya, Katja Hofmann, and Richard E Turner. Transformer neural autoregressive flows. arXiv preprint arXiv:2401.01855,

  4. [2020]

    Edmond Chow, Hartwig Anzt, Jennifer Scott, and Jack Dongarra

    URL https://arxiv.org/abs/1912.01865. Edmond Chow, Hartwig Anzt, Jennifer Scott, and Jack Dongarra. Using jacobi iterations and blocking for solving sparse triangular systems in incomplete factorization preconditioning. Journal of Parallel and Distributed Computing, 119:219–230,

  5. [2021]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

  6. [2024]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.