REVIEW 2 major objections 7 minor 14 references
Accelerate TarFlow Sampling with GS-Jacobi Iteration
T0 review · 2 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read TarFlow sampling can be solved as a parallel fixed-point system, speeding it up 2.5–5.3× without hurting image quality.
desk verdict Solid speedup for TarFlow sampling, but the 'no FID degradation' claim needs error bars before it is published as a blanket statement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the diagonalized nonlinear system $X^{(k+1)} = \Sigma(X^{(k)})Z + \mu(X^{(k)})$ derived from the TarFlow inverse block, together with its modular GS-Jacobi refinement. Two cheap metrics steer it: CRM = $\|\Sigma^{-1}(X)X\|_2\,\|W_s\|_2 + \|W_u\|_2$ ranks blocks by expected convergence difficulty, and IGM = $\|\Sigma(X^{(0)})Z + \mu(X^{(0)}) - X^*\|_2$ scores candidate initial guesses. Proposition 1 supplies the error-propagation structure: the iteration converges strictly within $T-1$ steps and error recurses through a strictly lower-triangular matrix $\Gamma^{(k)}$, so later positions see residual error from earlier ones until the diagonal fills in.
What would settle it
Measure the Jacobian norm $\|J_{\text{attn}}^{(l)}\|$ of the attention stack in each block of one of the four models and compare the per-block values; if they vary by more than a small factor, the CRM ranking should change once this term is included, and the reported strategies would no longer keep FID within 1% relative error on that model. A cheaper check: recompute the dominant-block selection from a different random batch of 128 training images and see whether the same 'tough' blocks are chosen.
Extended reading notes
Core claim
On the paper's own terms: the inverse pass of TarFlow, $x_t = \exp(s(x_{<t}))z_t + u(x_{<t})$, is a causal nonlinear system that is normally solved serially over the sequence length $T$. The authors rewrite it as the fixed-point problem $X = \Sigma(X)Z + \mu(X)$, so that the diagonal Newton / nonlinear Jacobi iteration $X^{(k+1)} = \Sigma(X^{(k)})Z + \mu(X^{(k)})$ updates all positions in parallel and provably converges in at most $T-1$ iterations. Naive parallel Jacobi fails in practice, which they trace to two block-dependent effects: a small number of blocks dominate the transformation and converge slowly, and some blocks are sensitive to the initial guess and overflow numerically. They therefore define the Convergence Ranking Metric (CRM), a cheap proxy based on weight norms and the non-volume-preserving term, to classify blocks as simple or tough, and the Initial Guessing Metric (IGM) to choose between $Z$ and zero-padded initial values. Tough blocks are segmented into modules and solved by Gauss-Seidel-Jacobi iteration with more per-module Jacobi steps, while simple blocks get few steps; across four TarFlow models this preserves FID (relative error <1%) while accelerating sampling by 2.51–5.32×.
Load-bearing premise
The approach stands on the premise that attention layers in different TarFlow blocks have similar-sized derivatives (Jacobian norms), so the Convergence Ranking Metric can ignore them; the paper calls this 'a very strong assumption' and notes it holds in its experiments but is hard to verify strictly.
Editorial extensions
If this is right
- TarFlow sampling no longer needs to be serial: with the reported strategies, Img128cond runs 4.53× faster, AFHQ 5.32×, Img64uncond 2.96×, and Img64cond 2.51×, all at FID relative error below 1%.
- The speed-up grows when fewer blocks have dominant CRM values and when the patched sequence length $T$ is longer, since those conditions give the parallel Jacobi phase more work to hide.
- Simple blocks can be solved with as few as 8–10 Jacobi iterations, while tough blocks need roughly 30 module-local iterations after segmentation, giving a practical recipe for setting iteration budgets.
- The acceleration is a preprocessing change plus a sampling-time loop change: the same trained weights are used, so no retraining or distillation is needed.
Reading between the lines
- The CRM/IGM recipe suggests a general design principle for autoregressive generative models: redundancy across blocks means sequential decoding can often be replaced by parallel fixed-point solving, with the per-block effort set by a cheap norm-based ranking; testing this on other causal-transformer flows or autoregressive decoders is a natural next step the paper does not take.
- The toughness distribution may reflect an information bottleneck: the one or two dominant blocks per model likely perform the bulk of the non-volume-preserving deformation, so splitting them into more modules could trade a little extra latency for a larger safe speed-up on harder images.
- A direct extension would be to compute IGM/CRM adaptively per batch or per image during sampling rather than once from a fixed training batch; if the dominant blocks shift with image content, static strategies could misallocate iterations.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript targets the sampling bottleneck of TarFlow, a transformer-based autoregressive normalizing flow, where the inverse pass through each block is a nonlinear triangular (causal) map that forces sequential computation. The authors reformulate the inverse pass as a fixed-point system and solve it with Jacobi iteration, then with a modular Gauss-Seidel-Jacobi variant that updates blocks of time steps in parallel and passes serially between modules. To configure the solver they propose two quantities computed in a preprocessing pass over 128 training images: the Initial Guessing Metric (IGM), which chooses between the feature-map initialization Z and a zero-padded initialization Z0 to avoid numerical overflow, and the Convergence Ranking Metric (CRM), which predicts which TarFlow blocks converge slowly and therefore need modular GS treatment with more iterations. Experiments on four TarFlow models (Img128cond, AFHQ, Img64uncond, Img64cond) report wall-clock speedups of 4.53x, 5.32x, 2.96x, and 2.51x, respectively, with FID relative errors below 1% for the selected configurations, plus a convergence and error-propagation analysis (Prop. 1, Appendix A), strategy tables (Tables 3a-3d), convergence curves (Figures 2 and 5), and visual comparisons (Figure 7).
Significance. The result, if statistically supported, is practically significant: TarFlow achieves state-of-the-art generative quality but its sampling is notoriously slow, and a factor 2.5-5.3 speedup at near-constant FID would make the model considerably more usable. The paper has several genuine strengths: it reports many failing configurations alongside the successful ones rather than hiding them in the tables; code and checkpoints are promised publicly; and the authors explicitly and honestly flag the 'very strong assumption' underlying CRM in Appendix B. The underlying fixed-point observation (Prop. 1) is essentially correct for the triangular map, and the empirical convergence curves corroborate the tough-block identification on the four tested models. The two weaknesses that need work before the central claim is acceptable are the absence of any uncertainty quantification for the FID comparisons and the under-specified, partly self-referential selection rule for the CRM strategy; neither undermines the plausibility of the speedups, but both affect whether the headline 'without degrading FID' claim is established.
major comments (2)
- [Section 4.3, Tables 3a-3d] The claim that GS-Jacobi sampling 'maintains FID within 1% relative error' rests on single-run FID evaluations with no reported seeds, no repeated runs, and no confidence intervals. The bolded entries differ from their baselines by only 0.03-0.13 FID (Img128cond 5.06 to 5.09; AFHQ 13.60 to 13.73; Img64uncond 14.67 to 14.80), which is the same magnitude as the typical noise of a 50k-sample FID estimate, so the 1% criterion is not statistically established. The post-hoc selection compounds this: Tables 3a-3d scan many configurations and bold only those that landed under 1%, while closely related configurations fail the same criterion (e.g., [6-4-32-10] at 3.20% in Table 3a). I recommend repeating the FID evaluations over several seeds (reporting means and standard deviations, or bootstrap intervals) and stating whether the final strategies were fixed independently of the FID outcomes.
- [Section 3.3 and Appendix B] The CRM is derived by dropping a term involving the attention Jacobians under an assumption the authors call 'very strong' (that these norms behave similarly across TarFlow blocks), and it is validated only on the same four models used to develop the method, so the general claim that CRM identifies simple vs. tough blocks for TarFlow models in general is not yet supported. On the tested models, Figure 2 independently corroborates the chosen tough blocks, so the empirical results themselves appear to stand; the issue is rather that the selection rule is not reproducible: 'dominant' is never defined quantitatively, and Table 2 bolds Block7 of Img64cond at 10.46% while not bolding Block3 of Img128cond at 10.93%. Please give an explicit, pre-specified rule for choosing the stacked blocks and test it on at least one held-out TarFlow configuration, or clearly scope the claim to the four reported models.
minor comments (7)
- [Section 3.3] The sentence after Figure 2 lists the tough blocks as 'Block6 of Img128cond, Block7 of AFHQ, Block0 of Img64cond, Block6 of Img64uncond,' but the stacking rule in Section 3.4 includes Block0 of Img64uncond, whose CRM (50.71%) is the largest in that model; please make the two lists consistent.
- [Appendix A] The statement that the diagonal-Newton/fixed-point iteration is 'superlinear convergence' is not justified by the preceding argument; the strictly triangular error recursion (7) gives finite-step convergence in T-1 iterations in exact arithmetic, which is a stronger property, and the standard superlinear-order claim needs a separate argument or should be removed.
- [Section 4.3] The relative-error column appears to truncate or floor negative values to 0.00% (e.g., rows with FID 5.04 vs. 5.06 and 4.38 vs. 4.42 are both listed as 0.00%); please state the exact formula, including the treatment of improvements, so that the 1% criterion is unambiguous.
- [Section 4.3 and Abstract] The timing units are inconsistent between the abstract (213 s per 100 images on a single A800) and Table 3a (133.19 in units of '100 s' for 50k images on 8 GPUs); please state hardware, batch size, and the exact unit for all timing measurements.
- [Table 4] In the Img64cond block, Block7 reads '0.12 .26 1.95' with a missing leading zero in the Wu entry; please correct the typo and add a caption defining the three component norms.
- [Section 3.4 and Algorithm 1] Proposition 1 analyzes the pure Jacobi iteration (4), whereas Algorithm 1 uses modular GS-Jacobi updates; please add an explicit statement of how the convergence and error-propagation analysis transfers to the modular scheme.
- [Throughout] Typos: 'Non-V olume' (Section 1), 'Guass-Seidel' (Section 3.4 title), 'vaild' (Appendix B), and 'simpe' (Section 4.2) should be corrected.
Circularity Check
No significant circularity: the speedup and FID claims rest on external benchmarks, and the proposed metrics are heuristics checked against independent convergence data.
full rationale
The derivations are self-contained rather than circular. The GS-Jacobi iteration is a standard numerical solver applied to the explicit inverse fixed-point equation X = Σ(X)Z + μ(X); Proposition 1's finite-time convergence follows from the triangular causality of the attention computation, not from any fitted value. IGM (Eq. 5) and CRM (Eq. 6) are defined directly from model weights and a forward pass on training patches; they are not defined in terms of the FID outcomes they are used to explain. The paper validates the tough-block identification against independently computed convergence curves (Figure 2), and it explicitly flags the limitation: 'Now we make a very strong assumption: the norm of Jacobi matrix of attention layers between TarFlow blocks behave similar, or in coordination with the previous item, which can be dropped out. It's hard to strictly verify this but it holds in the experiments.' That is a generalizability caveat about the heuristic, not a circular reduction. The central speedup claim is an external, wall-clock/FID evaluation against the original TarFlow sampler, and the selected strategies are reported transparently in Tables 3a–3d. The tuning of strategy hyperparameters and the absence of FID confidence intervals are statistical selection/uncertainty concerns, not evidence that any prediction is forced by construction. No load-bearing step relies on an author self-citation or on redefining a fitted parameter as a prediction.
Assumptions & free parameters
free parameters (3)
- Jacobi iteration counts J_l =
Varies per model, e.g., J=32 for the tough block and J=10 for others in Img128cond strategy [6-8-32-10]
- GS module count =
8 for most tough blocks, sometimes 16 or 32 (e.g., Img64cond uses GS=16 for Block0 and 8 for Block7)
- CRM dominance criterion =
None: the rule is 'select blocks with large CRM until no dominant block remains', with 'dominant' left undefined
assumptions (3)
- ad hoc to paper Attention-layer Jacobian norms are approximately equal across TarFlow blocks
- domain assumption IGM and CRM computed from 128 training images are representative of the test distribution
- standard math First-order Taylor expansion accurately tracks the error propagation for the number of iterations used
Cite this review
Pith. "Pith review of Accelerate TarFlow Sampling with GS-Jacobi Iteration." pith.science (2026). https://pith.science/paper/CYJIYR5U
@misc{pith2026250512849,
author = {Pith},
title = {Pith review of: Accelerate TarFlow Sampling with GS-Jacobi Iteration},
year = {2026},
howpublished = {\url{https://pith.science/paper/CYJIYR5U}},
note = {Machine review of arXiv:2505.12849}
}
read the original abstract
Image generation models have achieved widespread applications. As an instance, the TarFlow model combines the transformer architecture with Normalizing Flow models, achieving state-of-the-art results on multiple benchmarks. However, due to the causal form of attention requiring sequential computation, TarFlow's sampling process is extremely slow. In this paper, we demonstrate that through a series of optimization strategies, TarFlow sampling can be greatly accelerated by using the Gauss-Seidel-Jacobi (abbreviated as GS-Jacobi) iteration method. Specifically, we find that blocks in the TarFlow model have varying importance: a small number of blocks play a major role in image generation tasks, while other blocks contribute relatively little; some blocks are sensitive to initial values and prone to numerical overflow, while others are relatively robust. Based on these two characteristics, we propose the Convergence Ranking Metric (CRM) and the Initial Guessing Metric (IGM): CRM is used to identify whether a TarFlow block is "simple" (converges in few iterations) or "tough" (requires more iterations); IGM is used to evaluate whether the initial value of the iteration is good. Experiments on four TarFlow models demonstrate that GS-Jacobi sampling can significantly enhance sampling efficiency while maintaining the quality of generated images (measured by FID), achieving speed-ups of 4.53x in Img128cond, 5.32x in AFHQ, 2.96x in Img64uncond, and 2.51x in Img64cond without degrading FID scores or sample quality. Code and checkpoints are accessible on https://github.com/encoreus/GS-Jacobi_for_TarFlow
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Iterative sparse triangular solves for precondi- tioning
Hartwig Anzt, Edmond Chow, and Jack Dongarra. Iterative sparse triangular solves for precondi- tioning. In Euro-Par 2015: Parallel Processing: 21st International Conference on Parallel and Distributed Computing, Vienna, Austria, August 24-28, 2015, Proceedings 21 , pages 650–661. Springer,
work page 2015
-
[4]
Imagenet: A large-scale hier- archical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255,
work page 2009
-
[8]
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International conference on machine learning, pages 2078–2087. PMLR,
-
[13]
Conditional density estimation with bayesian normalising flows
Brian L Trippe and Richard E Turner. Conditional density estimation with bayesian normalising flows. arXiv preprint arXiv:1802.04908,
-
[14]
Normalizing flows are capable generative models
Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, and Josh Susskind. Normalizing flows are capable generative models. arXiv preprint arXiv:2412.06329,
-
[2003]
Accelerating transformer inference for translation via parallel decoding
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427,
-
[2009]
Laurent Dinh, David Krueger, and Yoshua Bengio
doi: 10.1109/CVPR.2009.5206848. Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516,
arXiv 2009
-
[2014]
Density estimation using real nvp
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803,
Show all 14 references
-
[2015]
Domain overlap for iterative sparse triangular solves on gpus
Hartwig Anzt, Edmond Chow, Daniel B Szyld, and Jack Dongarra. Domain overlap for iterative sparse triangular solves on gpus. In Software for Exascale Computing-SPPEXA 2013-2015, pages 527–545. Springer,
2013
-
[2016]
Parallel complexity of forward and backward propagation
Maxim Naumov. Parallel complexity of forward and backward propagation. arXiv preprint arXiv:1712.06577,
-
[2017]
Transformer neural autoregressive flows
Massimiliano Patacchiola, Aliaksandra Shysheya, Katja Hofmann, and Richard E Turner. Transformer neural autoregressive flows. arXiv preprint arXiv:2401.01855,
-
[2020]
Edmond Chow, Hartwig Anzt, Jennifer Scott, and Jack Dongarra
URL https://arxiv.org/abs/1912.01865. Edmond Chow, Hartwig Anzt, Jennifer Scott, and Jack Dongarra. Using jacobi iterations and blocking for solving sparse triangular systems in incomplete factorization preconditioning. Journal of Parallel and Distributed Computing, 119:219–230,
1912 arXiv
-
[2021]
Classifier-free diffusion guidance
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,
-
[2024]
Sdxl: Improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.