{"id":"9e54f59e-fff6-41f8-beb8-641776f799f2","arxiv_id":"2504.21314","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Auto-regressive diffusion models provably control conditional-distribution sampling error with only a factor-K increase in inference cost, unlike vanilla diffusion where conditional error can blow up despite small joint error.","lead":"This paper proves the first sampling-error bounds for auto-regressive diffusion models, which generate data in patches, and shows they can track conditional distributions better than standard diffusion models. The result is relevant because standard diffusion models are known to miss high-level feature dependencies, and the analysis explains why patch-by-patch generation can fix this at a modest extra inference cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption A2 excludes the Gaussian case used to motivate the vanilla-diffusion failure, so Theorem 4.3 does not cover the paper's central comparison.","rationale":"The reader's weakest-assumption analysis identifies Assumption A2's pointwise score bound as the main restriction, and my reading agrees. This is the single most load-bearing concern because the paper's motivating negative result for vanilla diffusion is a Gaussian construction (Lemma 4.2), and the positive theorem for AR diffusion explicitly assumes away the Gaussian case. The point is not merely that a technical assumption is strong; the proof of Lemma 3.3 relies on the pointwise gradient bound to control the variance of ∂f in the marginal Hessian. Without it, the conditional score Lipschitz bound, which is used throughout the discretization and initialization estimates, is not established by the given argument. Prior work in the same line, such as Chen et al. (2023) and Benton et al. (2024), obtains polynomial convergence with only Hessian and second-moment assumptions, so the paper's 'mildest assumption' framing is not accurate. The theorem remains conditionally valuable, and the central idea is plausible, but the comparison to vanilla diffusion is not made on equal footing: the vanilla counterexample lies outside the assumptions under which the AR guarantee is proven. This supports the reader's CONDITIONAL verdict rather than a full acceptance. I do not see an additional concern strong enough to change the verdict; the A2 issue is already the decisive one, and the proposed concrete test would settle whether it is a genuine proof obstruction or merely a cosmetic assumption that can be relaxed.","tokens_in":35860,"tokens_out":22972,"duration_ms":245538,"concrete_test":"Set p* = N(0, I_d), L=1, m0=d. Check that A2 is violated since ∥∇ ln p*(x)∥ = ∥x∥ is unbounded. Then attempt to re-derive Lemma 3.3 and Theorem 4.3 replacing the pointwise gradient bound with the L2 bound E∥∇ ln p*∥² ≤ Ld, which follows from the Hessian bound via Lemma D.2. If the resulting conditional score Lipschitz constant becomes dimension-dependent (e.g., O(dL)) or if a distribution with bounded Hessian but unbounded conditional-score Hessian can be exhibited, the theorem's complexity and 'mildest assumption' claim need revision; if the proof can be patched, A2 should be reformulated as an L2 condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in the abstract and Section 4.1 rests on Theorem 4.3 (via Lemma 4.1) for AR diffusion and Lemma 4.2 as a negative result for vanilla diffusion. Lemma 4.2 is stated for Gaussian p*, but Assumption A2 in Section 2 requires the pointwise bound ∥∇ ln p*∥ ≤ √L. For any Gaussian, ∇ ln p* = -Σ^{-1}(x-μ) is unbounded, so A2 fails. The proof of Lemma 3.3 (Appendix A.2, Eq. 19) uses this pointwise gradient bound to show the marginal Hessian of p*_{[1:k]} is at most 2L; without it, the variance term Var(∂_x f*) is only controlled in expectation by Lemma D.2, giving a dimension-dependent constant or no uniform conditional score Lipschitz bound. Hence the main theorem's proof does not go through for the Gaussian example that motivates the paper, and the claim that the assumptions are 'possibly the mildest' (Abstract) is inaccurate: prior diffusion analyses (Chen et al. 2023; Benton et al. 2024) require only second-moment and Hessian control, not a global pointwise score bound. This is load-bearing because the central claim that AR diffusion has a provably smaller conditional-distribution gap than vanilla is established only under a restriction that excludes the simplest smooth distribution and the paper's own counterexample.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes autoregressive (AR) diffusion models, where the data vector is split into K patches and each patch is generated by a separate reverse OU process conditioned on previously generated patches. The authors introduce a global score-matching objective, prove its equivalence to the practical denoising objective, and derive an upper bound on the KL divergence between the generated and true joint distributions (Theorem 4.3), with inference complexity of order \\tilde{O}(K L^2 d \\epsilon^{-2}). They also prove a negative result (Lemma 4.2) showing that closeness of joint distributions does not imply closeness of conditional distributions, which they use to argue that vanilla diffusion can fail to capture conditional dependence. Experiments on two synthetic image tasks and on concatenated MNIST digits support the claim that AR diffusion better captures dependencies when the patch order aligns with the underlying feature relationships.","tokens_in":36114,"tokens_out":15503,"duration_ms":166679,"significance":"If the central claims hold, this is a useful first theoretical treatment of AR diffusion sampling error and provides a principled explanation for a commonly observed empirical phenomenon. The proof is a substantial adaptation of existing diffusion-analysis machinery to a stage-wise OU process, and the explicit gradient-complexity bound, the training-objective equivalence, and the careful experiments with ablation on patch order are strengths. The significance is currently tempered by two gaps: the main theorem relies on a pointwise score-norm assumption that excludes Gaussian targets, so it does not cover the motivating counterexample; and the comparison with vanilla diffusion mixes a pointwise lower bound with an averaged upper bound. These issues are correctable in revision, so the paper has clear potential but needs further work before the advertised conclusions are fully supported.","major_comments":[{"comment":"Assumption A2 requires the global pointwise bound \\|\\nabla \\ln p^*\\| \\le \\sqrt{L}. This excludes every Gaussian target, since \\|\\Sigma^{-1}(x-\\mu)\\| is unbounded as \\|x\\| grows. The proof of Lemma 3.3 (Appendix A.2, Eq. (19)) uses this pointwise bound to control the variance term Var(\\partial_x f^*) and to conclude that the marginal Hessian is bounded by 2L; without the pointwise gradient bound, only the expectation bound E\\|\\nabla f^*\\|^2 \\le Ld from Lemma D.2 is available, which leads to a dimension-dependent or non-uniform conditional-score Lipschitz constant. Because Theorem 4.3 relies on Lemma 3.3 through Lemma B.2, the main convergence theorem does not cover the Gaussian example used in Lemma 4.2 to motivate the failure of vanilla diffusion. Moreover, the abstract's phrase \"under (possibly) the mildest data assumption\" is not supported: the cited analyses of Chen et al. (2023) and Benton et al. (2024) require only second-moment and Hessian control, not a global pointwise gradient bound. Please either weaken A2, for example to a second-moment control of the score, and re-derive the affected constants, or explicitly restrict the scope of Theorem 4.3 and remove the \"mildest\" wording.","section":"Section 2 (Assumption A2) and Theorem 4.3"},{"comment":"Lemma 4.2 establishes a pointwise conditional-KL blow-up: for the constructed pair of Gaussians, KL(p^*(y|x)\\|\\hat{p}^*(y|x)) > M^2\\|x\\|^2 for a given x, while the joint KL is at most \\epsilon. However, in this construction the marginal law of x has variance of order \\epsilon^2/(d^2 M^2), so the averaged conditional KL, E_{x\\sim p^*}[KL(p^*(y|x)\\|\\hat{p}^*(y|x))], is of order \\epsilon^2/d and is small rather than divergent. By contrast, Theorem 4.3 and the proof of Lemma 4.1 bound the averaged conditional KL after taking x_{[1:k]} \\sim p^*. The central comparison therefore mixes two different notions: an averaged upper bound for AR diffusion and a pointwise lower bound for vanilla diffusion. Furthermore, Lemma 4.2 is a purely distributional construction; it does not show that a vanilla score-based diffusion model trained to small joint score error will realize this bad conditional distribution. Please restate the comparison in a single metric, or provide an averaged lower bound, or clearly state that the lower bound is a generic failure of joint closeness and not a property specific to vanilla diffusion.","section":"Section 4.1 (Lemma 4.2) and Theorem 4.3"},{"comment":"The abstract and the introduction claim that AR diffusion 'produces samples with a reduced gap in approximating the data conditional distribution' compared with typical diffusion models. As written, this is a comparative claim about the two algorithms under comparable assumptions. But the upper bound for AR diffusion is established under A1-A3, which include the pointwise score bound in A2, while the lower bound for vanilla diffusion is a counterexample stated in terms of arbitrary Gaussian joint distributions and does not involve the score-estimation error or the reverse diffusion dynamics. Thus the two results are not established within a common setting. To make the comparative claim precise, the authors should either prove a lower bound for vanilla diffusion under the same kind of score-error assumption used for AR diffusion, or substantially weaken the comparative statement, for example to 'AR diffusion admits a conditional-Divergence bound under the stated assumptions, while joint-only convergence does not imply pointwise conditional convergence in general.'","section":"Section 4.1 and Abstract (scope of the 'reduced gap' claim)"}],"minor_comments":[{"comment":"The paper explicitly acknowledges in Section 2 that an additional gradient norm upper bound is required compared with previous work, which is in tension with the abstract's 'mildest' phrasing; please harmonize these statements in the final version.","section":"Section 2 (Assumption A2)"},{"comment":"The displayed inequality in Lemma 3.3 appears to be missing the minus sign between the two score terms; as printed it resembles a product rather than a difference, and should be corrected to \\|\\nabla \\ln p_{*,k|[1:k-1]}(x|y) - \\nabla \\ln p_{*,k|[1:k-1]}(x'|y)\\|.","section":"Lemma 3.3"},{"comment":"The parameter M is declared as any real number, but the bound involves M^2; please state M>0 and clarify the notation x(1:d_{k+1}), including the dimension of x and the sense in which the first d_{k+1} coordinates are used.","section":"Lemma 4.2"},{"comment":"The constant c defined near the end of Theorem B.1 appears garbled: 'c = ln sqrt(1/(4L^2)+4 + 1/(2L))^2 < 1' mixes logarithms, square roots, and squares ambiguously; the proof would benefit from a clean definition of c and of the relation between \\delta and c.","section":"Appendix B (Theorem B.1)"},{"comment":"The term 'gradient complexity' in Remark 3 actually refers to the number of reverse-time score evaluations at inference, not to training gradients; please rename it to something like 'score-evaluation complexity' or clarify the terminology. Also, the experimental estimate of the constant C in Eq. (65) is heuristic; please describe its limitations more explicitly.","section":"Remark 3 and experiments"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a solid core and the central technical machinery is plausible, but the Assumption A2 issue is load-bearing for the paper's advertised scope and for the comparison with vanilla diffusion. I would not reject the paper on this basis, because the analysis under the stated assumptions may still be correct and the gaps can likely be repaired by weakening A2 or by substantially rewording the claims. The pointwise-versus-averaged mismatch in the comparison should be addressed head-on rather than by cosmetic editing. If the authors can either prove the theorem under a second-moment score bound or carefully restrict all claims to distributions satisfying A2, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is the first sampling-error theory for autoregressive diffusion, and that part is new and real. The conditional-KL bound, the decomposition into per-patch control, and the gradient-complexity comparison are all derived carefully. The training-loss equivalence lemma is also a genuinely useful bridge between the global objective and score estimation. If I worked on diffusion theory, I would want to know about this paper.\n\nThe central comparison against vanilla diffusion, however, has a crack. Assumption A2 requires a global pointwise bound on the score, which fails for any Gaussian. Lemma 4.2, the negative result showing vanilla diffusion can have huge conditional KL despite tiny joint KL, is itself built on a Gaussian. So the example that motivates the entire comparison lies outside the assumptions under which Theorem 4.3 is proved. The abstract's phrase \"possibly the mildest data assumption\" is not earned: prior diffusion analyses require second-moment and Hessian control, not a pointwise score bound. The paper does admit the extra gradient bound in Section 2, but it does not then reconcile that admission with the abstract's claim.\n\nThe vanilla-failure argument is also worst-case pointwise. It constructs a particular Gaussian pair where joint KL is small but conditional KL blows up. That shows vanilla can fail, but it does not show vanilla typically fails, nor does it show AR provably does better under comparable noise. The comparison is real but more modest than the abstract suggests.\n\nOn the empirical side, the main-text experiments confound architecture: AR uses an MLP while DDPM uses a U-Net. The appendix repeats with matched backbones and the qualitative finding survives, so the empirical core is probably sound, but the main-figure choice is unfortunate and should be corrected.\n\nNone of this makes the theorem wrong. The proof follows standard diffusion-analysis techniques, and I do not see an internal contradiction. The soft spots are in the framing and the scope of the assumptions, not in the central argument's logic. This deserves peer review. A serious referee should push the authors to either relax A2 (perhaps to an expectation bound on the score norm) or explicitly state that the theory covers only bounded-score distributions and that the Gaussian counterexample lies outside the theorem's scope. With that fixed, this becomes a solid contribution to generative modeling theory.","headline":"First AR-diffusion sampling theory, but the headline comparison rests on an assumption that excludes the paper's own Gaussian counterexample.","tokens_in":36650,"tokens_out":2551,"would_cite":true,"duration_ms":29094,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Auto-regressive diffusion models provably capture the conditional dependencies that vanilla diffusion models miss.","keywords":["auto-regressive diffusion","conditional dependence","score-based generative models","KL divergence","Ornstein-Uhlenbeck process","sampling error bounds","generative modeling theory","conditional distributions"],"falsifier":"Fit an AR diffusion and a vanilla diffusion to a two-patch Gaussian with a known linear conditional $p_*(x_2|x_1)$, then measure the empirical conditional KL between the generated and true conditionals: if the AR conditional KL grows with $|x_1|$ or fails to shrink as T increases and the score error goes to zero, the claimed reduced-gap phenomenon fails outside Assumption A2. A simpler check is to verify whether the proof can be rerun with Assumption A2 replaced by only the second-moment bound, since Lemma 4.2 shows the Gaussian motivating example violates A2.","tokens_in":35648,"feed_emoji":"🔗","tokens_out":6903,"duration_ms":74749,"temperature":0.7,"pith_summary":"This paper argues that vanilla diffusion models underperform on data with high-level, cross-patch relationships because they do not adequately represent conditional dependence, and that auto-regressive diffusion fixes this by generating the data patch by patch, each new patch conditioned on the already generated ones. Its main theorem gives the first convergence bound for auto-regressive diffusion: the KL divergence between the true and generated joint distributions is controlled by the per-patch score error, the discretization step size, and the OU mixing time, with gradient complexity only a factor of K larger than vanilla DDPM. The paper also proves a contrasting negative result: a vanilla diffusion model can match the joint distribution arbitrarily well while its conditional distributions are arbitrarily wrong, so the AR advantage is structural rather than a matter of tuning. If correct, auto-regressive diffusion is the practical choice whenever data has sequential or hierarchical dependencies, at a modest inference-time cost.","feed_headline":"AR diffusion provably captures conditional dependence","feed_subtitle":"New proof shows the autoregressive variant learns cross-patch rules; vanilla DDPM can fail even with a perfect joint fit.","key_machinery":"The load-bearing object is the stage-wise forward Ornstein-Uhlenbeck process: the data vector is split into K patches, and the forward process is run K times, each stage adding noise to one patch while freezing the already-noised earlier patches; Algorithm 1 reverses this stage-wise process patch by patch. The argument is carried by the chain-rule decomposition of KL divergence, $\\mathrm{KL}(p_*\\|\\hat p_*)\\le \\mathrm{KL}(p_{*,1}\\|\\hat p_{*,1})+\\sum_{k=1}^{K-1}\\mathbb{E}_{x_{[1:k]}\\sim p_{*,[1:k]}}\\mathrm{KL}(p_{*,k+1|[1:k]}(\\cdot|x_{[1:k]})\\|\\hat p_{*,k+1|[1:k]}(\\cdot|x_{[1:k]}))$, which turns joint generation quality into per-patch conditional convergence. Each conditional transition is analyzed with a reverse OU step whose discretization error is controlled by a time schedule that is linear early in the process and exponentially decaying near T, removing the need for score smoothness along the entire SDE trajectory.","core_discovery":"The paper establishes that auto-regressive diffusion---generating each data patch conditioned on previously generated patches---provably approximates the data's conditional distributions, and that this is a structural advantage over vanilla diffusion. Its main theorem shows that under Assumptions A1-A3, the generated joint distribution satisfies $\\mathrm{KL}(p_*\\|\\hat p_*)\\lesssim 2e^{-2T}L(m_0+d)+(L^2R\\eta^2+T\\eta)d+\\eta m_0+\\eta K R\\epsilon_{\\mathrm{score}}^2$, with gradient complexity $\\tilde{O}(KL^2 d\\epsilon^{-2})$. The proof routes through a stage-wise Ornstein-Uhlenbeck forward process and an adaptive reverse-time discretization, and it decomposes the joint error via the chain rule of KL divergence into a sum of expected conditional KL errors. The contrast is Lemma 4.2: for any tolerance, there exist Gaussian pairs whose joint KL divergence is as small as desired while the conditional KL divergence is arbitrarily large, so a vanilla diffusion model can match the joint distribution and still fail to capture dependencies. The paper also connects score estimation error to training loss: a K-patch AR model needs score error $\\tilde{O}(K^{-1/2}\\epsilon)$ and its inference complexity grows only by a factor K.","pith_inferences":["If the theorem extends beyond Assumption A2, one testable prediction is that AR diffusion's advantage over vanilla diffusion grows with the strength of inter-patch dependence; a systematic sweep over correlation strength in synthetic Gaussian mixtures would quantify this.","The ablation study implies patch order is a design choice rather than a fixed hyperparameter, so a natural extension is to learn a patch ordering or condition graph that maximizes the expected reduction in conditional KL divergence.","The stage-wise argument suggests AR diffusion could serve as an evaluation protocol for world-model video generators: temporal patches must satisfy physical conditional laws, which is exactly the setting where the paper predicts a gap.","The factor-K inference cost and the $\\sqrt{K}$ score-error allowance suggest a model-scale tradeoff: larger K improves conditional capture but demands more reverse steps, so a hierarchical coarse-to-fine patch schedule might preserve both benefits."],"forward_implications":["AR diffusion attains the bound $\\mathrm{KL}(p_*\\|\\hat p_*)\\lesssim 2e^{-2T}L(m_0+d)+(L^2R\\eta^2+T\\eta)d+\\eta m_0+\\eta K R\\epsilon_{\\mathrm{score}}^2$, with gradient complexity $\\tilde{O}(KL^2d\\epsilon^{-2})$---only a factor K worse than vanilla DDPM.","For any fixed conditioning tokens $x_{[1:k]}$, the generated next-patch distribution converges to $p_{*,k+1|[1:k]}(\\cdot|x_{[1:k]})$ in KL divergence as T grows and the step size shrinks.","Vanilla diffusion has no such guarantee: Lemma 4.2 constructs Gaussian pairs with arbitrarily small joint KL but arbitrarily large conditional KL, so joint accuracy does not imply correct dependencies.","The global denoising objective (12) has the same gradient as the conditional score-matching objective (Lemma 3.1), so minimizing the practical training loss directly controls the score estimates required by Assumption A3.","In experiments with synthetic sun-shadow data and MNIST composites, AR diffusion satisfies inter-feature rules more often than DDPM when the patch order aligns with the dependence structure, and its training loss gap predicts inference-time performance."],"supporting_citations":[{"why":"Defines the AR diffusion training objective and inference scheme that the paper formalizes and analyzes.","marker":"[16]"},{"why":"Provides the nearly d-linear convergence analysis for vanilla diffusion whose discretization arguments the paper adapts.","marker":"[2]"},{"why":"Supplies the minimal smoothness (Hessian-only) score estimation analysis and reverse-process lemmas used for the conditional bounds.","marker":"[3]"},{"why":"Motivates avoiding score-smoothness-along-trajectory assumptions that would complicate the AR analysis.","marker":"[4]"},{"why":"Supplies the synthetic world-model tasks and geometric feature extraction used in the experiments.","marker":"[10]"},{"why":"Supplies the KL chain-rule and reverse-transition inequalities used to decompose joint error into conditional errors.","marker":"[12]"},{"why":"Supplies the log-Sobolev / OU convergence bound used for the initialization error in Lemma B.3.","marker":"[23]"}],"fun_headline_variants":["AR diffusion provably beats vanilla on conditional dependence","Proof: AR diffusion captures dependencies vanilla misses","Auto-regressive diffusion: proven edge in dependency capture","AR diffusion's theorem: conditional dependence captured","AR diffusion provably fits conditional distributions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theorem's proof requires the data log-density to have a pointwise bounded gradient for all inputs (Assumption A2), which no Gaussian or other unbounded-support distribution satisfies.","fun_headline_variants_meta":{"raw":{"variants":["AR diffusion provably beats vanilla on conditional dependence","Proof: AR diffusion captures dependencies vanilla misses","Auto-regressive diffusion: proven edge in dependency capture","AR diffusion's theorem: conditional dependence captured","AR diffusion provably fits conditional distributions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000496,"raw_usage":{"total_tokens":2490,"prompt_tokens":1063,"completion_tokens":1427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":1359}},"tokens_in":679,"tokens_out":1427,"duration_ms":12811,"temperature":1.0,"reasoning_tokens":1359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:08:00.295027+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit an AR diffusion and a vanilla diffusion to a two-patch Gaussian with a known linear conditional $p_*(x_2|x_1)$, then measure the empirical conditional KL between the generated and true conditionals: if the AR conditional KL grows with $|x_1|$ or fails to shrink as T increases and the score error goes to zero, the claimed reduced-gap phenomenon fails outside Assumption A2. A simpler check is to verify whether the proof can be rerun with Assumption A2 replaced by only the second-moment bound, since Lemma 4.2 shows the Gaussian motivating example violates A2.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the nearly d-linear convergence analysis for vanilla diffusion whose discretization arguments the paper adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the minimal smoothness (Hessian-only) score estimation analysis and reverse-process lemmas used for the conditional bounds."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates avoiding score-smoothness-along-trajectory assumptions that would complicate the AR analysis."},{"cited_title":"and Wibisono, A","cited_arxiv_id":null,"evidence_quote":"Supplies the log-Sobolev / OU convergence bound used for the initialization error in Lemma B.3."}],"review_version":1}