{"id":"27a8ff9e-cb11-4dc8-8400-b3224b835da5","arxiv_id":"2507.11441","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Fine-tuned Vision AutoRegressive models mostly beat a strong diffusion baseline on downstream image generation, but DP fine-tuning yields poor FID scores.","lead":"The authors benchmark ways to fine-tune Vision AutoRegressive image generation models, comparing full fine-tuning, LoRA, and LayerNorm tuning against a leading diffusion adaptation method. They find VAR adapts well without privacy protection, but differentially private fine-tuning of these models produces poor images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 contradicts the central claim: DiffFit (FID 5.48) beats both VAR-d16 FFT (5.74) and VAR-d20 FFT (5.58) on CUB-200-2011, so 'FFT outperforms DiffFit across all models and datasets' is false as stated.","rationale":"The reader correctly identified the DiffFit comparison as the weakest pillar of the central claim, focusing on the unverified protocol match of quoted FID numbers. My stress-test found a more direct problem: even before any protocol critique, Table 1 contradicts the paper's own wording. On CUB-200-2011, DiffFit's quoted FID of 5.48 is better than VAR-d16 FFT (5.74) and VAR-d20 FFT (5.58), so the statement that FFT outperforms DiffFit 'across all models and datasets' cannot be true. This is an internal inconsistency, not a disagreement with external consensus, and it is load-bearing because the abstract, introduction, and conclusion all rest on the claim that VAR outperforms the SOTA DM adaptation strategy. I am not alleging misrepresentation; the likely explanation is loose summarization of a table that is mostly favorable to VAR. The implementation contribution remains valuable: the paper provides released code, detailed patching strategies for Opacus compatibility, and a benchmark of several adaptation families, and the DP negative result is honestly scoped to Oxford Flowers. The path to a sound version is straightforward: correct the overstatement by reporting per-dataset results, and either re-run DiffFit under the same FID protocol or explicitly state that quoted values are not directly comparable. Because these fixes are achievable and the underlying engineering contribution is real, I retain the reader's CONDITIONAL verdict rather than moving to rejection.","tokens_in":14496,"tokens_out":5181,"duration_ms":61221,"concrete_test":"Re-run the DiT-XL-2 DiffFit baseline on CUB-200-2011 under the exact Section 3.1 protocol (generate as many images per class as in the test set, using the same FID implementation), and independently re-verify the reported VAR-d16 and VAR-d20 FFT FIDs on the same split. If DiffFit's FID remains below 5.74 and 5.58, the 'across all datasets' claim is false and must be retracted. If DiffFit's FID rises above those values under the unified protocol, the contradiction is partly a protocol artifact, but the paper must still reword the claim to match its measured comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is an internal contradiction in the paper's own headline comparison. Table 1 reports DiT-XL-2 DiffFit at FID 5.48 on CUB-200-2011, while VAR-d16 FFT is 5.74 and VAR-d20 FFT is 5.58; lower FID is better, so both VAR FFT variants are worse than DiffFit, and every reported VAR adaptation on that dataset is worse. Yet Section 3.2 states that 'FFT outperforms [DiffFit] across all models and datasets', and the conclusion repeats that fine-tuned VAR 'performs better than the SOTA adaptation strategy for DMs'. The claim is therefore not merely protocol-sensitive; on the paper's own numbers it is false for CUB-200-2011. A related but secondary issue is that the DiT-XL-2 entries are quoted from prior work rather than measured under the Section 3.1 FID protocol (generating as many samples per class as in the test set, with a specific FID implementation), so the favorable comparisons on Food-101 and Oxford Flowers could shift if the baselines were re-run consistently. Both issues bear on the central assertion, but the CUB contradiction alone means the comparative conclusion needs revision.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper implements and benchmarks several adaptation methods for the Vision AutoRegressive (VAR) image generation model: full fine-tuning (FFT), LoRA, and LayerNorm tuning, on five downstream datasets, and compares the resulting FID scores with the diffusion-model adaptation method DiffFit. It also implements differentially private (DP) fine-tuning of VAR using DP-SGD with augmentation multiplicity, evaluated on Oxford Flowers. The central claims are that fine-tuned VAR outperforms the state-of-the-art DM adaptation strategy DiffFit, that VAR converges quickly, and that DP adaptations of VAR currently suffer from low generation quality and slow convergence.","tokens_in":14860,"tokens_out":5494,"duration_ms":63757,"significance":"If the comparative claims held, this would be a useful empirical data point for the IAR-versus-diffusion debate and would provide the first systematic study of DP fine-tuning for VAR. The released code and the detailed implementation patches for Opacus compatibility are concrete contributions, and the paper is honest about the limitations of its DP compute-cost analysis. However, the headline comparison with DiffFit is undermined by an internal contradiction in Table 1 and by the use of externally quoted baseline FID numbers under a potentially different evaluation protocol; the DP results also contain inconsistencies between tables. The significance is therefore conditional on correcting these issues.","major_comments":[{"comment":"The claim that 'FFT outperforms [DiffFit] across all models and datasets' is contradicted by Table 1 on CUB-200-2011: DiffFit achieves FID 5.48, while VAR-d16 FFT achieves 5.74 and VAR-d20 FFT achieves 5.58, so both VAR FFT variants are worse than DiffFit on that dataset. Additionally, on Oxford Flowers, VAR-d20 LoRA (11.16) beats VAR-d20 FFT (11.65), contradicting the statement in the same section that FFT performs best according to Table 1. The conclusion repeats the unsupported comparative statement. The authors should revise the comparative claims or provide corrected results.","section":"Section 3.2 / Table 1"},{"comment":"The FID comparison with DiT-XL-2 is not apples-to-apples: the DiT-XL-2 and DiffFit numbers are quoted from prior work, whereas the VAR numbers are computed with the protocol described in Section 3.1, which generates as many samples per class as there are test samples and uses a particular FID implementation. Without re-running the diffusion baselines under the same protocol, the favorable comparisons on Food-101, Oxford Flowers, and Stanford Cars are uncertain. The CUB contradiction already shows that the comparison is sensitive to evaluation choices; the authors should either re-run the baselines or explicitly report and discuss the protocol differences.","section":"Section 3.1 / Table 1"},{"comment":"The DP experiments contain an inconsistency: Table 2 reports VAR-d16 LoRA k=128 FID 63.24 and VAR-d20 LoRA k=128 FID 59.29, while Table 3 reports ε=10, k=32 FIDs of 60.24 and 63.38 for the same models; Appendix B lists k=128 as the augmentation multiplicity for DP-ε-10. The ordering between k=32 and k=128 is also non-monotonic across model sizes. Since the augmentation-multiplicity conclusions depend on these comparisons, the authors should state which configuration is the main reported DP result, reconcile the table entries, and explain the non-monotonic behaviour.","section":"Section 4.2 / Tables 2 and 3 / Appendix B"},{"comment":"The paper states that DP adaptations of VAR 'suffer from low generation quality' and 'necessitate further research', but it does not compare against any differentially private diffusion-model baseline (such as DPDM, DP-LDM, or DP-LoRA) under the same evaluation protocol. If the claim is meant only as an absolute statement about VAR's DP performance, that should be stated explicitly; if it is meant as a comparative statement against DM DP adaptations, the missing baseline comparison is a load-bearing gap.","section":"Section 4.2 / Abstract"}],"minor_comments":[{"comment":"The citation 'Gal et al.' in the Introduction is incomplete; it should include the full reference with year and venue for 'An Image is Worth One Word'. The first reference in the bibliography, 'The mnist database of handwritten digits', is also malformed and unattributed.","section":"References"},{"comment":"The x-axis of Figure 2 has tick labels '1 100 1000 19000 ...' that do not clearly indicate whether a log scale is used and at which step counts the curves are evaluated; please clarify the axis scale and the reporting interval.","section":"Figure 2"},{"comment":"In Table 9, the Stanford Cars entry for VAR-d20 FFT reads '59.01' without a percent sign, unlike the other entries; this appears to be a typo.","section":"Table 9"},{"comment":"The motivation mentions 'medical data generation' as a downstream task, but no medical dataset is used in the experiments; either add such an experiment or remove this motivation.","section":"Abstract / Section 1"},{"comment":"Table 2 reports only k=1 for LNTuning, although Appendix B specifies k=128 for DP LNTuning; the missing k=128 LNTuning result should be reported or explicitly explained.","section":"Table 2"},{"comment":"The statement that LoRA 'matches FFT's FID performance' is not well supported by Table 1, where, for example, VAR-d20 FFT achieves 5.38 on Food-101 while VAR-d20 LoRA achieves 6.97, and on Stanford Cars the FFT/LoRA gap is 6.31 vs 9.42; please qualify this claim.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style empirical paper whose main value is its implementation and benchmarking effort. The internal contradictions in Table 1 and the DP tables are fixable in revision, but they currently invalidate the paper's headline claims as stated. I would ask the authors to re-run or transparently contextualize the DiffFit baselines, correct the comparative statements, and reconcile the DP table entries before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first benchmark of PEFT and DP fine-tuning for VAR, with real engineering patches to make Opacus work and code to match. It deserves to be read as an empirical contribution, but the headline claim is overstated in a way that is checkable from the paper's own Table 1. On CUB-200-2011, DiffFit (5.48) beats both VAR FFT variants (5.58 for d20, 5.74 for d16). So the sentence in Section 3.2 saying FFT outperforms DiffFit \"across all models and datasets\" is false as written, and the conclusion repeats the overstatement.\n\nWhat's actually new: nobody had benchmarked LoRA, LayerNorm tuning, and full fine-tuning for VAR before, and the authors also got DP-SGD working with VAR via buffer removal and forward-pass patching, which are non-obvious implementation steps. They release code and a compute-cost analysis across model sizes, including the larger d24 and d30 variants. The DP experiments honestly show that private adaptation of VAR is still poor, and they include an ablation of where LoRA should be placed, which is a useful sanity check. The appendix candidly flags the bias in profiling DP compute and declines to report misleading PFLOPS totals.\n\nThe contradiction above is the main issue. Secondary: the DiT-XL-2 numbers are quoted from prior work rather than re-run under the paper's own FID protocol (samples per class, FID implementation), so the favorable comparisons on Food-101 and Oxford Flowers could shift if measured consistently. Also, the DP conclusions rest on a single dataset (Oxford Flowers), which is acceptable for a first benchmark but not enough to claim a broad failure of DP for VAR. Minor: the abstract says VAR outperforms DMs, but only DiffFit, one specific DM adaptation method, was compared.\n\nI'd take this as a solid workshop-level empirical contribution. The fix is straightforward: correct the CUB-200-2011 claim, caveat or re-run the baselines, and temper the generalization from one DP dataset. It deserves referee time because the gap is real and the code is reproducible; the overstatement is a fixable flaw, not a fatal one.","headline":"Useful first benchmark on VAR adaptation with reproducible code, but the paper's own Table 1 contradicts its headline claim on CUB-200-2011.","tokens_in":15326,"tokens_out":2207,"would_cite":true,"duration_ms":24774,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned Vision AutoRegressive image models beat diffusion adaptation on small datasets, but private fine-tuning lags far behind.","keywords":["vision autoregressive model","image generation","fine-tuning","parameter-efficient fine-tuning","LoRA","LayerNorm tuning","differential privacy","DP-SGD"],"falsifier":"Rerun DiT-XL-2 with DiffFit under the paper's stated protocol, generating as many images per class as the test set and computing FID against the test split with the same implementation, and compare with the Table 1 VAR numbers; the CUB-200-2011 row, where the paper's own table shows DiffFit at 5.48 versus VAR FFT at 5.74 and 5.58, is the decisive cell. For the DP claim, run DP fine-tuning of VAR-d16 on Oxford Flowers with augmentation multiplicity beyond k equal to 128 and more training steps; if FID approached the non-private range of roughly 12 to 13, the claim that private adaptation founders would be contradicted.","tokens_in":14341,"feed_emoji":"🖼️","tokens_out":10914,"duration_ms":115252,"temperature":0.7,"pith_summary":"The paper asks whether Vision AutoRegressive (VAR) generators, a recently introduced alternative to diffusion models, can be adapted to narrow downstream tasks such as medical data generation. It implements full fine-tuning, LoRA, and LayerNorm tuning for VAR, and benchmarks them against the diffusion state-of-the-art adaptation method, DiffFit, on five image datasets. The central finding is that VAR adaptations converge in very few steps and that full fine-tuning reaches lower Frechet Inception Distance (FID) than DiffFit on the reported benchmarks, while LoRA nearly matches full fine-tuning with a small parameter budget. The paper then adds differentially private fine-tuning via DP-SGD and reports that the resulting models need very large privacy budgets and still produce poor images, so private adaptation of VAR remains an open problem.","feed_headline":"Fine-tuned VAR beats diffusion baselines in five image datasets","feed_subtitle":"Full fine-tuning tops the diffusion state of the art; private adaptation still founders.","key_machinery":"The object that carries the argument is VAR's next-scale prediction: instead of predicting tokens in raster-scan order, VAR predicts a sequence of 2D token grids from low to high resolution, trained with a direct cross-entropy token-prediction loss. The authors credit this objective with VAR's fast convergence, since the gradient signal is not diluted across noise levels the way a diffusion denoising loss is. The adaptations under study are full fine-tuning (all parameters updated), LoRA (low-rank adapters inserted into the attention, projection, and Adaptive LayerNorm modules), LayerNorm tuning (only Adaptive LayerNorm parameters), and, for privacy, DP-SGD with per-sample gradient clipping, Gaussian noise, and augmentation multiplicity; making these work required patching the model's buffers and its forward function so that per-sample gradient accounting becomes possible.","core_discovery":"The paper's central discovery, as its authors state it, is that a Vision AutoRegressive generator can be adapted to small domain-specific image datasets more effectively than the diffusion model representing the current state of the art: full fine-tuning of VAR-d16 and VAR-d20 reaches lower FID than DiffFit on the benchmark datasets they report, while costing far fewer training steps, and LoRA nearly matches full fine-tuning with only a fraction of the parameters. A second, negative discovery is that differentially private fine-tuning built on DP-SGD does not transfer cleanly to VAR: on Oxford Flowers even with epsilon equal to 10 and augmentation multiplicity k equal to 128, LoRA reaches FID around 63, compared with non-private LoRA FID around 13, and convergence is slow. The authors present these results as evidence that autoregressive image models are a viable and efficient alternative to diffusion models for non-private adaptation, and that private adaptation of VAR still needs fundamentally better methods.","pith_inferences":["The paper's own Table 1 contains one row that the headline 'outperforms DiffFit across all models and datasets' does not obviously cover: on CUB-200-2011, DiffFit's FID of 5.48 is lower than VAR-d16 FFT's 5.74 and VAR-d20 FFT's 5.58, so a like-for-like reproduction of that cell is the first thing to check.","The buffer and forward-pass patches described in the appendix are transferable: any autoregressive image model with registered non-trainable buffers and a multi-argument forward method could use the same conversion to properties and a single concatenated input tensor to make DP-SGD-style per-sample accounting work.","A natural next step, not tested in the paper, is a noise schedule matched to next-scale token prediction, for instance clipping gradients at the scale level or adding token-level noise multiplicity analogous to what diffusion-model DP work calls noise multiplicity.","If the head-to-head comparison is reproduced under one unified FID protocol, LoRA would likely become the default non-private VAR adapter, since it nearly matches full fine-tuning while requiring far fewer trainable parameters."],"forward_implications":["Non-private VAR adaptation is a usable alternative to diffusion fine-tuning: full fine-tuning gives the best FID, LoRA gives near-full-tuning quality with a small trainable parameter budget, and LayerNorm tuning is the cheapest adapter.","Because VAR converges in a few thousand steps, adaptation runs can be much shorter than diffusion fine-tuning, especially on small datasets.","The patched implementations released with the paper should let other researchers apply LoRA and DP-SGD to VAR-family autoregressive image models.","Differentially private adaptation of VAR is not yet practical: acceptable image quality appears to require implausibly large privacy budgets, so privacy-preserving uses remain open.","Using augmentation multiplicity, the paper's DP experiments can be extended toward better private utility, though the reported gains are modest and the compute cost scales directly with the number of views."],"supporting_citations":[{"why":"Introduces the VAR next-scale autoregressive architecture that this paper fine-tunes.","marker":"[Tian et al., 2024]"},{"why":"DiffFit, the diffusion-model adaptation strategy that serves as the comparison baseline; its FID numbers are quoted.","marker":"[Xie et al., 2023]"},{"why":"Defines DiT-XL-2, the diffusion model whose DiffFit results are used as the baseline in Table 1.","marker":"[Peebles and Xie, 2023]"},{"why":"Defines Frechet Inception Distance, the generation-quality metric used for all comparisons.","marker":"[Heusel et al., 2017]"},{"why":"DP-SGD, the private training algorithm whose clipping and noise mechanism the DP adaptations build on.","marker":"[Abadi et al., 2016]"},{"why":"Augmentation multiplicity, the technique the paper tests to improve differentially private fine-tuning.","marker":"[De et al., 2022]"}],"fun_headline_variants":["VAR fine-tuning beats DiffFit on FID with fewer steps","Private VAR adaptation lags: FID 63 vs 13 on Flowers","LoRA nearly matches full fine-tuning; DP adaptation lags","Vision AutoRegressive adaptation outperforms diffusion, but DP struggles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison against diffusion rests on the assumption that the published DiffFit FID numbers were produced under the same evaluation protocol as the new VAR numbers, because the diffusion baselines were not rerun in this paper.","fun_headline_variants_meta":{"raw":{"variants":["VAR fine-tuning beats DiffFit on FID with fewer steps","Private VAR adaptation lags: FID 63 vs 13 on Flowers","LoRA nearly matches full fine-tuning; DP adaptation lags","Vision AutoRegressive adaptation outperforms diffusion, but DP struggles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":2964,"prompt_tokens":882,"completion_tokens":2082,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2008}},"tokens_in":498,"tokens_out":2082,"duration_ms":16093,"temperature":1.0,"reasoning_tokens":2008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:07:38.828933+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun DiT-XL-2 with DiffFit under the paper's stated protocol, generating as many images per class as the test set and computing FID against the test split with the same implementation, and compare with the Table 1 VAR numbers; the CUB-200-2011 row, where the paper's own table shows DiffFit at 5.48 versus VAR FFT at 5.74 and 5.58, is the decisive cell. For the DP claim, run DP fine-tuning of VAR-d16 on Oxford Flowers with augmentation multiplicity beyond k equal to 128 and more training steps; if FID approached the non-private range of roughly 12 to 13, the claim that private adaptation founders would be contradicted.","supporting_citations":[],"review_version":1}