{"id":"4bae58ca-379a-4850-aec2-bd5b017540fc","arxiv_id":"2412.16112","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CLEAR replaces full attention in pre-trained diffusion transformers with local circular-window attention and distills the teacher into a student that keeps quality at a fraction of the compute.","lead":"Diffusion Transformers for image generation are slow because attention grows quadratically with resolution. This paper shows that replacing full attention with local circular windows, then fine-tuning only attention layers on 10K self-generated images, can make a pre-trained model like FLUX.1 run up to 6.3x faster at 8K while keeping output quality about the same.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Global-structure stress test is missing; the paper's own Appendix D admits possible loss of long-range coherence, so the 'comparable results' claim may not generalize beyond aggregate COCO FID.","rationale":"The paper's central empirical claim has independent support: the distillation setup is clearly specified, the FLOPs and timing tables are internally consistent, and the appendix provides a second DiT (SD3.5-L) and zero-shot plugin results. I do not regard the missing Neighborhood Attention baseline or the absence of error bars as decisive, because the core question is whether local-only attention can match the teacher in the first place. The load-bearing point is that the method's defining design choice—removing all direct long-range image-image interactions—is validated only by qualitative locality evidence at one scale and is explicitly conceded in Appendix D to risk losing global structure. Aggregate COCO metrics cannot detect a systematic failure on global-coherence prompts, so a targeted stress test is the right arbiter. The proposed global-token ablation isolates whether adding a minimal amount of global context changes quality; if it does, the 'comparable' claim is only partially true and should be stated with that caveat. This does not move the verdict from the reader's CONDITIONAL: the paper should be published with this condition addressed or clearly bounded.","tokens_in":22641,"tokens_out":13061,"duration_ms":117855,"concrete_test":"Run the same 10K-iteration distillation with a variant that adds a small set of global image tokens (e.g., G=64 additional query tokens attending to all image keys) and evaluate it against CLEAR on (i) the COCO benchmark in Tab. 2 and (ii) a 200-prompt global-structure stress set (symmetry, reflections, explicit spatial relations), using FID, CLIP-T, and blind human preference. If the G-token variant is significantly better on the stress set at comparable FLOPs, then CLEAR's exclusion of all long-range image interactions is the source of a real quality gap and the comparable-results claim must be qualified; if it is not, the locality premise is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CLEAR achieves teacher-comparable quality rests on the premise that long-range image-image attention in pre-trained DiTs is dispensable (Sec. 3.1). The evidence is qualitative: attention-map visualization (Fig. 4) and RoPE perturbation (Fig. 5), both at a single scale and without a layer-by-layer or timestep-wise breakdown. More importantly, the paper's Appendix D concedes that 'due to the absence of explicit long-distance token interactions, our method may underperform in capturing overall structural properties, such as potential symmetry.' This is not a side remark: because only attention parameters are trainable (Sec. 3.3), the student cannot recover global layout except by routing it through many local hops; whether that routing is sufficient is never tested. Aggregate FID/CLIP-T parity on COCO (Tab. 2) does not rule out systematic failure on prompts requiring image-wide coherence (e.g., symmetry, reflection, multi-object spatial relations). If such failures exist, the 'comparable results' claim holds only on average and not as a general statement about linearizing DiTs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLEAR, a convolution-like local circular-window attention mechanism that replaces all attention layers in a pre-trained diffusion transformer while keeping text-token attention global. Only attention-layer parameters are fine-tuned, using 10K self-generated 1024x1024 samples and 10K iterations with a distillation objective combining flow matching, prediction matching, and attention-output matching. The authors report aggregate quality comparable to the original FLUX-1.dev on 5,000 COCO validation images (e.g., against-real FID 34.88 vs 34.93 at r=32), a 99.5% reduction in attention computation, and a 6.3x wall-clock speedup at 8K resolution. They also report zero-shot transfer to FLUX-1.schnell and ControlNet, and multi-GPU patch-parallel inference.","tokens_in":22829,"tokens_out":5131,"duration_ms":45095,"significance":"If the parity and efficiency results hold, CLEAR is a simple and practical way to convert a pre-trained DiT into a linear-complexity model, which is directly relevant for ultra-high-resolution generation. The paper's strengths include independent against-real metrics at 1024x1024 (FID, LPIPS, CLIP-T), arithmetically consistent efficiency numbers in Tab. 7, cross-model and plugin generalization experiments, and public code and models. The main caveat is that the headline 'comparable results' claim rests on aggregate COCO statistics and on a locality premise that is supported by qualitative evidence only; the paper's own Appendix D concedes a possible weakness on global structural properties such as symmetry.","major_comments":[{"comment":"Appendix D concedes that 'due to the absence of explicit long-distance token interactions, our method may underperform in capturing overall structural properties, such as potential symmetry.' This is load-bearing for the central parity claim, because with only attention parameters trainable (Sec. 3.3), the student cannot recover image-wide layout except by routing information through many local hops. Aggregate against-real FID/LPIPS/CLIP-T numbers in Tab. 2 cannot detect systematic failures on prompts requiring mirror symmetry, reflection, or multi-object spatial relations. Please add a targeted stress-test suite or per-category breakdown (e.g., symmetry, object counting, relative positions), and if any category degrades, qualify the 'comparable results' claim accordingly.","section":"Appendix D; Sec. 3.3; Tab. 2"},{"comment":"The main parity comparison is reported without any error bars or repeated-run statistics. Since the headline differences are small (against-real FID 34.88 vs 34.93; CLIP-T 31.00 vs 31.06 for r=32), it is important to know the run-to-run variability of the evaluation metrics. Please report multiple evaluation seeds or bootstrap confidence intervals on the 5,000-image sample, so that 'comparable to the teacher' can be distinguished from noise.","section":"Tab. 2; Sec. 4.2"},{"comment":"The premise that pre-trained DiTs rely primarily on local interactions is supported by a single attention-map visualization and one RoPE perturbation experiment at one scale and on one model. Fig. 4 shows selected heads at an intermediate denoising step, and clipping RoPE relative distances is not equivalent to zeroing the attention weights. Please quantify locality across layers and timesteps (e.g., fraction of attention mass within radius r), and report the same locality check for SD3.5-L to substantiate the claimed cross-architecture generality of the locality principle.","section":"Sec. 3.1; Fig. 4; Fig. 5"}],"minor_comments":[{"comment":"The phrase 'Motivated on this' should read 'Motivated by this.'","section":"Sec. 3.2"},{"comment":"The sentence 'where z0 is denotes the feature' contains a grammatical error; it should read 'where z0 denotes the feature.'","section":"Sec. 3.3"},{"comment":"The sentence 'Lattn is applied on single transformer blocks of FLUX, whose layer indices are 20 ~ 57' is ambiguous; please specify whether Lattn is applied to all blocks 20-57 or to a selected subset.","section":"Sec. 4.1"},{"comment":"The caption of Tab. 10 explains why r=16 at 1024x1024 is unavailable, but the main text does not mention this limitation of the multi-GPU scheme at lower resolutions; please add a sentence.","section":"Sec. 4.3; Tab. 10"},{"comment":"The 'Maybe' entry for Agent Attention in the Locality column is never explained; a one-sentence clarification would help the reader interpret the table.","section":"Tab. 1; Sec. 2.3"},{"comment":"In Eq. (7), the notation Vp is used without specifying whether it contains only image tokens of patch p or includes text tokens as well; please clarify.","section":"Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the central idea is likely defensible after targeted additions. I do not see evidence of duplicate submission or problematic citation behavior. The main risk is overgeneralization of the word 'comparable' beyond the COCO aggregate: adding a global-structure stress test and error bars would materially strengthen the paper. My recommendation is major revision rather than rejection because the load-bearing issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline, two things. The method is not architecturally new: CLEAR is Neighborhood Attention with a circular window plus global text tokens, and the paper says as much itself — NA is the only scheme satisfying all four of its own criteria. What is new is the recipe around it: fine-tuning only the attention layers of a pre-trained DiT for 10K iterations on 10K self-generated 1024x1024 samples with a loss matching the teacher's denoising predictions and attention outputs, and the findings that come with the recipe — zero-shot transfer of those distilled layers to FLUX-1.schnell and to pre-trained ControlNet, and a linear-complexity multi-GPU patch-parallel inference scheme. The efficiency numbers are arithmetically consistent: the FLOPs in Table 7 match the 99.5% attention-compute reduction and 6.3x speedup at 8K, and the paper is candid that at 1024 the practical speedup is negligible.\n\nThe quality claim holds as an aggregate claim, and that is what the data actually shows. The independent evidence is decent: against-real FID 34.88 vs 34.93 and CLIP-T 31.00 vs 31.06 for r=32 with distillation, and similar parity on SD3.5-Large in the appendix. Also note the paper itself separates 'against original' from 'against real' columns, so the high CLIP-I and FID-to-teacher scores that the distillation loss directly optimizes are not presented as the primary evidence — a point the circularity concern should credit.\n\nSoft spots, in proportion. The missing quantitative Neighborhood Attention baseline is the most annoying: they benchmark six alternatives in Table 2 but not the one scheme they identify as satisfying all four factors. Easy fix, should be required. No error bars; single-seed evaluation across the board is worth at least a sentence of robustness discussion. The global-structure concern is real but not load-bearing: Appendix D openly concedes possible underperformance on overall structural properties like symmetry, so the honest reading is 'comparable on average,' which is what the metrics support. The locality evidence (Figs. 4 and 5) is suggestive on one model, but successful distillation on two DiT families is reasonable empirical support.\n\nThis paper is for people working on efficient attention, high-resolution diffusion inference, and distillation. It deserves a serious referee; the missing baseline and error bars are revision-level fixes, and the paper ships code and models. My recommendation: send it to review, and push for those fixes.","headline":"CLEAR is neighborhood attention with a circular window plus a smart distillation recipe; the aggregate parity claim holds, but the closest baseline is missing from the quantitative table and the paper's own symmetry caveat should temper how the result is sold.","tokens_in":23422,"tokens_out":5441,"would_cite":true,"duration_ms":45226,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pre-trained diffusion transformer can be made linear by replacing every attention layer with a circular local-window mechanism and fine-tuning only those layers for 10K iterations, yielding output comparable to the original.","keywords":["diffusion transformers","linear attention","local window attention","knowledge distillation","high-resolution image generation","efficient inference","text-to-image generation","FLUX"],"falsifier":"Compute, for FLUX-1.dev, the fraction of image-to-image attention mass that falls outside the r=32 circular window, aggregated over all attention layers and denoising timesteps; if that fraction is substantial and a linearized student trained with the same 10K-iteration recipe on any DiT with such long-range heads fails to reach teacher parity, the locality premise is refuted. A complementary check supplied by the paper itself is wall-clock speed: at 1024x1024 and below, CLEAR is no faster than the original model, so the linear-time advantage is an empirical claim that must be re-measured at each target resolution.","tokens_in":22360,"feed_emoji":"⚡","tokens_out":11573,"duration_ms":90094,"temperature":0.7,"pith_summary":"The paper asks whether a pre-trained Diffusion Transformer can be converted from quadratic to linear attention without losing generative quality, and answers yes on the FLUX-1.dev text-to-image model. The recipe is CLEAR, a convolution-like local attention in which each image query attends to all text tokens plus image tokens inside a circular window of fixed radius, so per-query work is constant and total cost is linear in the number of image tokens. Only the attention layers are fine-tuned, for 10K iterations on 10K images generated by the teacher itself, using a distillation objective that also aligns predictions and per-layer attention output with the frozen teacher. The student then scores comparably to the teacher (against-real FID 34.88 versus 34.93, CLIP-T 31.00 versus 31.06 at r=32), cuts attention computation by 99.5 percent at 8K resolution, and generates 8K images 6.3 times faster. If the claim holds, expensive pre-trained DiTs can be made linear for about a day of fine-tuning, with the distilled local layers transferring zero-shot to sibling models and plugins.","feed_headline":"Linear-time attention matches FLUX image quality at 8K","feed_subtitle":"A 10K-step fine-tune of attention layers alone cuts 99.5% of attention compute and speeds 8K generation 6.3x.","key_machinery":"The load-bearing object is CLEAR's circular-window attention mask: a fixed-radius Euclidean neighborhood around each query that keeps the per-query key-value count constant as resolution grows, making overall complexity $\\mathcal{O}(n)$ in the number of image tokens and costing about $\\pi/4$ the compute of a square window. It is the only scheme among the surveyed families (formulation variation, key-value compression, key-value sampling) that satisfies all four factors the paper identifies as essential: locality, formulation consistency with softmax attention, high-rank attention maps, and preservation of raw query/key/value features. The second mechanism is the distillation objective: the flow-matching loss $\\mathcal{L}_{\\text{fm}}$ together with prediction consistency $\\mathcal{L}_{\\text{pred}}$ and per-layer attention-output consistency $\\mathcal{L}_{\\text{attn}}$ against the frozen teacher, trained on 10K self-generated samples chosen for distributional closeness to the teacher's training data. The third is the patch-wise text-token approximation $O_{\\text{text}}\\approx \\frac{1}{N}\\sum_{p=1}^{N}\\mathrm{softmax}(Q_{\\text{text}}K_p^{\\top}/\\sqrt{c})V_p$, which removes the need to synchronize all key-value tokens for text queries and thereby makes multi-GPU patch-parallel inference communication-light.","core_discovery":"The central claim is that pre-trained DiTs such as FLUX-1.dev rely predominantly on local token interactions, so long-range image-to-image attention can be discarded and re-learned locally without hurting generation quality. The paper supports this with attention-map visualizations and with rotary-position-embedding perturbation experiments: clipping remote feature distances to a small radius leaves output largely intact, while perturbing local distances collapses it. On this basis it replaces every attention layer with CLEAR, whose mask admits a text-image pair whenever either token is a text token or the two image tokens lie within Euclidean radius $r$ ($M_{ij}=1$ if $i\\le n_{\\text{text}}$ or $j\\le n_{\\text{text}}$ or ${d^{(x)}_{ij}}^2+{d^{(y)}_{ij}}^2