{"id":"82e5cd63-0e0c-41fd-a7a1-a31b0f165463","arxiv_id":"2411.18552","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FAM Diffusion lets pre-trained Stable Diffusion models generate images at up to 4x resolution without training by steering low-frequency structure and attention maps from a native-resolution pass.","lead":"FAM Diffusion adds two small steering modules to a pre-trained image diffusion model, letting it generate images at up to 4x its training resolution without retraining: one module keeps the global structure from a low-resolution draft, the other stabilizes local textures using attention maps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AM module's quantitative benefit is unproven: Eq. 7 up-samples native attention maps without any ablation showing FM+AM beats FM alone, so the claimed pair-of-modules advantage rests on a single qualitative figure.","rationale":"The paper's own novelty argument rests on AM: the introduction and related work emphasize that local-texture inconsistency is 'largely ignored' and that AM fixes it; the abstract lists it as one of two modules. The strongest claim is about the pair of modules, not FM alone. Yet the only evidence for AM is qualitative: Fig. 3e and Fig. 4. In contrast, FM has a formal derivation, a time-aware ablation (Fig. 7), and is the main mechanism in Eqs. 4-5. If a careful quantitative ablation showed AM adds no significant improvement, the paper would still have a working FM method, but it would not support the central two-module SOTA claim. This concern is therefore load-bearing for the headline claim. It is also easy to settle with a standard three-arm ablation. The reader flagged the same fragility of upsampled attention maps in their weakest_assumption; I narrow that to a single decisive experiment. The derivation itself is sound, so I do not see a correctness error requiring rejection; the right verdict remains CONDITIONAL pending this ablation.","tokens_in":13234,"tokens_out":12722,"duration_ms":115399,"concrete_test":"Run a controlled ablation on the same evaluation set as Table 1 at 2× and 4× scales: (a) FM-only, (b) FM+AM with λ=0.7, (c) FM+AM with λ=0 (no AM), each with at least 3 seeds; report FIDc, KIDc, CLIP mean and 95% CI. If (b) does not significantly improve over (a) on all metrics, the AM contribution is not established and the central two-module claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that FM and AM jointly deliver state-of-the-art high-resolution generation. FM is supported by a derivation and by the FM-only ablation shown in Fig. 3d. AM, however, is validated only by one attention visualization (Fig. 4) and one qualitative example (Fig. 3e). No quantitative ablation isolates AM: Table 1 compares full FAM against baselines, not against FM-only. The AM formula (Eq. 7) spatially upsamples the native attention matrix M^n by factor s and blends it with the high-resolution matrix M^m. Because M^m has s^2 more tokens, upsampling forces all query tokens inside a native cell to share the same attention distribution; whether such maps remain semantically valid for the diffused-from-upsampled high-res trajectory is not established. If AM is neutral or harmful on average, the paper's named novelty beyond FM collapses, and the 'best in all cases' claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes FAM Diffusion, a training-free method for generating images at resolutions higher than a pretrained latent diffusion model's training resolution. The method combines two inference-time modules: a Frequency Modulation (FM) module that mixes low-frequency Fourier components of an upsampled native-resolution diffused latent into the high-resolution denoising trajectory, and an Attention Modulation (AM) module that blends attention maps from the native-resolution denoising pass into selected layers of the high-resolution pass. The authors claim that FAM Diffusion integrates seamlessly into any latent diffusion model, requires no additional training, achieves state-of-the-art FIDc, KIDc, and CLIP scores at 2x, 3x, and 4x resolutions, and adds negligible latency compared with direct high-resolution inference. The paper includes a derivation of the time-domain equivalent of frequency modulation, qualitative results, and ablations on SDXL and other Stable Diffusion variants.","tokens_in":13398,"tokens_out":8408,"duration_ms":74299,"significance":"If the results hold, this is a practically valuable contribution: it offers a simple, one-pass, training-free alternative to patch-based high-resolution generation methods, with a plausible frequency-domain mechanism for preserving global structure and an attention-based mechanism for local texture consistency. The derivation in Appendix B.1 is a strength, as is the broad experimental coverage across backbones, aspect ratios, and the ControlNet integration. The main significance is currently limited by the lack of quantitative evidence isolating the AM module, the ambiguous evaluation protocol vis-a-vis baseline subsets, and the absence of sensitivity analysis for the hand-defined frequency mask.","major_comments":[{"comment":"The paper's central contribution is the pair of FM and AM modules, but no quantitative experiment isolates the contribution of AM. Table 1 compares the full FAM method against external baselines, while the ablation in Sec. 4.3 is qualitative (Fig. 3d vs 3e) and Fig. 4 is a single attention visualization. This leaves open the possibility that the reported gains come entirely from FM and that AM is neutral or harmful on average. Please add a quantitative ablation comparing FM-only with FM+AM on the same evaluation split, reporting FIDc, KIDc, and CLIP, ideally with multiple seeds.","section":"§3.3, §4.3 (Fig. 3, Table 1)"},{"comment":"The paper states that it randomly samples 10K images and 1K captions from LAION-5B, but it does not state whether the baseline numbers in Table 1 are recomputed on this same subset or taken from prior publications. Because FID and KID depend on the real-image reference set, comparisons across different subsets are not meaningful. Please specify the re-evaluation protocol; if baselines were not rerun, either rerun them on the same split or use the exact splits from the cited papers, and provide the random seed and caption list for reproducibility.","section":"§4.1 (Evaluation set) and Table 1"},{"comment":"The time-varying rectangular Fourier mask K(t) with a fixed cutoff scale c=0.5 is a crucial, hand-designed component. The paper does not analyze sensitivity to c or to the mask shape, despite claiming that the method is backbone- and resolution-agnostic. This is load-bearing for the generality claim. Please provide an ablation over c (e.g., 0.25, 0.5, 0.75) and, if possible, justify the rectangular low-pass assumption in latent space, for example by visualizing the effective frequency bands at several timesteps.","section":"§3.2, Eq. (11), Appendix B.1"},{"comment":"The claim that 'FAM diffusion achieves the best overall performance on FIDc, KIDc, and CLIP Score in all cases' is not strictly supported by Table 1 as printed. At 3x, HiDiffusion+FAM has CLIP 32.26 versus SDXL+FAM's 32.25, and the KIDc values are tied at 0.010. Please clarify the intended comparison (e.g., FAM applied to the best base model versus all rows containing FAM) and report all metrics with a consistent number of significant digits so that ties are not ambiguous.","section":"§4.2, Table 1"}],"minor_comments":[{"comment":"The text says 'We empirically set λ used in Eq (6) to 0.7', but Eq. (6) defines softmax attention and contains no λ; the parameter appears in Eq. (7). The cross-reference should be corrected.","section":"Appendix B.2"},{"comment":"The text says 'guidance from the denoised latent ~zt becomes more significant as t→0', but ~zt denotes the diffused latent from the forward process, not the denoised latent; please align the terminology with Eq. (4).","section":"§3.2"},{"comment":"Panel (e) is labeled 'FM-AM' while the text refers to it as 'AM' or the combination; use a consistent name such as 'FM+AM'.","section":"Figure 3 caption"},{"comment":"The evaluation reports point estimates without seeds or confidence intervals; given the stochasticity of diffusion sampling, please include seeds or report the variance across multiple runs for the main table.","section":"§4.1"},{"comment":"The KIDc values have inconsistent decimal places (e.g., 0.010 versus 0.0084), which makes ties and small differences hard to interpret; standardize the formatting.","section":"Table 1"},{"comment":"There is a typo: 'natire resolution' should be 'native resolution'.","section":"Related Work"},{"comment":"References [11] and [12] are the same work (ECCV and arXiv versions); consider citing one consistently to avoid duplication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central derivation for FM appears sound and the practical framing is strong, but the AM module's contribution needs a quantitative ablation and the evaluation protocol needs clarification. These are fixable within revision scope, so major revision is appropriate. I would also encourage the editor to require the authors to release the exact evaluation split and seeds, as the random subset choice directly affects comparability of the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look. The paper's real contribution is the Frequency Modulation module: a time-varying Fourier-domain replacement for DemoFusion's skip residual that preserves global structure while leaving high frequencies to the denoiser. The derivation in Appendix B.1 checks out, and the FM-only ablation (Fig. 3d) shows a clear improvement over skip residual. The low latency overhead and the clean integration with HiDiffusion and multiple SD backbones are also genuinely nice. The Attention Modulation idea is plausible and the qualitative figures suggest it helps with local texture, but the evidence stops there.\n\nThe soft spots are real but fixable. Most important: there is no quantitative ablation isolating AM. Table 1 compares full FAM against baselines, never FAM with and without AM. So the claim that the pair of modules delivers the state of the art is not fully supported. The stress-test note is correct on this point. However, I would not say the paper collapses if AM turns out neutral; FM alone is already a meaningful step beyond skip residual. What collapses is the \"best in all cases\" claim and the novelty attributed to AM.\n\nThe other weaknesses are standard for this area: no error bars or seed variance, a randomly sampled 10K LAION subset that may not match the baselines' subsets, hyperparameters (c, lambda, AM layer choice) tuned on the same benchmark without sensitivity analysis, and missing baselines like ResMaster and HiPrompt. These are all addressable in revision.\n\nThe math is sound, the writing is clear, and the method is simple enough to reproduce. I would send this to peer review, but I would insist on the FM-only vs. FM+AM comparison, plus at least a small sensitivity study on lambda and c.","headline":"A genuinely useful training-free high-resolution generation method whose frequency-modulation core is well supported, but the attention-modulation module's payoff is asserted more than demonstrated.","tokens_in":13956,"tokens_out":1402,"would_cite":true,"duration_ms":14732,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FAM diffusion claims a training-free way to generate high-resolution images from Stable Diffusion by mixing in frequency and attention information from a low-resolution draft.","keywords":["high-resolution image generation","diffusion models","frequency domain","attention modulation","training-free adaptation","Stable Diffusion","latent diffusion"],"falsifier":"Generate the same prompt at 3x with the frequency mask cutoff swept from 0.2 to 0.8 and with Attention Modulation disabled; if the output quality barely changes or a texture-heavy scene (for example, a wall of bricks) still shows repetition, the claim that the low-frequency band of the diffused native latent carries the necessary structure fails. A sharper test: corrupt the low-frequency band of the native latent with noise. If the output remains structurally clean, FM is not actually using that band.","tokens_in":1595,"feed_emoji":"🖼️","tokens_out":1959,"duration_ms":46477,"temperature":0.7,"pith_summary":"The paper claims that high-resolution images can be generated from pretrained latent diffusion models without retraining by splitting the problem into two modulations: a frequency-domain mix that imports global structure from a low-resolution draft, and an attention mix that imports local semantic relationships from the same draft. Both modules are applied at inference time only, are small, and work with any latent diffusion model. Together they eliminate the repetitive patterns and texture inconsistencies that appear when sampling above training resolution. The authors report best FIDc, KIDc, and CLIP scores against prior training-free methods at 2x, 3x, and 4x scales while adding only 0.2 to 0.7 minutes of latency over direct inference. If true, this gives a practical way to upgrade existing text-to-image models to higher resolutions without costly retraining.","feed_headline":"Two modules let Stable Diffusion render crisp 4x images","feed_subtitle":"Frequency and attention mixing from a low-res draft removes repetition artifacts with near-zero extra latency.","key_machinery":"The central object is the Frequency Modulation (FM) equation (4), a time-varying high-pass filter $K(t)$ that is rectangular in the shifted Fourier domain. At each denoising step it keeps the high-frequency coefficients of the high-resolution latent and replaces low-frequency coefficients with those of the forward-diffused, upsampled native latent; equivalently, it adds a non-learnable convolutional layer with a global receptive field (Eq. 5). The second object is Attention Modulation (AM), which replaces each up-block self-attention matrix $M^m$ with a weighted average of the upsampled native attention matrix and the high-resolution one (Eq. 7, with $\\lambda = 0.7$), transferring semantic layout from the low-resolution pass.","core_discovery":"Starting from a native-resolution output, the method runs forward diffusion to obtain diffused latents, then in each high-resolution denoising step mixes the low-frequency Fourier coefficients of the diffused native latent with the high-frequency coefficients of the current high-resolution latent (Eq. 4), and blends high-resolution self-attention maps with upsampled native attention maps in up-block layers (Eq. 7). The paper's claim is that this two-part modulation, FM for global structure and AM for local texture, is sufficient to keep pretrained latent diffusion models coherent far outside their training resolution, with negligible latency cost.","pith_inferences":["Because FM injects a global receptive field at inference time, the same principle could plausibly be applied to other grid- or sequence-based generative models beyond image LDMs (for example, video or 3D generation), though the paper does not test this.","The rectangular Fourier mask and its cutoff scale $c = 0.5$ are hand-chosen; a natural extension the paper does not explore is adapting these per prompt, resolution, or base model, which could improve robustness across content types.","AM is applied only to self-attention in up-blocks; combining it with cross-attention or applying it at multiple up-blocks might yield further texture gains, but the paper only evaluates the single-layer setting.","The method's reliance on a native-resolution draft means generation becomes two-pass; for very large scales this could be extended to a hierarchical multi-pass version, at the cost of latency, which the paper does not address."],"forward_implications":["Any latent diffusion model can be used at 2x to 4x training resolution without retraining or architectural changes, simply by wrapping it in the two modulations.","The method integrates into single-pass models like HiDiffusion, preserving their low latency while improving quality, so it can serve as a general upscaling layer for efficient high-resolution generation.","Structural artifacts such as object repetition are reduced because the low-frequency band of the native draft anchors the global layout, while the denoiser retains control over high-frequency detail.","Local texture inconsistencies are reduced because attention maps from the native resolution encode which image regions are semantically related, and these relationships are transferred to the high-resolution pass.","The latency overhead is small (0.2 to 0.7 minutes at 2x to 4x), making the method practical for interactive and batch applications."],"supporting_citations":[{"why":"Introduces the test-time diffuse-denoise strategy and skip residuals that FAM diffusion extends and improves.","marker":"[3]"},{"why":"Provides the frequency-domain baseline (FouriScale) that FAM diffusion compares against and outperforms.","marker":"[12]"},{"why":"Presents the single-pass HiDiffusion baseline and the integration partner that demonstrates the method's modularity.","marker":"[34]"},{"why":"Supplies the SDXL backbone on which the main experiments are run.","marker":"[19]"},{"why":"Establishes the latent diffusion model framework and the notation the paper builds on.","marker":"[22]"},{"why":"Provides the attention-swapping idea that inspires the Attention Modulation mechanism.","marker":"[13]"},{"why":"Represents the architecture-modification baseline (ScaleCrafter) that motivates the training-free approach.","marker":"[7]"}],"fun_headline_variants":["Frequency and attention mods eliminate diffusion artifacts at scale","Two modules enable high-res generation without retraining","Diffusion models gain resolution flexibility via FM and AM","Near-zero latency fix for high-res latent diffusion","Structural and texture consistency via frequency and attention mixing"],"cache_read_input_tokens":16128,"weakest_assumption_plain":"The method assumes that a hand-chosen, time-varying rectangular cutoff in the Fourier domain reliably separates global structure from detail for every prompt and scale, and that upsampled native-resolution attention maps remain semantically valid guides for high-resolution textures.","fun_headline_variants_meta":{"raw":{"variants":["Frequency and attention mods eliminate diffusion artifacts at scale","Two modules enable high-res generation without retraining","Diffusion models gain resolution flexibility via FM and AM","Near-zero latency fix for high-res latent diffusion","Structural and texture consistency via frequency and attention mixing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2712,"prompt_tokens":846,"completion_tokens":1866,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":1793}},"tokens_in":462,"tokens_out":1866,"duration_ms":12284,"temperature":1.0,"reasoning_tokens":1793,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:05:16.330388+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate the same prompt at 3x with the frequency mask cutoff swept from 0.2 to 0.8 and with Attention Modulation disabled; if the output quality barely changes or a texture-heavy scene (for example, a wall of bricks) still shows repetition, the claim that the low-frequency band of the diffused native latent carries the necessary structure fails. A sharper test: corrupt the low-frequency band of the native latent with noise. If the output remains structurally clean, FM is not actually using that band.","supporting_citations":[{"cited_title":"DemoFusion: Democratising high- resolution image generation with no $$$","cited_arxiv_id":null,"evidence_quote":"Introduces the test-time diffuse-denoise strategy and skip residuals that FAM diffusion extends and improves."},{"cited_title":"HiDiffusion: Unlocking higher- resolution creativity and efficiency in pretrained diffusion models","cited_arxiv_id":null,"evidence_quote":"Presents the single-pass HiDiffusion baseline and the integration partner that demonstrates the method's modularity."},{"cited_title":"SDXL: Improving latent diffusion models for high-resolution image synthesis","cited_arxiv_id":null,"evidence_quote":"Supplies the SDXL backbone on which the main experiments are run."},{"cited_title":"High-resolution image syn- thesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Establishes the latent diffusion model framework and the notation the paper builds on."},{"cited_title":"Scalecrafter: Tuning-free higher- resolution visual generation with diffusion models","cited_arxiv_id":null,"evidence_quote":"Represents the architecture-modification baseline (ScaleCrafter) that motivates the training-free approach."}],"review_version":1}