{"id":"8d6e1c8a-48c9-4a02-a11c-e0162d647161","arxiv_id":"2507.01309","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Phase-aware sampling cuts Stable Diffusion's compute by roughly 2.4x to 5.7x with only small CLIP-score changes, and the accompanying FPGA accelerator turns this into 2.7x to 6.0x energy savings over an Nvidia V100 GPU.","lead":"SD-Acc is a jointly designed algorithm and chip for running Stable Diffusion image generation with less computation and energy. It skips most of the denoising network's work in the late refinement phase, reusing earlier results, and reports up to a few times less compute with roughly unchanged image quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing concern is that PAS's phase schedule and the 'without compromising image quality' claim rest on statistics and quality checks taken from the same MS-COCO/PartiPrompts calibration distribution, with no variance or held-out transfer evidence; the reader's CONDITIONAL verdict is…","rationale":"I read the paper in good faith as a co-design claim: an algorithm-level approximate-inference scheme plus an FPGA accelerator that together reduce StableDiff MACs by up to 5.7x and energy by 2.7x to 6.0x against a V100. The hardware side is credible: the address-centric dataflow, 2-stage streaming for nonlinearities, and adaptive reuse/fusion are described in enough detail to be internally consistent, and the ablation and roofline analysis show the reported speedups are close to the theoretical limits. The weak point is the algorithm-level foundation. The phase-division observation in Sec. III-A is computed on a small calibration set, and the framework in Sec. III-C selects hyperparameters using quality constraints evaluated on a subset of the same MS-COCO distribution later used for the headline quality metrics. The paper explicitly claims D* is robust to prompt randomness but provides no error bars, no sensitivity analysis, and no evaluation on an unseen prompt distribution. Because the entire MAC reduction depends on the phase boundary and the outlier-block set, this is the single most load-bearing assumption. The reader's weakest_assumption names exactly this transferability issue, so I agree. An end-to-end held-out evaluation with bootstrap confidence intervals would settle whether the quality-preservation claim actually transfers, and would also indirectly test the stability of the phase schedule across prompt distributions. Since this concern is the same one that motivated the conditional verdict, my read does not change the reader's verdict.","tokens_in":28007,"tokens_out":8210,"duration_ms":93403,"concrete_test":"Evaluate the framework-selected configuration (e.g., {Tsketch=25, Tcomplete=4, Tsparse=5, Lsketch=2, Lrefine=2}) on a held-out prompt set not used in calibration, such as 1,000 prompts from PartiPrompts or DrawBench, using the same PNDM-50 scheduler and one image per prompt. Compute CLIP score, FID, and IS relative to the unmodified model on the same held-out set, with bootstrap 95% confidence intervals over the generated images (e.g., 10,000 resamples). If the 95% interval for ΔCLIP excludes the quality-gate bound (-0.01) or the interval for ΔFID excludes the +1.0 bound, the 'without compromising image quality' claim fails to transfer to a new prompt distribution. This tests the full calibration-to-evaluation pipeline without re-tuning on the test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that phase-aware sampling (PAS) preserves image quality while cutting MACs by 2.4x to 5.7x. Every element of the schedule is derived from one shift-score analysis: Eq. (1) is averaged over 100 PartiPrompts prompts with PNDM-50 (Sec. III-A), the phase boundary D* and outlier set come from that curve (Eq. 2), and the final hyperparameters {Tsketch=25, Tcomplete=4, Tsparse=5, Lsketch=2, Lrefine=2} are selected by a quality-gated search on 5% of the MS-COCO target set (Sec. III-C, Fig. 7). Table II then reports quality on the same MS-COCO validation split, without error bars, while CLIP declines monotonically with compression and FID improves, an effect explained only post hoc. Thus the 'no quality compromise' result is in-sample: if D*, the outlier blocks, or the quality response shift with prompt distribution, scheduler, or model variant, the headline result does not transfer. The paper asserts D* is 'quite robust to the randomness of the prompt' but gives no variance or sensitivity analysis. This is load-bearing because the entire computational saving is gated on the two-phase structure and on the top-two-block outlier set; a shift in either would change the schedule, the reuse distances, and the quality/complexity trade-off. The hardware and dataflow contributions are plausible and internally consistent, but the algorithm-level claim is the foundation of the reported energy gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SD-Acc, an algorithm-hardware co-design for accelerating Stable Diffusion U-Net inference. At the algorithm level, it analyzes shift scores of U-Net activations across denoising timesteps (Eq. 1) and identifies a two-phase structure: a sketching phase and a refinement phase, separated by a transition timestep D* computed via a K-means-style objective (Eq. 2). During refinement, the method executes only the top Lrefine blocks of the U-Net, reusing activations from the nearest complete timestep, which yields reported MAC reductions of 2.39x to 5.68x across StableDiff v1.4, v2.1, and XL (Table II). A general framework (Sec. III-C) searches hyperparameters {Tsketch, Tcomplete, Tsparse, Lsketch, Lrefine} under quality and performance constraints. At the hardware level, the paper introduces an address-centric dataflow that maps convolutions onto a simple systolic array, a 2-stage streaming scheme for nonlinear operations, a reconfigurable vector processing unit, and adaptive reuse/fusion strategies (Secs. IV and V). The accelerator is synthesized on a VCU118 FPGA, and the paper reports up to 2.7x-6.0x energy savings over an NVIDIA V100 GPU (Fig. 19) and projected speedups of 2.2x-4.7x over V100 after scaling to 1 GHz and 4096 MACs (Fig. 20). The hardware claims are supported by per-optimization ablations and a roofline analysis, while the algorithm-level quality claim is the main load-bearing concern.","tokens_in":28275,"tokens_out":4971,"duration_ms":168556,"significance":"If the central claims hold, this is a useful retraining-free approximate-inference scheme for a widely deployed generative model, combined with a working FPGA implementation. The hardware work is substantial: the design is synthesized on VCU118, the ablations isolate each hardware optimization (address-centric dataflow, adaptive dataflow, 2-stage streaming), and the roofline accounting is internally consistent, with measured speedup reaching about 95% of the theoretical value for PAS-25/4. The phase-aware sampling idea is interesting and plausibly generalizable. However, the significance of the algorithm-level contribution depends on whether the 'without compromising image quality' claim survives evaluation on held-out prompt distributions and with proper statistical treatment, which the current manuscript does not provide. The comparison with prior accelerators and with CPU/GPU baselines is generally fair in its energy metrics, though the speed comparison in Fig. 20 is explicitly a projection rather than a measured result.","major_comments":[{"comment":"The headline claim of preserving image quality is evaluated in-sample. The hyperparameters are calibrated on 5% of the MS-COCO prompt set (Sec. III-C), and Table II reports CLIP/FID/IS on the same MS-COCO validation split, with no held-out prompt distribution, no variance across calibration seeds, and no variance across image generations. Since every schedule parameter (Tsketch, Tcomplete, Tsparse, Lsketch, Lrefine, D*) is derived from this calibration, the absence of a transfer experiment leaves the 'without compromising image quality' claim unsupported for new prompts, schedulers, or model variants. Please add an evaluation on a disjoint prompt dataset (e.g., PartiPrompts or DrawBench) and report confidence intervals over multiple calibration seeds and generated images.","section":"Sec. III-C and Table II"},{"comment":"The quality metrics in Table II are mutually inconsistent with the 'no quality compromise' claim. For every model, CLIP score monotonically decreases with MAC reduction (e.g., StableDiff v1.4: 0.3004 -> 0.2966 for PAS-25/5), while FID improves. The paper does not state a pre-specified acceptance criterion for quality preservation, and the monotone CLIP decline is evidence of quality loss under any fixed tolerance. The FID improvement is explained only post hoc as a regularization effect, which is not substantiated. Please specify a quality tolerance (e.g., ΔCLIP within measurement noise, FID not statistically worse) and report per-configuration error bars or a paired statistical test.","section":"Table II and Sec. VI-A"},{"comment":"The phase boundary D* is a single point estimate derived from averaged min-max-normalized shift scores over 100 PartiPrompts prompts with one scheduler (Sec. III-A), yet Tsketch=25 is applied uniformly to StableDiff v1.4, v2.1, and XL in Table II. No sensitivity analysis around D* or Tsketch is reported for any model. This is load-bearing because if the true transition timestep for a given model or prompt distribution differs from 25, the quality/complexity trade-off reported in Table II is not the one that a user would obtain in practice. Please report the computed D* for each model and a sweep over nearby Tsketch values for at least one model, together with the resulting quality metrics.","section":"Sec. III-A, Eq. (2), Table II"},{"comment":"The speedups over the NVIDIA V100 reported in Fig. 20 (2.2x-4.7x) are projections obtained by scaling the accelerator from 200 MHz and 1024 MACs to 1 GHz and 4096 MACs, not measured results. The DRAM bandwidth (38.4 GB/s) is not scaled correspondingly, which is important because the roofline position of the projected design changes with the compute-to-memory ratio. Please label Fig. 20 as a projected speedup, state whether the memory system is also scaled, and discuss the sensitivity of the reported speedup to this assumption.","section":"Sec. VI-F and Fig. 20"}],"minor_comments":[{"comment":"Eq. (1) cites [21] for the 50-timestep scheduler used in the shift-score analysis, but Sec. VI-A states that the PNDM scheduler [33] is used for sampling; please reconcile this citation inconsistency.","section":"Sec. III-A vs. Sec. VI-A"},{"comment":"The statement that 'D* is quite robust to the randomness of the prompt' is currently unsupported; please provide the supporting data, for example the distribution of D* across multiple calibration subsets.","section":"Sec. III-C"},{"comment":"The caption states '100 x 100 image generations'; please clarify whether this means 100 prompts with 100 images per prompt, and report the exact number of images used in the shift-score averaging.","section":"Fig. 4 caption"},{"comment":"The y-axis label 'Layer Buffer' appears to refer to the global buffer size; please correct the label and state the normalization point explicitly.","section":"Fig. 16 (right)"},{"comment":"The table would be easier to interpret if the hyperparameters for each model were included as columns or in the row labels, since Tcomplete differs between v1.4 (4) and v2.1/XL (3).","section":"Table II"},{"comment":"There are several typos, e.g., 'approiate' for 'approximate' and 'Deepcahe' for 'Deepcache' in Sec. III-B; a final proofread is recommended.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"This is a genuine algorithm-hardware co-design paper with a synthesized FPGA, sensible ablations, and an internally consistent roofline analysis. The main risk is the unsupported transferability of the phase-aware schedule and the in-sample quality claim; both are fixable with additional experiments. If the authors can provide held-out quality evaluation, error bars, and sensitivity analysis for D* and Tsketch, I would support publication. The projected speedup in Fig. 20 should also be labeled clearly as a projection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a genuine co-design effort with a credible new algorithm-level structure, but the quality-preservation claim is only demonstrated in-sample. If you read it expecting a working FPGA accelerator plus a plausible phase-aware sampling scheme, you get that. If you read it expecting a robust demonstration of 'no quality compromise', that part is weaker than advertised.\n\nWhat's actually new: the two-phase scheduling idea—run the full U-Net early, then keep only the top two blocks with activations reused from the nearest complete step—plus a calibration framework that picks the schedule {Tsketch, Tcomplete, Tsparse, Lsketch, Lrefine} under a user quality constraint. The paper openly credits DeepCache and CacheMe for layer skipping and T-Stitch for step-dependent network sizes, which is honest. The hardware side is also solid: actual VCU118 synthesis, per-optimization ablations, a roofline model, and the measured speedup is about 95% of the theoretical for PAS-25/4. The address-centric dataflow for convolution on a plain systolic array looks like a genuinely useful trick, and the 2-stage streaming for softmax/layernorm is well-motivated.\n\nSoft spots, in proportion. The 'without compromising image quality' claim is the load-bearing one, and it rests on single-run metrics on a 5,000-prompt MS-COCO validation set. The hyperparameters are tuned on 5% of that same set, so the headline numbers are effectively in-sample. CLIP scores decline monotonically as you compress, which the paper acknowledges; the FID improvements are explained only post hoc. There is no variance or sensitivity analysis for D*, even though the paper asserts it is robust to prompt randomness. The SOTA accelerator comparison relies on self-built simulators, and the fp16 hardware path is not end-to-end validated for image quality. None of these sink the engineering story—the MAC reductions and the hardware numbers can stand on their own—but the quality claim needs a held-out evaluation and error bars before I'd trust the transfer.\n\nBottom line: this deserves a serious referee. It's a real, detailed, mostly honest co-design paper. The revision should add a proper held-out quality evaluation, report variance across seeds, and ideally release code/artifacts.","headline":"A credible co-design paper with a genuinely useful phase-aware sampling scheme, but the 'no quality compromise' headline is only proven in-sample; deserves peer review with revisions.","tokens_in":28960,"tokens_out":1900,"would_cite":true,"duration_ms":20928,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Phase-aware sampling lets Stable Diffusion skip most of the U-Net in the later denoising phase, cutting compute up to 5.7x without retraining while a co-designed FPGA raises energy efficiency up to 6.0x over a V100 GPU.","keywords":["Stable Diffusion","diffusion model acceleration","phase-aware sampling","FPGA accelerator","systolic array dataflow","algorithm-hardware co-design","U-Net block skipping","text-to-image generation"],"falsifier":"Recompute the phase boundary of Eq. (2) on many random 5% subsets of the MS-COCO prompt set for StableDiff v1.4: if the argmin $D^*$ varies by more than a few timesteps across subsets, or if the top-two outlier blocks differ for v2.1 or XL, the fixed PAS schedule is an artifact of one calibration run. A second decisive experiment is to run the most aggressive tested schedule on a prompt set dominated by fine-grained text and measure CLIP and FID; a sharp drop would show that the refinement phase needs more than the top two blocks when local texture carries semantic content.","tokens_in":27631,"feed_emoji":"⚡","tokens_out":16698,"duration_ms":181662,"temperature":0.7,"pith_summary":"SD-Acc claims that Stable Diffusion's 50-step denoising process splits into a sketching phase, which fixes the image's global structure, and a refinement phase, which adjusts local texture. Analyzing how much each U-Net block's activations change between adjacent timesteps, the paper argues that in the refinement phase only the first two U-Net blocks need to compute, reusing activations from the nearest fully computed timestep. This phase-aware sampling (PAS) schedule needs no retraining and, across StableDiff v1.4, v2.1, and XL, removes a factor of 2.4x to 5.7x of multiply-accumulate work while keeping image-quality metrics close to the original. The paper's hardware half is a co-designed FPGA that maps both convolution and attention onto a simple systolic array, hides softmax/layernorm/GELU latency with two-stage streaming, and adapts data reuse per layer, which together give 2.7x to 6.0x lower energy per generated image than an Nvidia V100 GPU. If correct, SD-Acc offers a retraining-free path to cheaper, faster deployment of text-to-image models on energy-limited devices.","feed_headline":"Two-phase sampling cuts Stable Diffusion compute 5.7x and energy 6x","feed_subtitle":"Without retraining, only the top two U-Net blocks run in the later phase; a co-designed FPGA reaps the savings.","key_machinery":"The load-bearing object is the normalized shift score $\\bar{S}_t$ derived from Eq. (1), averaged over calibration prompts and min-max scaled per U-Net block; it is the evidence that high-level features stabilize while the top blocks remain outliers, and Eq. (2) converts it into the phase boundary $D^*$. Phase-aware sampling is the schedule built on that boundary: the full U-Net runs at the start and periodically in the sketching phase, and in the refinement phase only the first $L_{\\mathrm{refine}} = 2$ blocks run, fed by activations reused from the nearest complete timestep. On the hardware side, the carry mechanism is the address-centric dataflow, which decomposes each convolution into $1\\times1$ kernels that behave as matrix multiplications and routes partial sums to output addresses through fixed offsets such as $l \\to l+1$, removing im2col's irregular memory traffic and letting one systolic array handle convolution and attention together. The two-stage streaming unit then hides nonlinear operations by splitting softmax and layernorm into numeric-characteristic acquisition and normalization and scheduling those stages inside the systolic array's streaming writes and reads.","core_discovery":"Phase-aware sampling is the paper's answer to the observation that Stable Diffusion's denoising is not uniformly redundant. The authors define a shift score $S_i^t = \\frac{\\|A_i^t - A_i^{t-1}\\|_2}{\\|A_i^{t-1}\\|_2}$ on the main-branch activations of each upsampling block, min-max normalize it per block, average over 100 PartiPrompts prompts using the 50-timestep scheduler, and find that most blocks settle into low variation after roughly the first half of denoising while the top blocks stay active. They read this as a sketching phase followed by a refinement phase, locate the boundary $D^*$ with a k-means-style sweep, and then execute only the first $L_{\\mathrm{sketch}} = L_{\\mathrm{refine}} = 2$ blocks during most of the refinement timesteps, using activations from the nearest complete timestep as the entry point. Across StableDiff v1.4, v2.1, and XL this removes a factor of 2.4x to 5.7x of the MAC work at nearly unchanged CLIP score and no retraining. The hardware half of the paper shows that the same saving can be realized: an address-centric dataflow maps convolution onto a plain systolic array as decomposed 1x1 matrix products, the two-stage streaming unit hides softmax/layernorm/GELU latency in the data stream, and adaptive reuse and fusion cut off-chip traffic, yielding 2.7x to 6.0x lower energy per image than an Nvidia V100 GPU.","pith_inferences":["The shift-score analysis is reported for one scheduler and one calibration distribution; if the same two-phase boundary appears under faster solvers such as DDIM or DPM-Solver, PAS could be combined with step distillation to compound the savings, but the paper does not test this.","The fixed choice of the top two blocks as the only refinement-phase survivors is a statistical choice; a stress test with text-heavy prompts, where local texture carries semantic information, would reveal whether the phase division is a general property of denoising or an artifact of the MS-COCO distribution.","Because the FPGA is estimated to consume over five times the power of an ASIC at the same node, the reported energy savings are a lower bound for a production ASIC implementation; the paper notes this but does not quantify it.","The address-centric convolution decomposition is a generic way to make one systolic array handle mixed CNN-Transformer workloads, so the hardware mechanism could transfer to other U-Net-like backbones such as video diffusion models, though the paper evaluates only text-to-image generation."],"forward_implications":["Across StableDiff v1.4, v2.1, and XL, the PAS schedules of Table II remove a factor of 2.4x to 5.7x of multiply-accumulate work while keeping CLIP scores within about 0.005 of the original, with FID slightly improved on v1.4 and v2.1 and slightly worse on XL.","Because the schedule hyperparameters are searched against user-specified quality and performance targets, the same framework can tune StableDiff for a desired speed/quality trade-off instead of applying one fixed pruning recipe.","The co-designed FPGA converts the algorithm savings into 2.7x to 6.0x lower energy per image than an Nvidia V100 GPU and 14.7x to 44.9x lower than the tested CPUs.","The address-centric dataflow lets one static systolic array execute both convolutions and attention with high PE utilization, and the two-stage streaming unit hides most nonlinear-operator latency, so the hardware gains are largely orthogonal to the phase-aware sampling gains.","Because the retained top blocks run at the largest resolutions, the pruned workload has higher operational intensity, moving StableDiff toward the compute-bound region where the systolic array is most effective."],"supporting_citations":[{"why":"Provides the Stable Diffusion architecture (text encoder, U-Net, VAE) that phase-aware sampling prunes and the original image-quality baseline.","marker":"[49]"},{"why":"Supplies the 50-timestep scheduler used in the calibration study that produces the shift-score curves behind phase division.","marker":"[21]"},{"why":"Defines the PNDM scheduler used for the 50-step evaluations and the PAS quality/performance numbers in Table II.","marker":"[33]"},{"why":"Supplies the PartiPrompts prompts used as the calibration set to estimate the phase boundary D* and outlier blocks.","marker":"[63]"},{"why":"Supplies the MS-COCO validation captions that form the target prompt distribution for the 5000-image quality evaluation.","marker":"[32]"},{"why":"The block-skipping baseline (DeepCache) that PAS extends with phase awareness; the paper's Table III compares against it.","marker":"[38]"},{"why":"The distillation-based compressed StableDiff baseline (BK-SDM) that PAS must match or beat without retraining.","marker":"[22]"},{"why":"Cambricon-D, a diffusion-model accelerator baseline used for the throughput-scaled speedup comparison.","marker":"[25]"},{"why":"The online-normalizer softmax method that the tile-decoupled two-stage softmax scheduling builds on.","marker":"[40]"},{"why":"The im2col-style systolic-array baseline that the address-centric dataflow ablation is measured against.","marker":"[11]"}],"fun_headline_variants":["Phase-aware sampling reduces SD MACs 5.7x, energy 6x","SD-Acc: 5.7x less compute, 6x less energy for Stable Diffusion","Stable Diffusion compute cut 5.7x with phase-aware sampling","Co-design cuts Stable Diffusion compute 5.7x and energy 6x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two-phase pattern seen in activation changes on 100 calibration prompts with one scheduler is stable enough that the phase boundary and the choice to keep only the top two U-Net blocks transfer to the target prompts and to all three StableDiff versions.","fun_headline_variants_meta":{"raw":{"variants":["Phase-aware sampling reduces SD MACs 5.7x, energy 6x","SD-Acc: 5.7x less compute, 6x less energy for Stable Diffusion","Stable Diffusion compute cut 5.7x with phase-aware sampling","Co-design cuts Stable Diffusion compute 5.7x and energy 6x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000375,"raw_usage":{"total_tokens":2121,"prompt_tokens":1188,"completion_tokens":933,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":804,"completion_tokens_details":{"reasoning_tokens":843}},"tokens_in":804,"tokens_out":933,"duration_ms":10338,"temperature":1.0,"reasoning_tokens":843,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:58:14.802256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the phase boundary of Eq. (2) on many random 5% subsets of the MS-COCO prompt set for StableDiff v1.4: if the argmin $D^*$ varies by more than a few timesteps across subsets, or if the top-two outlier blocks differ for v2.1 or XL, the fixed PAS schedule is an artifact of one calibration run. A second decisive experiment is to run the most aggressive tested schedule on a prompt set dominated by fine-grained text and measure CLIP and FID; a sharp drop would show that the refinement phase needs more than the top two blocks when local texture carries semantic content.","supporting_citations":[{"cited_title":"High- resolution image synthesis with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"Provides the Stable Diffusion architecture (text encoder, U-Net, VAE) that phase-aware sampling prunes and the original image-quality baseline."},{"cited_title":"Deepcache: Accelerating diffusion models for free,","cited_arxiv_id":null,"evidence_quote":"The block-skipping baseline (DeepCache) that PAS extends with phase awareness; the paper's Table III compares against it."},{"cited_title":"Bk-sdm: Archi- tecturally compressed stable diffusion for efficient text-to-image gen- eration,","cited_arxiv_id":null,"evidence_quote":"The distillation-based compressed StableDiff baseline (BK-SDM) that PAS must match or beat without retraining."},{"cited_title":"Cambricon-D: Full-network differential acceleration for diffusion models,","cited_arxiv_id":null,"evidence_quote":"Cambricon-D, a diffusion-model accelerator baseline used for the throughput-scaled speedup comparison."},{"cited_title":"Gemmini: Enabling systematic deep- learning architecture evaluation via full-stack integration,","cited_arxiv_id":null,"evidence_quote":"The im2col-style systolic-array baseline that the address-centric dataflow ablation is measured against."}],"review_version":1}