{"id":"6de0a17c-01ff-4789-b21e-15dfc5c6140b","arxiv_id":"2412.16822","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DiffCR learns per-layer and per-timestep token compression ratios for diffusion transformers, improving FID at similar latency relative to uniform token pruning.","lead":"Diffusion transformers can now learn which image patches to skip at every layer and every denoising step, with compression amounts learned separately per layer and timestep. Tests on text-to-image and inpainting show lower FID than prior compression methods at similar speed and memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is the differentiable-ratio surrogate in Sec. 3.3: training optimizes a linear blend of two discrete MoD branches, while deployment snaps to the nearest bin, and the paper never validates that the surrogate's loss landscape transfers to the hard selection.","rationale":"The reader's weakest assumption is exactly the concern I would flag after rereading Secs. 3.3-3.4 and Fig. 1b. The paper states that training uses two MoD branches and a weighted combination, while inference selects the nearest bin, but no experiment distinguishes the interpolated training objective from the hard deployed selection. I found no internal contradiction in the method; the issue is an unvalidated transfer between a differentiable proxy and a discrete decision. If the proxy is faithful, the central claim stands; if it is not, the learned ratios could be miscalibrated even though the routers remain useful. The trajectory analyses show convergence of average ratios, not quality at deployment, so they do not close this gap. Missing error bars and missing code make the gap harder to resolve but are secondary to the surrogate question. I therefore keep the conditional verdict: the paper needs a direct validation of surrogate-to-deployment transfer before the claimed quality-efficiency trade-offs are accepted as established.","tokens_in":18478,"tokens_out":4400,"duration_ms":39741,"concrete_test":"On the trained DiffCR-LT checkpoints (T2I and inpainting), regenerate the same 10K evaluation images under two protocols: (i) the deployed nearest-bin ratios used in Tabs. 1-2, and (ii) the fractional interpolated ratios used during training, with the same learned scalar values kept fixed. Compare FID and latency on identical prompts. If protocol (ii) is not clearly better than protocol (i), or if protocol (i) cannot reproduce the reported FID at the reported latency, then the surrogate is not a faithful proxy for deployment. A complementary check is to fine-tune from scratch with ratios clamped to the nearest bins and see whether the same FID/latency trade-off still emerges.","verdict_should_be":"UNCHANGED","load_bearing_attack":"During training (Sec. 3.3, Fig. 1b), a layer with learned ratio r computes a convex combination of two MoD passes: one at the lower bin and one at the upper bin, with weights set by proximity to the 10% discrete bins. This is not a MoD pass at ratio r; it is a blend of two passes with different top-k token sets. At inference the ratio snaps to the nearest bin, so the deployed computation is MoD(nearest_bin), not the blended output seen by the loss. The diffusion loss is therefore minimized against a blended output that can, in effect, borrow information from two different token subsets, while the deployed model sees only one. The only transfer evidence in the paper is the ratio trajectory in Figs. 4 and 8, which shows that the average learned ratio converges to the target; it does not show that FID at the snapped ratios matches the loss surface encountered during training. If the true quality-versus-ratio surface is nonconvex, or if blending is systematically overoptimistic, the learned per-layer and per-timestep ratios can be miscalibrated even when the routers are well trained. In that case the headline gains of DiffCR-LT over uniform MoD (10.68 vs. 22.78 FID on T2I; 13.42 vs. 18.34 on inpainting) could be attributable to the fine-tuned routers rather than to the differentiable-ratio mechanism. This is a correctness risk to the central claim, not merely a missing baseline, because the central claim is precisely that the learned ratios deliver the improved trade-off.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiffCR, a method to make mixture-of-depth (MoD) token routing in diffusion transformers differentiable with respect to compression ratios. Alongside a token-level router, DiffCR learns per-layer and per-timestep compression ratios by interpolating between two discrete MoD branches during training and snapping to the nearest 10% bin at inference. The method is evaluated on text-to-image with PixArt-Σ and on inpainting with Lazy Diffusion, reporting lower FID than ToMe, AT-EDM, and uniform MoD at roughly comparable latency and memory. The authors also analyze learned ratio patterns, report human-preference scores, and include supplementary comparisons with caching baselines.","tokens_in":18904,"tokens_out":6491,"duration_ms":53622,"significance":"If the results hold, DiffCR would be a practically relevant extension of MoD to vision, and the idea of learning compression ratios via a differentiable surrogate is attractive because it avoids per-layer manual tuning. The paper is also commendable for the breadth of its evaluation: two tasks, comparisons against training-free and learned baselines, router visualizations, ratio-trajectory analysis, human-preference scores, and supplementary caching-baseline comparisons. However, the central mechanism is currently under-validated: the paper does not show that optimizing the blended training surrogate transfers to the hard nearest-bin selection used at deployment, and the reported efficiency gains over uniform MoD are small and lack error bars. These gaps are fixable and do not invalidate the potential of the approach, but they need to be addressed before the trade-off claim can be accepted.","major_comments":[{"comment":"The load-bearing step is the claim that the differentiable-ratio surrogate is a faithful proxy for the hard MoD selection. During training, a layer with learned ratio r computes a convex combination of the outputs at the lower and upper 10% bins; at inference the ratio snaps to the nearest bin. The paper provides no experiment showing that the blended objective's optimum matches the FID-optimal discrete bin assignment, nor that the loss landscape is convex or smooth enough for the transfer to hold. The trajectory plots (Figs. 4 and 8) only show convergence of the average ratio to the target, not that per-layer or per-timestep ratios are well-calibrated on the deployed path. I request a validation experiment, e.g., after training, evaluate FID for every discrete bin assignment (or a substantial random subset), and compare the surrogate's predicted quality with actual FID at the snapped ratios; this would test whether the learned ratios are optimal rather than merely MSE-converged.","section":"Sec. 3.3 and Fig. 1(b)"},{"comment":"The efficiency advantage of DiffCR-LT over uniform MoD is not actually demonstrated. For T2I, DiffCR-LT has 179.71 s latency and 1.664 GB memory versus 178.89 s and 1.659 GB for MoD; for inpainting at BS=128, it has 13.89 s and 34.88 GB versus 13.02 s and 32.58 GB for MoD. The entire case for DiffCR-LT therefore rests on the FID differences (10.68 vs 22.78 and 13.42 vs 18.34), but no error bars, number of seeds, or statistical significance tests are reported. Since fine-tuning alone changes the T2I FID from 151.0 to 11.93 (Table 1), the reader cannot exclude that the FID gap is partly due to training noise or to a different effective fine-tuning budget. Please report mean and std over at least three seeds for the key comparisons, and state the fine-tuning budget (steps/epochs and FLOPs) for each method.","section":"Tables 1 and 2"},{"comment":"The ablation DiffCR-L versus MoD changes two things at once: the per-layer/timestep ratio mechanism and the final fine-tuned weights. To attribute the FID gain (10.5 on T2I, 4.81 on inpainting) to the differentiable-ratio mechanism, the paper should include a control that isolates the ratio mechanism—for example, a uniform-ratio model fine-tuned for the same number of steps with the same router architecture, or a DiffCR variant whose routers are frozen while only the ratios are learned. As written, the improvement over the MoD baseline could in principle come from the extra training dynamics or from the different ratio schedule rather than from the surrogate itself.","section":"Sec. 4.3 and Tables 1-2"}],"minor_comments":[{"comment":"The sentence 'DiffCR achieves a higher human preference score of 4.685/0.847 compared to previous compression methods, ToMe and vanilla MoD' is ambiguous; the numbers are the differences relative to ToMe and MoD, respectively, and should be stated as such.","section":"Sec. 4.4"},{"comment":"The HPS tables are not discussed in detail; please clarify how HPSv2 scores are computed for the inpainting task, since HPSv2 was originally designed for text-to-image evaluation.","section":"Tables 3 and 5"},{"comment":"The figure would benefit from stating the model and resolution used and whether the x-axis is the actual compression ratio achieved or the target ratio; the current axis labels 'Compression Ratios (%)' are too terse.","section":"Fig. 3"},{"comment":"The phrase 'learning a compression ratio from a zero initialization' is imprecise because the scalar is initialized so that the resulting ratio is 0%, while the MSE target is nonzero; please clarify the initialization and the parameterization.","section":"Sec. 3.3"},{"comment":"The internal 220M-image dataset is described in one sentence; please provide more detail on filtering, mask generation, and availability, since the inpainting results cannot be reproduced otherwise.","section":"Sec. 4.1"},{"comment":"The comparison with caching baselines reports 'approximately 25% latency savings' but does not specify how the savings are matched across methods; please state the exact latency and FID for each method.","section":"Supplementary Sec. H"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a vision or efficient-inference venue, but the novelty is incremental relative to DiffRate [5] plus MoD [33]; the main contribution is the application to DiTs and the timestep dimension. The most serious concern is the surrogate validity, which the authors can address with a discrete search or a calibration experiment. I would be willing to see a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on DiffCR. The genuinely new bit is the differentiable per-layer and per-timestep compression ratios on top of MoD token routing. That combination hasn't been done before, and the paper shows it gives a real quality boost over uniform MoD on both inpainting and text-to-image: going from 22.78 to 10.68 FID on PixArt-Sigma at roughly the same latency, and from 18.34 to 13.42 on LazyDiffusion. The learned ratio pattern—more compression at noisy timesteps, less at clear ones—is plausible and consistent with earlier findings. The \"by-product models\" angle, where the MSE coefficient gives you a family of models at different compression ratios, is a nice bonus.\n\nThe evaluation is fairly extensive: 10K images for FID, latency/memory on A100, HPS, ablations that separate the router, layer-wise, and timestep-wise contributions. The ablation supports the claim that each component helps.\n\nThe soft spots are real but not disqualifying. First, the differentiable surrogate in Sec. 3.3 is the load-bearing piece: training blends two MoD branches, inference snaps to the nearest bin, and the paper never directly validates that the proxy tracks the true quality-ratio surface. The stress-test's worry that the headline gains might come from the fine-tuned routers rather than the ratios is not supported by the data—the MoD baseline is also fine-tuned—but the surrogate's transfer is nonetheless an assumption that a referee should probe. A simple experiment, like fixing the learned ratios and retraining the routers without the blend, would settle it. Second, there are no error bars; the FID differences are large, so this is minor. Third, the inpainting data is internal, and key cited baselines like DiffRate and SpeeD are not benchmarked; I'd like to see those numbers. Fourth, DiffCR-LT is consistently a bit slower and more memory-hungry than uniform MoD, so the win is quality-at-same-cost, not strictly speed. Finally, the \"first to apply MoD to vision\" claim is unsupported—no citation for why that's true.\n\nOverall, the central claim—that learned layer/timestep ratios improve the quality-efficiency frontier—holds up. The paper is a solid engineering contribution for anyone working on efficient diffusion transformers. It deserves a serious peer review; a good referee would push for the surrogate-validation experiment and the missing baselines.\n\nMy recommendation: send it to review, and ask for those additions.","headline":"DiffCR gives a real quality-efficiency win with a new differentiable per-layer/timestep compression idea, but the unvalidated interpolation surrogate is the main soft spot.","tokens_in":19443,"tokens_out":5123,"would_cite":true,"duration_ms":42356,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DiffCR learns where to skip compute in diffusion transformers.","keywords":["diffusion transformers","dynamic inference","token routing","differentiable compression ratios","mixture-of-depths","text-to-image generation","image inpainting","model efficiency"],"falsifier":"Run the same fine-tuning recipe with the interpolation replaced by a straight-through estimator that snaps each learned ratio to the nearest discrete bin in the forward pass while keeping gradients flowing through the scalar, then compare FID and latency on the same 10K-image evaluation set; if the straight-through model matches or beats DiffCR, the interpolation proxy is not what delivers the gains, and if it is much worse, the proxy is carrying the method.","tokens_in":18314,"feed_emoji":"⚡","tokens_out":11656,"duration_ms":93032,"temperature":0.7,"pith_summary":"DiffCR is a way to make diffusion transformers spend computation only where it matters during image generation. Instead of applying one global token-pruning ratio to every layer and every denoising step, it lets each layer and each timestep region learn its own ratio, with a lightweight router per layer deciding which tokens skip that layer's attention and MLP entirely. The ratios become trainable through a differentiable surrogate: during training the output at a learned continuous ratio is a weighted blend of the outputs at the two nearest discrete compression bins, and at inference the ratio snaps to the nearest bin. On text-to-image and inpainting benchmarks the paper reports lower FID (a standard image-quality score) at comparable or lower latency than uniform token routing, token merging, and attention-based pruning baselines. If the claim holds, it gives a practical way to deploy expensive diffusion transformers with less computation and without hand-tuning per-layer or per-timestep compression schedules.","feed_headline":"Diffusion transformers can learn their own token-skip schedule","feed_subtitle":"Each layer and denoising step picks its own compression ratio, beating uniform pruning at equal speed.","key_machinery":"The load-bearing mechanism is the differentiable compression-ratio surrogate built on top of mixture-of-depths (MoD) token routing. MoD is a routing scheme in which each layer's router scores tokens and only the top-k tokens are processed by the layer. DiffCR gives every layer a single learnable scalar compression ratio and, during training, queries the two nearest 10% ratio bins, runs the MoD layer at both ratios, and linearly combines the two outputs with weights proportional to the scalar's distance to each bin; a mean-squared-error loss drives the average ratio to a target. At inference the scalar snaps to the nearest bin, so no extra branches run. This mechanism is what makes the compression ratio differentiable, allowing routers, ratios, and model weights to be fine-tuned together.","core_discovery":"The central claim is that the right amount of token compression in a diffusion transformer is not a global hyperparameter but something the model can learn jointly with its weights. DiffCR adds to each DiT layer a router that scores every token; only the top-k tokens run through the layer, and the rest bypass it. A continuous scalar per layer (and per timestep region) controls k through the differentiable surrogate described above, and a mean-squared-error loss pulls the batch-averaged ratios toward a target. On PixArt-Sigma for text-to-image, DiffCR-LT achieves FID 10.68 at 179.71 seconds latency versus FID 22.78 at 178.89 seconds for a uniform-ratio MoD baseline; on Lazy Diffusion for inpainting it achieves FID 13.42 at 13.89 seconds versus FID 18.34 at 13.02 seconds for the same baseline. The learned pattern concentrates compression in redundant middle layers and noisier early timesteps, while later layers and clearer timesteps stay mostly uncompressed.","pith_inferences":["The same differentiable-ratio mechanism should transfer to other hard discrete choices in dynamic networks, such as selecting attention heads or channels, whenever a selection can be relaxed into a weighted blend of two neighboring options.","The learned noise-level pattern suggests that a simple hand-designed schedule that reduces compression as denoising proceeds could capture much of the gain; DiffCR's distinctive contribution would then be discovering that schedule automatically, layer by layer, rather than specifying it.","The routers' importance maps resemble attention maps, which suggests the approach could extend to other content-adaptive generation tasks such as video or super-resolution, where token redundancy varies spatially and temporally."],"forward_implications":["At a fixed latency budget, DiffCR should improve FID over uniform token compression because it moves pruning away from critical layers and clear-image timesteps.","DiffCR is compatible with orthogonal accelerations such as layer caching and few-step distillation, so its savings should stack with those methods.","One fine-tuning run produces intermediate checkpoints at a range of average compression ratios, yielding a family of deployable models along a quality-latency frontier.","Because the learned pattern assigns higher compression to noisier timesteps, the efficiency gain should grow as the number of sampling steps increases."],"supporting_citations":[{"why":"Supplies the mixture-of-depths token routing scheme, with top-k selection and router gradient rescaling, that DiffCR builds on.","marker":"[33]"},{"why":"Provides the Lazy Diffusion host model and inpainting setup that DiffCR is fine-tuned on and evaluated against.","marker":"[28]"},{"why":"Provides the PixArt-Sigma host model, 28-layer DiT structure, and 20-step sampling schedule for text-to-image experiments.","marker":"[4]"},{"why":"Token merging baseline whose compression-quality trade-off DiffCR is compared with.","marker":"[2]"},{"why":"Training-free attention-based pruning baseline whose compression-quality trade-off DiffCR is compared with.","marker":"[45]"},{"why":"Previous differentiable compression-rate method with discrete proxies whose design DiffCR replaces with a single continuous scalar per layer.","marker":"[5]"}],"fun_headline_variants":["DiTs learn where to skip tokens per layer and timestep","Learned token compression for faster diffusion transformers","Adaptive token routing cuts DiT compute without quality loss","Diffusion models learn their own per-step token budgets","Smarter token skipping: DiTs adapt compression per layer and step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that, during training, the weighted average of the two nearest discrete compression-ratio outputs faithfully predicts what the model would produce at the actual learned ratio, so that optimizing this blend carries over to the hard nearest-bin choice used at inference.","fun_headline_variants_meta":{"raw":{"variants":["DiTs learn where to skip tokens per layer and timestep","Learned token compression for faster diffusion transformers","Adaptive token routing cuts DiT compute without quality loss","Diffusion models learn their own per-step token budgets","Smarter token skipping: DiTs adapt compression per layer and step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000652,"raw_usage":{"total_tokens":3052,"prompt_tokens":1069,"completion_tokens":1983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":1902}},"tokens_in":685,"tokens_out":1983,"duration_ms":11781,"temperature":1.0,"reasoning_tokens":1902,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:14:53.504164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuning recipe with the interpolation replaced by a straight-through estimator that snaps each learned ratio to the nearest discrete bin in the forward pass while keeping gradients flowing through the scalar, then compare FID and latency on the same 10K-image evaluation set; if the straight-through model matches or beats DiffCR, the interpolation proxy is not what delivers the gains, and if it is much worse, the proxy is carrying the method.","supporting_citations":[{"cited_title":"Attention-driven training-free efficiency enhancement of diffusion models","cited_arxiv_id":null,"evidence_quote":"Training-free attention-based pruning baseline whose compression-quality trade-off DiffCR is compared with."},{"cited_title":"Diffrate: Differentiable compression rate for efficient vision transformers","cited_arxiv_id":null,"evidence_quote":"Previous differentiable compression-rate method with discrete proxies whose design DiffCR replaces with a single continuous scalar per layer."}],"review_version":1}