{"id":"e46de635-8b82-4831-b246-7e4b90fb515b","arxiv_id":"2504.14365","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A pruning method and a digital compute-in-memory accelerator that jointly support layer-wise flexible N:M sparsity, improving LLM perplexity and zero-shot accuracy over fixed N:M baselines while cutting simulated inference latency and energy.","lead":"FLOW picks per-layer N:M sparsity patterns for LLMs using outlier count and distribution, and FlexCiM is a digital compute-in-memory accelerator that executes those flexible patterns. If the reported simulations hold, the pairing offers up to 1.75x lower latency and better accuracy than fixed-pattern sparse baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 1.75x latency and 1.5x energy gains assume all weights fit in global SRAM; for 7B/70B LLMs at 8-bit this is unrealistic, so the hardware results are upper bounds until a DRAM-inclusive memory model is evaluated.","rationale":"The reader already flagged the same SRAM assumption, and I agree that it is the most load-bearing concern. I considered the unvalidated 'optimal' ILP claim in Equation 2, but the decisive issue for the paper's central co-design claim is the memory assumption, because it is explicitly confessed in Section VI-A, it applies to exactly the models in the headline hardware figures, and the decode stage is memory-bound, so off-chip traffic can dominate and erase the reported margins. A DRAM-inclusive rerun is a single, decisive experiment. The algorithm's heuristic is concerning but the paper's practical contribution includes a working accelerator; if the hardware gains vanish, the co-design story collapses. The verdict remains conditional: the paper should be accepted only if the reported accelerator advantages are re-reported with a DRAM-inclusive memory model or the scope is explicitly narrowed to models that fit entirely on chip.","tokens_in":15890,"tokens_out":5773,"duration_ms":52979,"concrete_test":"Recompute the Figure 1 energy and Figure 7 latency numbers for LLaMA2-7B and LLaMA3-70B with an explicit DRAM tier, e.g., HBM3 at 512 GB/s and 4 pJ/bit, streaming weight tiles that do not fit in the L2 SRAM. If the FlexCiM-vs-baseline ratios remain above 1.3x, the caveat is minor; if they fall below 1.1x, the central hardware claim holds only for sub-2GB models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The hardware half of the central claim rests on the Section VI-A assumption: \"We assume all model parameters fit within the global SRAM for simplicity.\" At 8-bit weights, LLaMA2-7B alone needs roughly 7 GB and LLaMA3-70B roughly 70 GB of on-chip SRAM; no realistic DCiM system has this, and Figure 1/7 report normalized energy/latency for exactly these models. Decode-stage LLM inference is memory-bound, so once weights must stream from DRAM, the energy and latency differences between FlexCiM and VEGETA/dense arrays may be dominated by off-chip access. The statement in Section VI-C that off-chip access \"will be approximately similar for DCiM or digital accelerators\" does not fix this: equal DRAM cost can still mask the claimed 1.5x/1.75x advantages, and if all designs are DRAM-bandwidth limited, the latency gap can collapse. Thus the headline hardware numbers are valid only for entirely on-chip models, which are not the evaluated 7B/70B class.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FLOW is a post-training pruning method that assigns layer-wise N:M sparsity patterns to LLM weights based on both the fraction and the spatial distribution of outliers, formulated as an integer linear program (Eq. 2). FlexCiM is a digital compute-in-memory (DCiM) macro that supports multiple N:M patterns by partitioning the macro into sub-macros and adding distribution and merging units. The paper evaluates FLOW on transformer LLMs, SSMs, and VLMs, reporting perplexity and zero-shot accuracy gains over SparseGPT, Wanda, and OWL, and evaluates FlexCiM via Verilog RTL synthesis and a cycle-accurate simulator, reporting up to 1.75x lower latency and 1.5x lower energy than VEGETA and SDP baselines. The two central caveats are that the hardware claims assume all model weights fit in global SRAM, which is unrealistic for the evaluated 7B/70B class, and the 'optimal' N:M selection is asserted from an ILP with hand-picked hyperparameters and no oracle comparison.","tokens_in":16183,"tokens_out":5447,"duration_ms":47305,"significance":"If the results hold, the paper makes a useful contribution: it identifies outlier distribution as a signal for choosing M per layer, provides a concrete ILP-based allocation, and demonstrates a low-overhead hardware mechanism for flexible N:M sparsity in DCiM. Strengths include machine-checked RTL synthesis, a cycle-accurate simulator, public code release, and evaluation across a broad set of models including SSMs and VLMs. The perplexity improvements over OWL at 60% sparsity and the up-to-36% zero-shot accuracy gains are substantial and externally evaluated on WikiText2 and standard tasks, so the algorithm-side results are not circular. However, the significance is tempered by the on-chip SRAM assumption in the hardware evaluation, which makes the 1.75x/1.5x claims upper bounds for the models actually tested, and by the lack of an oracle baseline to validate the 'optimal' selection language.","major_comments":[{"comment":"Sections VI-A and VI-C: The energy and latency claims (Figures 1 and 7) are computed under the assumption that 'all model parameters fit within the global SRAM for simplicity' (Section VI-A). For the evaluated LLaMA2-7B and LLaMA3-70B at 8-bit weights, this requires roughly 7 GB and 70 GB of on-chip SRAM, which is not realistic for a DCiM system. Since LLM decode is memory-bound, once weights must be fetched from off-chip DRAM, the reported 1.5x energy and 1.75x latency advantages over dense and VEGETA baselines are dominated by DRAM access; the statement in Section VI-C that off-chip access 'will be approximately similar for DCiM or digital accelerators' does not compensate, because equal DRAM cost can mask the claimed benefits, and if all designs are DRAM-bandwidth limited the latency gap can collapse. The paper should either re-scope the hardware claims to models that fit on-chip or add a DRAM-inclusive memory model to the simulation.","section":"VI-A and VI-C"},{"comment":"Eq. (2), Section IV-B: The ILP objective encodes the paper's own hypothesis that N_l is proportional to outlier fraction O_l and M_l is proportional to (1 - ND_l), with hyperparameters (alpha, beta, k, h) = (1, 4, 8, 8) chosen empirically on the evaluated models. Calling the result 'optimal N:M' is unsupported because no comparison against an oracle search over the feasible set (N in {1,2,4,8}, M in {2,4,8}) is reported. Since this search space is only 12 patterns per layer, an exhaustive-oracle baseline is computationally cheap and would directly validate whether the ILP objective and its weights are justified. Without it, the central algorithmic claim of optimality is an assertion rather than a demonstrated result.","section":"IV-B, Eq. (2)"},{"comment":"Section IV-A: The outlier threshold tau is stated as 'we empirically identify tau = 3 or tau = 5 provide the best results,' but no ablation is shown. Because tau directly determines O_l and ND_l, which drive the entire FLOW allocation in Eq. (2), the sensitivity of final perplexity and zero-shot accuracy to tau should be reported for at least one model. This is load-bearing for the algorithm's central claim: if results change significantly with tau, the method's robustness and the 'empirically identify' statement need stronger support.","section":"IV-A"}],"minor_comments":[{"comment":"The latency improvement number is inconsistent: the abstract and conclusion say 'up to 1.75x lower inference latency,' while Section VI-C reports 'up to 1.72x' for the same results. Please harmonize these numbers.","section":"Abstract, VI-C, Conclusion"},{"comment":"The prose before Eq. (1) says D_l measures 'the summation of the pairwise distance between the outliers averaged over the number of outliers,' but the equation shows an average over nC2 outlier pairs. Please make the description match the equation.","section":"III-A, Eq. (1)"},{"comment":"The block size of 128x128 used to compute D_l^b is not justified; since it is a hyperparameter that affects ND_l, adding a one-sentence rationale or a small sensitivity check would improve reproducibility.","section":"IV-B"},{"comment":"Figure 5 is very dense and difficult to parse; the walkthrough in Section V-D would benefit from larger fonts, clearer sub-macro labels, and a more explicit mapping between the metadata bits and the multiplexer select lines.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid algorithm-hardware co-design effort with good artifacts (code, RTL synthesis, cycle-accurate simulator). The main risks are overclaiming 'optimality' without an oracle comparison and presenting 1.75x/1.5x hardware gains under an unrealistic on-chip SRAM assumption for the 7B/70B models evaluated. The algorithm-side results are convincing and externally benchmarked, so I recommend major revision rather than rejection; the authors can address the issues within the manuscript's scope by adding an oracle baseline and a DRAM-inclusive memory model or by re-scoping the hardware claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Things to know: this paper does something genuinely new. It jointly selects per-layer N and M for N:M pruning (FLOW), going beyond OWL's fixed-M mixed-N scheme, and it introduces FlexCiM, the first DCiM accelerator I know of that supports flexible N:M patterns via partitioned sub-macros and distribution/merging units. The perplexity and zero-shot results on transformers and SSMs are clean and consistently favor FLOW, often by a large margin at 60% sparsity. The ablation in Figure 6b shows the flexibility itself helps, not just the search. The hardware work is also real: Verilog RTL, synthesis at 28nm, a cycle-accurate simulator, ~6% area overhead, and sane comparisons against VEGETA and SDP. Code and RTL are promised, though I did not verify the repository contents.\n\nSoft spots, in proportion. Calling the ILP result \"optimal N:M\" is overreach. Equation 2 minimizes a hand-picked objective with (α,β,k,h)=(1,4,8,8) and no oracle or exhaustive-search sanity check. That is not fatal—the method works as a heuristic—but it should be described as one, and the hyperparameters deserve sensitivity analysis. Zero-shot results have no error bars; for an \"up to 36%\" claim that matters, though the ranking looks stable across tasks.\n\nThe bigger caveat is the one the stress-test note flags. The energy and latency numbers (1.5x, 1.75x) assume all model parameters fit in global SRAM (Section VI-A), for LLaMA2-7B and LLaMA3-70B. At 8-bit, 7B is roughly 7GB and 70B roughly 70GB; no realistic DCiM system has that on-chip. The claim that DRAM access is similar across designs does not salvage the headline numbers, because if all designs are DRAM-bandwidth-limited, the relative gaps can collapse. So Figures 1 and 7 should be read as upper bounds for an on-chip regime, not as end-to-end results for the models listed. The authors state the assumption explicitly, so this is a framing problem rather than a hidden one, but the abstract overstates the hardware gains.\n\nWho is this for? People working on LLM pruning or sparse accelerators. A serious architecture venue should send it to referees; it needs revision rather than rejection: rename \"optimal,\" add sensitivity analysis and error bars, and re-report hardware with a DRAM-inclusive model or clearly scope the claims to on-chip inference. I would engage with this work.","headline":"FLOW + FlexCiM is a credible algorithm-hardware co-design that extends N:M pruning to per-layer N and M and builds a low-overhead DCiM macro for it; the pruning results look solid, but the headline hardware numbers rest on an all-weights-on-chip assumption that does not hold for the evaluated 7B/70B models.","tokens_in":16722,"tokens_out":1891,"would_cite":true,"duration_ms":18432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tuning both $N$ and block size $M$ per layer, based on outlier density and spread, preserves more LLM accuracy at high sparsity than fixed or $N$-only $N{:}M$ patterns, while a partitioned digital compute-in-memory macro accelerates the…","keywords":["N:M sparsity","structured pruning","outlier-aware pruning","digital compute-in-memory","LLM inference acceleration","layer-wise sparsity","FLOW","FlexCiM"],"falsifier":"Run the paper's cycle-accurate simulator with the global SRAM assumption removed for a 70B-class 8-bit model, letting weights stream from DRAM, and compare FlexCiM's end-to-end energy and latency against the dense systolic-array baseline at iso-throughput; if the advantage drops below the claimed 1.5x energy / 1.75x latency, the end-to-end efficiency claim is refuted.","tokens_in":15682,"feed_emoji":"⚡","tokens_out":13556,"duration_ms":104938,"temperature":0.7,"pith_summary":"FLOW is a pruning-side method that assigns each layer of an LLM its own $N{:}M$ structured-sparsity pattern by looking at both the fraction of outlier weights and how spread out those outliers are. FlexCiM is a companion digital compute-in-memory accelerator that can execute any supported $N{:}M$ pattern without rebuilding the macro, by splitting a macro into sub-macros and adding distribution and merging units. The paper's central claim is that this paired flexibility is worth having: FLOW keeps perplexity closer to the dense model than fixed $N{:}M$ or $N$-only-varied baselines at 50-60% sparsity, and FlexCiM delivers up to 1.75x lower inference latency and 1.5x lower energy than existing sparse accelerators with only about 6% area overhead. If correct, it means one memory-compute macro can serve a whole family of sparsity patterns, so the choice of sparsity shape becomes a per-layer optimization problem rather than a hardware constraint.","feed_headline":"Flexible layer-wise sparsity cuts LLM inference cost by up to 1.75x","feed_subtitle":"Tuning both sparsity knobs per layer keeps accuracy; one compute-in-memory macro runs all patterns with ~6% area overhead","key_machinery":"The load-bearing mechanism is the combination of two per-layer outlier signals with a partitioned-macro dataflow. FLOW computes the outlier fraction $O_l$ from an importance score based on weight magnitude times input-activation L2 norm, thresholded at a multiple of the layer's standard deviation, and the outlier distribution $D_l$ as the mean pairwise L1 distance among outliers inside 128x128 blocks, normalized across layers. An integer linear program then assigns $N_l$ and $M_l$ so that $N$ tracks the outlier fraction and $M$ tracks one minus the normalized outlier distribution, subject to the average sparsity budget. On the hardware side, FlexCiM splits a 128x32x8 DCiM macro into $P=4$ sub-macros; a distribution unit of $P:1$ multiplexers feeds activations to sub-macros, CSC metadata selects which sub-macro receives which input, and a merging adder tree combines partial sums. This division of labor keeps large multiplexers out of the memory cells and confines per-cell logic to a 2:1 mux, which is why flexible $N{:}M$ costs only about 6% area.","core_discovery":"The central claim is that both dimensions of $N{:}M$ sparsity should vary by layer, not just $N$. A layer's tolerance to pruning is set by two separate properties of its outliers: their abundance (presence) and their pairwise distance (distribution). Clustered outliers call for a larger block $M$ so the pruning pass has room to keep them; sparse, low-count outliers call for smaller blocks and lower $N$. FLOW encodes this as an integer program with $N$ restricted to 1, 2, 4, or 8 and $M$ restricted to 2, 4, or 8, using the outlier fraction and a normalized outlier-distribution score to guide the assignment while matching a target average sparsity. On the hardware side, FlexCiM partitions one DCiM macro into four row-wise sub-macros and uses distribution and merging units so that a block of $M$ activations can be handled by one or several sub-macros according to $N$. The paper reports that FLOW improves zero-shot accuracy by up to 36% relative to prior pruning methods and improves perplexity by up to 18% over OWL-style $N$-only assignment, while FlexCiM reduces inference latency by up to 1.75x and energy by up to 1.5x versus the sparse accelerators it is compared with.","pith_inferences":["If a benchmark includes DRAM traffic for a 70B-class model, the end-to-end advantage over dense baselines would likely shrink below the headline 1.5x energy figure, because weight streaming adds a cost both designs pay.","The outlier-distribution score could serve as a general layer-difficulty signal for other compression choices, such as per-layer quantization bit width or mixture-of-experts routing; the paper does not test those uses.","Allowing non-power-of-two $M$ values would expand the representational freedom FLOW can select, at the price of wider multiplexers in the distribution unit; that trade-off is not explored.","A sensitivity study of the ILP weights would show whether the chosen $N{:}M$ patterns transfer across model families without retuning; the paper fixes these values once."],"forward_implications":["A FLOW-pruned model keeps WikiText2 perplexity closer to the dense baseline than fixed 4:8/3:8 patterns and than OWL's $N$-only assignment at the same target sparsity.","FlexCiM accelerates dense, 1:2, 1:4, 2:4, 1:8, 2:8, and 4:8 patterns in one macro, so layer-wise heterogeneous sparsity no longer forces a fixed hardware pattern.","The 1.75x latency and 1.5x energy gains come with flexible $N{:}M$ support at about 6% area overhead over a fixed 1:2 sparse DCiM.","FLOW also improves perplexity of Mamba-based state space models, so the method is not tied to transformers.","Under an unconstrained sparsity budget, FLOW achieves both higher average sparsity and lower perplexity than OWL on all tested models."],"supporting_citations":[{"why":"Baseline that assigns different $N$ values with fixed $M$ based on layer outlier ratios; FLOW generalizes this by also selecting $M$.","marker":"[38]"},{"why":"Flexible $N{:}M$ digital accelerator baseline and source of the CSC sparse storage format FlexCiM adapts.","marker":"[14]"},{"why":"Fixed 1:2 sparse DCiM macro that FlexCiM partitions and extends with distribution and merging units.","marker":"[35]"},{"why":"Two-level SRAM hierarchy and the all-parameters-in-global-SRAM assumption used to compute end-to-end energy and latency.","marker":"[32]"},{"why":"One-shot Hessian-based pruning baseline that FLOW is compared against on perplexity and accuracy.","marker":"[9]"},{"why":"Activation-aware pruning baseline that FLOW is compared against and whose importance score FLOW reuses for outlier detection.","marker":"[33]"}],"fun_headline_variants":["Layer-wise N:M sparsity: up to 1.75x faster LLM inference","Flexible sparsity boosts LLM accuracy up to 36%, cuts latency 1.75x","FLOW+FlexCiM: flexible N:M sparsity for efficient LLMs","Adaptive N:M pruning with digital CiM: 1.5x less energy, 1.75x faster","Outlier-aware N:M sparsity: LLM accuracy up 36%, latency down 1.75x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that all model weights fit inside the on-chip global SRAM, so the energy and latency comparisons exclude DRAM access; for a 70B-parameter model even at 8-bit precision this is optimistic, and any weight streaming from DRAM would erode the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Layer-wise N:M sparsity: up to 1.75x faster LLM inference","Flexible sparsity boosts LLM accuracy up to 36%, cuts latency 1.75x","FLOW+FlexCiM: flexible N:M sparsity for efficient LLMs","Adaptive N:M pruning with digital CiM: 1.5x less energy, 1.75x faster","Outlier-aware N:M sparsity: LLM accuracy up 36%, latency down 1.75x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000813,"raw_usage":{"total_tokens":3643,"prompt_tokens":1101,"completion_tokens":2542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":717,"completion_tokens_details":{"reasoning_tokens":2416}},"tokens_in":717,"tokens_out":2542,"duration_ms":15685,"temperature":1.0,"reasoning_tokens":2416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:50:54.279194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's cycle-accurate simulator with the global SRAM assumption removed for a 70B-class 8-bit model, letting weights stream from DRAM, and compare FlexCiM's end-to-end energy and latency against the dense systolic-array baseline at iso-throughput; if the advantage drops below the claimed 1.5x energy / 1.75x latency, the end-to-end efficiency claim is refuted.","supporting_citations":[{"cited_title":"Outlier weighed layerwise sparsity: A missing secret sauce for pruning llms to high sparsity,","cited_arxiv_id":null,"evidence_quote":"Baseline that assigns different $N$ values with fixed $M$ based on layer outlier ratios; FLOW generalizes this by also selecting $M$."},{"cited_title":"Vegeta: Vertically-integrated extensions for sparse/dense gemm tile acceleration on cpus,","cited_arxiv_id":null,"evidence_quote":"Flexible $N{:}M$ digital accelerator baseline and source of the CSC sparse storage format FlexCiM adapts."},{"cited_title":"Sdp: Co-designing algorithm, dataflow, and architecture for in-sram sparse nn acceleration,","cited_arxiv_id":null,"evidence_quote":"Fixed 1:2 sparse DCiM macro that FlexCiM partitions and extends with distribution and merging units."},{"cited_title":"SparseGPT: Massive language models can be accurately pruned in one-shot,","cited_arxiv_id":null,"evidence_quote":"One-shot Hessian-based pruning baseline that FLOW is compared against on perplexity and accuracy."}],"review_version":1}