{"id":"61f2fb69-a5f1-4a4b-8702-468bb17ce28a","arxiv_id":"2505.10782","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"EdgeMM is a 22nm multi-core CPU that combines systolic-array and compute-in-memory coprocessors with adaptive activation-aware pruning, reported as 2.84x faster than a laptop RTX 3060 for multimodal LLMs.","lead":"This paper describes EdgeMM, a multi-core CPU design for edge devices that adds two kinds of AI helper units: a systolic array for compute-heavy image and text matrix math, and a compute-in-memory unit for memory-heavy token generation. The authors report a 2.84x speedup over a laptop RTX 3060 GPU for multimodal AI models, but the results are simulation-based and contain inconsistent energy numbers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 138 tokens/s claim implies roughly 300GB/s of weight traffic for TinyLlama-1.1B, yet no DRAM bandwidth or CIM capacity is reported; the 0.217/0.28 token/J efficiency values are also inconsistent with 112mW, so the 2.84x speedup is not auditable.","rationale":"The architecture itself is plausible: heterogeneous systolic-array and digital-CIM coprocessors around a Snitch cluster, with activation-aware pruning, is a reasonable direction for edge MLLMs. The 22nm RTL implementation and post-P&R power estimate are legitimate artifacts. However, the paper's headline result rests on an evaluation that omits the memory-system specification needed to check the throughput claim and contains an internal efficiency inconsistency. The reader's verdict is already CONDITIONAL, which is the right level: the paper should not be accepted as-is, but the concerns are fixable by releasing the simulator, reporting DRAM bandwidth and CIM capacity, correcting the efficiency units, and validating pruning accuracy on more than one model. I do not see a reason to move to REJECT, because the central architectural idea may still be sound and the reported numbers could become consistent after a proper memory-system audit. My concern is more specific than the reader's weakest assumption, which focused on simulator fidelity in general; the sharper issue is that the claimed throughput's memory bandwidth is not even stated, and the stated efficiency cannot be true. Hence partial agreement.","tokens_in":11292,"tokens_out":9740,"duration_ms":104962,"concrete_test":"","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 2.84x speedup over RTX 3060. For that to hold, EdgeMM's simulated evaluation must model a memory system able to supply GEMV weight streams at the claimed 138 tokens/s. SPHINX-Tiny's decoder is TinyLlama-1.1B; at BF16, each decoding token requires reading at least ~2.2GB of weights if the model is not fully on-chip. That is ~304GB/s sustained traffic at 138 tokens/s, close to the RTX 3060's 336GB/s and far above what a typical 22nm edge multicore DRAM system provides. Section V-A only says performance is evaluated by RTL simulation and an in-house simulator; it reports no DRAM bandwidth, no CIM macro capacity, no fraction of weights resident on-chip, and no external memory configuration. Thus the dominant memory-system assumption behind the headline speedup is unauditable. The same evaluation pipeline also produces internally inconsistent efficiency numbers: 138 tokens/s at 112mW is 1232 tokens/J (0.812mJ/token), not 0.217 token/J in Section I or 0.28 token/J in Section V/Table II. Since these numbers cannot all be true, the simulator's output has not been reconciled, and the 2.84x speedup cannot be taken at face value.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EdgeMM, a multi-core RISC-V CPU with heterogeneous AI extensions for edge multimodal LLM inference: compute-centric cores with a systolic-array GEMM coprocessor and memory-centric cores with a digital compute-in-memory GEMV coprocessor. It also introduces layer-wise dynamic top-k activation-aware weight pruning and token-length-driven DRAM bandwidth management. The headline claim, from the abstract, introduction, and conclusion, is a 2.84x performance speedup over a laptop RTX 3060 GPU for representative MLLMs such as SPHINX-Tiny and Karmavlm, at 138 tokens/s and 112mW in commercial 22nm technology. The paper reports a 42% latency reduction from the pruning scheme and a 2.14x throughput improvement from bandwidth management at certain output token lengths.","tokens_in":11522,"tokens_out":6448,"duration_ms":65010,"significance":"If the quantitative claims were fully supported, the paper would make a solid systems contribution: it identifies the GEMM/GEMV phase imbalance in edge MLLMs, maps those phases to heterogeneous in-core coprocessors, and couples the design with a dynamic activation-channel pruning scheme. A 22nm RTL implementation with place-and-route power and area numbers is a concrete artifact. However, the significance is conditional: the central speedup and efficiency claims rest on an internally inconsistent set of efficiency numbers and on an evaluation pipeline whose memory-system assumptions are not reported. The architecture-level ideas are credible and worth pursuing, but the paper as written does not yet provide an auditable quantitative case.","major_comments":[{"comment":"The reported energy-efficiency numbers are internally inconsistent. The contribution bullet in Section I reports 0.217 token/J, while Section V-C and the conclusion report 0.28 token/J. From the stated throughput and power of 138 tokens/s and 112mW, the efficiency is 1232 tokens/J (0.812 mJ/token). These three values cannot all be true. This is a load-bearing inconsistency because the 2.84x speedup claim depends on the same simulation/power report; the authors must reconcile the methodology that produced these numbers.","section":"Section I, Section V-C, Table II, Conclusion"},{"comment":"The memory system behind the headline throughput is not specified. For SPHINX-Tiny, whose decoder is TinyLlama-1.1B, reading the full weight set at BF16 requires roughly 2.2GB per decoding token; at 138 tokens/s that is about 304GB/s of sustained weight traffic, or about 176GB/s even with the reported 42% pruning. The paper reports no DRAM bandwidth, no CIM macro capacity, no on-chip weight-residency fraction, and no external memory configuration. Without these parameters, the 138 tokens/s and the resulting 2.84x speedup are not auditable. Please add a memory hierarchy specification and a bandwidth/roofline analysis.","section":"Section V-A"},{"comment":"The cycle counts for the systolic array and CIM coprocessors, L_SA = 2R+C+M-3 and L_CIM = M*W+1, are asserted without derivation or validation. These equations directly determine the modeled performance and hence the comparison with the RTX 3060. Please provide a derivation, or better, a validation table showing modeled cycles against RTL-simulated cycles for representative matrix shapes and data types.","section":"Section III-B, Equations (2)-(3)"},{"comment":"The generality of the activation-aware pruning scheme is not established. The threshold t=16 is fixed, the first layer is skipped because, as the text states, pruning it causes significant accuracy degradation, and the profiling is performed on only two MLLMs. Accuracy impact is reported only as cosine similarity and a qualitative 'minimal score reduction in VQA' with no end-task numerical scores. The claimed 42% latency reduction and the additional speedup in Table II therefore need a sensitivity analysis over t, a justification for skipping the first layer, and end-task accuracy numbers on a broader set of models.","section":"Section IV-A and Section V-C, Figure 12"}],"minor_comments":[{"comment":"The comparison with the RTX 3060 Laptop is underspecified: please report the GPU power, software stack, batch size, output token length, and precision used for the GPU baseline, and state whether the GPU number is measured or taken from a prior source.","section":"Table II"},{"comment":"The name 'activation-aware weight pruning' is potentially misleading because the scheme prunes weight rows corresponding to low-magnitude activation channels; a term such as 'dynamic activation-channel row pruning' would be more precise.","section":"Section IV-A"},{"comment":"Figure 3(b) should define the x-axis and the color/height scale in the caption; in Section V-C, the sentence 'with pruning ratio of a randomly chosen core' should clarify whether the curve is a single core or an average over cores and why that is representative.","section":"Figure 3(b) and Section V-C"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the architecture direction is timely and the 22nm implementation effort is commendable, but the paper in its current form contains internally contradictory efficiency numbers and omits the memory-system parameters needed to evaluate the headline 2.84x claim. These are fixable with additional methodology and validation, so I am not recommending rejection, but the authors should be required to reconcile the efficiency figures and provide the missing system-level details before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jean, quick take on arXiv:2505.10782. The architectural package is genuinely interesting: a multi-core RISC-V CPU with systolic-array coprocessors for compute-heavy GEMM and digital-CIM coprocessors for memory-bound GEMV, plus activation-aware weight pruning and bandwidth scheduling. That combination is new as a system, and the workload analysis in Fig. 2 is well done. I'd give credit for the design insight.\n\nThe problem is the evaluation. The paper reports 138 tokens/s at 112mW, which is about 1232 tokens/J, yet the abstract says 0.217 token/J and Sec. V/Table II say 0.28 token/J. Those are off by orders of magnitude and the discrepancy is not discussed. That's load-bearing because the headline 2.84x speedup over an RTX 3060 depends on the simulator being right.\n\nThe stress-test also makes a sharp point: for TinyLlama-1.1B at BF16, each decoding token needs roughly 2.2GB of weight traffic, so 138 tokens/s implies about 304GB/s sustained DRAM bandwidth. The paper reports no DRAM bandwidth, no CIM capacity, and no on-chip weight residency split. Without those, the memory system is not auditable. The RTX 3060 baseline is also thin: just a '1x' column, no measured power, bandwidth, or tokens/s.\n\nThe pruning work is more defensible: it extends Wanda and CATS to per-layer dynamic k, and they show cosine similarity plus some VQA scores. But the threshold t=16 and the skip-first-layer rule are chosen after seeing the accuracy results, and only two MLLMs are profiled. That's a post-hoc risk, but not fatal.\n\nThe paper ships no code or bitstreams, and the CIM macro is in-house IP. That's normal for architecture papers, but combined with the internal inconsistency, the current numbers are not credible as reported.\n\nWho should read it: architecture folks working on edge LLM accelerators, especially those mixing CIM and systolic arrays. It deserves a serious referee—the idea is real and worth fixing—but the evaluation section needs a major rework: reconcile the efficiency numbers, report memory system parameters, and add sensitivity analysis. If I were the editor, I'd send it to review and expect the authors to address these before acceptance.","headline":"The architecture is a real integration of systolic-array and digital-CIM coprocessors for MLLMs, but the headline efficiency numbers are internally inconsistent and the memory system is unauditable, so the 2.84x claim cannot be taken at face value.","tokens_in":12192,"tokens_out":2079,"would_cite":false,"duration_ms":18746,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-core CPU extended with systolic-array and compute-in-memory coprocessors claims 2.84x speedup over a laptop RTX 3060 for edge multimodal LLM inference.","keywords":["multimodal large language model","edge inference","multi-core CPU","systolic array","digital compute-in-memory","activation-aware weight pruning","bandwidth management","RISC-V ISA extension"],"falsifier":"Measure the EdgeMM CIM macro and systolic array on a fabricated testchip or calibrated silicon-validated macros, then replay the same SPHINX-Tiny and KarmaVLM workloads with measured cycle counts, power, and DRAM traffic in the system simulator; if the measured GEMV throughput or DRAM reduction falls materially below the model, the claimed 2.84x speedup and 138 tokens/s do not survive.","tokens_in":10966,"feed_emoji":"⚡","tokens_out":6794,"duration_ms":62313,"temperature":0.7,"pith_summary":"EdgeMM argues that the place for multimodal LLM inference at the edge is not a GPU but a multi-core CPU densely extended with two specialized coprocessors: compute-centric systolic arrays for the GEMM-heavy vision encoder and prefill phase, and memory-centric digital compute-in-memory (CIM) macros for the GEMV-heavy decoding phase. On top of this heterogeneous substrate, the paper proposes dynamic activation-aware weight pruning that skips near-zero activation channels and their corresponding weight rows, reducing DRAM traffic, and token-length-driven bandwidth management that shifts memory budget between core types as output length grows. The claimed result is 2.84x higher throughput than a laptop RTX 3060 GPU on representative multimodal LLMs, reaching 138 tokens/s with roughly 112 mW chip power at 22nm. If correct, this makes privacy-preserving, real-time multimodal AI feasible on power-constrained devices without ASIC compiler effort, because the accelerators are exposed as CPU ISA extensions.","feed_headline":"Heterogeneous CPU chip claims 2.84x speedup over laptop GPU","feed_subtitle":"Systolic arrays plus compute-in-memory hit 138 tokens/s at 112 mW for multimodal LLMs on 22nm.","key_machinery":"The load-bearing mechanism is a heterogeneous coprocessor pair driven by extended RISC-V instructions: a systolic-array coprocessor (compute-centric, GEMM-optimized, $\text{L}_{SA}=2R+C+M-3$ cycles) and a digital CIM coprocessor (memory-centric, GEMV-optimized, $\text{L}_{CIM}=M W+1$ cycles) that keeps weights in SRAM and broadcasts activations bit-serially. Around these sits the hardware pruner implementing Algorithm 1: per-core activation vectors are Top-k selected, the maximum value masks channels below a threshold $t=16$, and the resulting index list drives the address generator so only non-pruned weight rows are fetched from DRAM and multiplied in the CIM array. Bandwidth management is the third piece: each cluster gets a memory-access budget $B$ enforced by performance-monitoring counters in the DMA, and the $B_c:B_m$ ratio is adjusted with output token length, switching to stream-based batch processing when the budget ratio saturates.","core_discovery":"The paper's central claim is that the two phases of multimodal LLM inference have opposite hardware bottlenecks—compute-bound GEMM in the encoder and prefill, memory-bound GEMV in decoding—so a single homogeneous accelerator wastes most of its silicon in the phase it was not built for. EdgeMM therefore pairs CC-cores, whose systolic-array coprocessor keeps weights stationary and streams activations, with MC-cores, whose digital CIM macro keeps weights in SRAM and broadcasts activations bit-serially, cutting load/store traffic. The authors further claim that activation vectors inside the gated-MLP are channel-sparse with layer-dependent outliers, and that their layer-wise dynamic Top-k pruning can drop many weight rows at DRAM access time with minimal accuracy loss, cutting decoding latency by 42%. Together with bandwidth budgeting that throttles CC-clusters when output tokens lengthen, EdgeMM reports 2.84x speedup over a laptop RTX 3060, 138 tokens/s, and about 112 mW power at 22nm.","pith_inferences":["A natural extension not tested in the paper is combining activation-aware row pruning with weight quantization and structured sparsity; the digital CIM's bit-serial multiply logic suggests that activation bit-width, not just active channels, could also be pruned on the fly.","The dynamic Top-k policy is validated by cosine similarity and one VQA score on two small MLLMs; applying it to larger MLLMs or to models with different FFN dimensions would show whether the layer-depth trend generalizes.","Because the architecture exposes coprocessors through ISA extensions, the same cluster graph could be retargeted to other token-processing workloads, such as speech encoders or streaming sensor fusion, both of which share the GEMM-then-GEMV phase split.","The reported 2.84x GPU comparison assumes a laptop GPU under specific bandwidth conditions; a fairer test would measure against a mobile GPU with comparable thermal and power envelope, which may change the speedup sign."],"forward_implications":["Edge devices could run multimodal LLMs on CPU-class hardware with ISA-visible coprocessors, inheriting the existing RISC-V compiler stack and avoiding bus-attached ASIC control latency.","Because the pruner cuts DRAM traffic by skipping weight rows, decoding throughput scales with activation sparsity rather than only with raw memory bandwidth.","The reported 42% decoding-latency reduction from dynamic Top-k pruning suggests similar gains for other gated-MLP small LLMs with channel-sparse activations.","Token-length-driven bandwidth allocation keeps the encoder/prefill and decoding pipeline balanced, and batch processing recovers throughput when output sequences grow past the budget range.","At 112 mW and 138 tokens/s, a sub-watt, 22nm-class chip can plausibly serve real-time MLLM tasks if the simulation assumptions hold in silicon."],"supporting_citations":[{"why":"Provides the multi-core cluster and host-core design that EdgeMM extends with its AI coprocessors.","marker":"[43]"},{"why":"Supplies SPHINX-Tiny as a representative MLLM for profiling, pruning evaluation, and latency measurement.","marker":"[25]"},{"why":"Supplies KarmaVLM as the second representative MLLM for workload profiling.","marker":"[30]"},{"why":"Defines the CLIP vision encoder used in the profiled models and motivates the compute-bound encoder phase.","marker":"[19]"},{"why":"Documents the gated-MLP FFN structure in mainstream LLMs that the activation-sparsity analysis targets.","marker":"[40]"},{"why":"Prior activation-aware pruning with fixed k that EdgeMM's dynamic layer-wise Top-k scheme extends.","marker":"[44]"},{"why":"Prior context-aware thresholding approach that the paper's dynamic pruning builds on conceptually.","marker":"[45]"},{"why":"Provides the VQA benchmark used to show that dynamic pruning causes only minimal accuracy reduction.","marker":"[35]"}],"fun_headline_variants":["Systolic arrays plus compute-in-memory speed multimodal LLMs 2.84x","EdgeMM chip pairs systolic arrays and compute-in-memory for 2.84x LLM speedup","Activation-aware pruning helps EdgeMM beat laptop GPU by 2.84x","Heterogeneous CPU with CIM and systolic arrays yields 138 tok/s at 112 mW","EdgeMM: 2.84x faster than laptop GPU for multimodal LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported speedup rests on the paper's own simulation and CIM-model tools faithfully representing both the EdgeMM chip and the laptop RTX 3060 baseline; if those models are optimistic, the 2.84x result collapses.","fun_headline_variants_meta":{"raw":{"variants":["Systolic arrays plus compute-in-memory speed multimodal LLMs 2.84x","EdgeMM chip pairs systolic arrays and compute-in-memory for 2.84x LLM speedup","Activation-aware pruning helps EdgeMM beat laptop GPU by 2.84x","Heterogeneous CPU with CIM and systolic arrays yields 138 tok/s at 112 mW","EdgeMM: 2.84x faster than laptop GPU for multimodal LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000775,"raw_usage":{"total_tokens":3415,"prompt_tokens":916,"completion_tokens":2499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":2382}},"tokens_in":532,"tokens_out":2499,"duration_ms":14629,"temperature":1.0,"reasoning_tokens":2382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:04:01.025090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the EdgeMM CIM macro and systolic array on a fabricated testchip or calibrated silicon-validated macros, then replay the same SPHINX-Tiny and KarmaVLM workloads with measured cycle counts, power, and DRAM traffic in the system simulator; if the measured GEMV throughput or DRAM reduction falls materially below the model, the claimed 2.84x speedup and 138 tokens/s do not survive.","supporting_citations":[{"cited_title":"Snitch: A tiny pseudo dual-issue processor for area and energy efficient execution of floating- point intensive workloads,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-core cluster and host-core design that EdgeMM extends with its AI coprocessors."},{"cited_title":"https://github.com/ thomas-yanxin/KarmaVLM, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies KarmaVLM as the second representative MLLM for workload profiling."},{"cited_title":"Learning transferable visual models from natural lan- guage supervision,","cited_arxiv_id":null,"evidence_quote":"Defines the CLIP vision encoder used in the profiled models and motivates the compute-bound encoder phase."},{"cited_title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answering,","cited_arxiv_id":null,"evidence_quote":"Provides the VQA benchmark used to show that dynamic pruning causes only minimal accuracy reduction."}],"review_version":1}