{"id":"3d7543ab-9eaa-4a2b-84c8-e803ef52317a","arxiv_id":"2506.11728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Mixed-precision integer GEMM micro-kernels for ARM NEON, SVE2, Intel AMX, ARM SME, and RISC-V IME give 1.7 to 2.3 times faster quantized inference on three edge CPUs than the authors' FP32 baseline.","lead":"This paper shows how to speed up quantized neural network inference on three CPU families by replacing floating-point matrix multiplication with mixed-precision integer GEMM kernels. On a Raspberry Pi, an NVIDIA Jetson, and a RISC-V board, the new kernels run ResNet and BERT models about 1.7 to 2.3 times faster, with accuracy drops near one percent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Speedup claims rest on an unbenchmarked FP32 baseline; a single external GEMM comparison would settle whether the headline ratios are real.","rationale":"The reader's weakest_assumption is exactly my load-bearing concern: the credibility of the MIP speedups depends on the FP32 baseline being near-optimal. The Pith reader already flags this and recommends CONDITIONAL with external baselines. My stress-test does not find a different or stronger objection: the quantization accuracy (Tables 6-7) is plausible, the micro-kernel descriptions are internally consistent, and the SpacemiT K1 IME measurements are a novel contribution. The energy numbers in Section 8 are indeed unsupported ('not fully measured' vs. precise Joule figures), which is a real flaw, but the paper's core claim is the speedup, not energy. The single most load-bearing check is therefore an external FP32 GEMM comparison on the same boards. Because the reader already captured this and the verdict (CONDITIONAL) is appropriate, I keep the verdict unchanged rather than escalating. A verification step worth running regardless: re-derive the aggregated speedups from per-layer Figure 6/7 data, since the paper reports per-layer GOPS and then aggregated speedups without showing the aggregation arithmetic (e.g., the 2x for A78AE with per-layer 'average speed-ups around 4.2x' vs. aggregated 2x is consistent with im2col/packing overhead but the paper does not show the breakdown).","tokens_in":24557,"tokens_out":1648,"duration_ms":13977,"concrete_test":"Run the same ResNet50/BERT-Large workload through OpenBLAS sgemm, BLIS sgemm (or ARM Performance Libraries on the two ARM boards) and the authors' FP32 GEMM on identical hardware, measuring end-to-end inference time or per-GEMM time. If the authors' FP32 baseline is within, say, 10-15% of the best tuned FP32 library on all three boards, the speedup ratios stand; if the libraries are meaningfully faster, the reported speedups should be recomputed against the faster FP32 baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 7.5: 1.67x/2x/2.1x end-to-end speedups for ResNet50 and similar BERT-Large figures) is a ratio: MIP time over FP32 time. The numerator (MIP) is new and described in detail, but the denominator is 'our FP32 GEMM that mimics GotoBLAS2' (Section 7.4), with competitiveness asserted only via citation [33] and never demonstrated in this paper on these three boards. The concern is not that the authors are dishonest; it is that the reported speedup is only meaningful if the FP32 baseline is close to the best achievable FP32 GEMM on each platform. On edge CPUs, vendor/optimized FP32 libraries (OpenBLAS, BLIS, ARM Performance Libraries, RISC-V optimized libraries) can differ by 2x or more from a hand-tuned but unvalidated kernel, especially on NEON without dot-product instructions and on the 8-core K1 with only 4 IMEs. Also, the paper itself concedes energy was 'not fully measured' (Section 8), yet quotes 0.98 vs 0.20 Joules; that is a secondary inconsistency, not the load-bearing issue. The load-bearing risk remains: if the FP32 baseline is, say, 1.5x slower than OpenBLAS/ARMPL, the reported '2x' MIP advantage drops to roughly 1.3x, changing the paper's headline contribution from 'roughly 2x' to 'modest gains on already-good FP32.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic study of mixed-precision integer (INT8 inputs, INT32 accumulation) GEMM kernels for quantized deep learning inference on CPU architectures. It reviews the evolution of SIMD and matrix-engine ISAs (NEON, ARMv8.2 dot product, SVE2, SME, Intel AMX, and the SpacemiT K1 IME), describes micro-kernel and packing designs for each, and evaluates two representative workloads (ResNet50v1.5 on ImageNet and BERT-Large on SST-2) on three edge platforms: Raspberry Pi (Cortex-A72), Jetson AGX Orin (Cortex-A78AE), and Banana Pi BPI-F3 (SpacemiT K1). The reported end-to-end speedups over a custom FP32 GEMM range from about 1.67x to 2.32x, with accuracy drops near 1%. The paper also makes qualitative and quantitative claims about energy savings. The central claims are plausible given the hardware throughput ratios, but the credibility of the speedup numbers depends on the competitiveness of the FP32 baseline, which is not demonstrated in the paper.","tokens_in":24814,"tokens_out":8329,"duration_ms":76690,"significance":"The paper addresses a timely and practically important topic: fast quantized GEMM on heterogeneous CPU/edge hardware. Its strengths include a broad taxonomy of mixed-precision ISA extensions, concrete micro-kernel and packing designs for several modern ISAs, direct measurements on three quite different platforms, and quantified accuracy impact for two standard DL workloads. The per-layer GOPS improvements are broadly consistent with the instruction-level throughput ratios reported in Table 1, which lends internal credibility to the implementation work. However, the headline speedups are ratios against a self-implemented FP32 GEMM whose competitiveness is only asserted via a previous self-citation and never benchmarked in this paper. The energy figures in the conclusions are also not backed by a described measurement. If the baseline comparison is added and the energy claims are corrected, the paper would be a useful reference for practitioners implementing quantized inference on CPUs.","major_comments":[{"comment":"The speedups reported in Section 7.5 for ResNet50v1.5 (1.67x, 2x, 2.1x) and in Section 7.6 for BERT-Large (2.25x, 1.67x, 2.32x) are ratios against 'our FP32 GEMM' described in Section 7.4. The text claims this baseline is competitive with BLIS, OpenBLAS, and ARM PL only by citing the authors' previous work [33]; no benchmark of the FP32 baseline against those libraries is included for the three boards evaluated here. If the FP32 baseline is slower than a well-tuned vendor library, all headline speedups are inflated. Please add a head-to-head FP32 GEMM comparison (GOPS and/or total inference time) against at least one established BLAS implementation on each platform, and report the achieved fraction of the theoretical FP32 peak. It would also strengthen the paper to compare the MIP GEMM against an existing quantized GEMM library (e.g., ruy, QNNPACK, or oneDNN) on the same platforms.","section":"Section 7.4"},{"comment":"The conclusion states that energy was 'not fully measured' but then gives precise energy values: 0.98 J vs 0.20 J for a single-image ResNet50 inference on the ARM Cortex-A78AE and 0.79 J vs 0.29 J for BERT-Large with l=512, b=1. These numbers are not backed by any described measurement methodology, and the caveat 'not fully measured' directly contradicts their presentation as results. Please either add a full energy measurement methodology (power measurement device, sampling rate, CPU frequency/power-state settings) and make energy a real experimental result, or delete the quantitative Joule figures and keep only the qualitative statement.","section":"Section 8"},{"comment":"The dynamic quantization method is not specified precisely enough to reproduce the accuracy results. The paper does not say whether the INT8 scale factors are per-tensor or per-channel, how the activation ranges are determined (e.g., calibration set, running statistics, or runtime min/max), how biases and scaling are fused, or which parts of the models (softmax, layer normalization, GELU, the M5/M7 attention GEMMs) remain in FP32. Since Tables 6 and 7 make a quantitative claim of at most about 1% accuracy drop, please provide a complete, step-by-step description of the quantization pipeline.","section":"Section 7.3"},{"comment":"The FP32 baseline top-1 accuracy of 0.711 on 50,000 ImageNet images is approximately five percentage points below the expected performance of ResNet50v1.5 on ImageNet (typically around 0.76). If the evaluation pipeline (preprocessing, resize/crop, batch size, or data split) differs from standard practice, please describe it; otherwise this suggests a measurement issue, and the claim that quantization drops accuracy by only about 1% may not transfer to standard conditions.","section":"Table 6"}],"minor_comments":[{"comment":"There are several typos and spacing issues, including 'ResNet50v1.9' in Section 8 (should be v1.5), 'propietary', 'descructively', 'atractive', 'wokloads', and inconsistent spacing in 'A VX' and 'V oltage'.","section":"Throughout"},{"comment":"The right-hand 'Aggregated time' plots label the x-axis as 'Layer' but appear to show cumulative time over layers; please clarify the x-axis semantics and how the cumulative curve is constructed.","section":"Figures 6 and 7"},{"comment":"The sentence 'In a previous work [33] we demonstrated...' is a self-citation; if the revised paper includes the requested head-to-head baseline benchmarks, this sentence should be updated or removed so that the competitiveness claim is evidenced in this manuscript.","section":"Section 7.4"},{"comment":"The statement that average per-layer speedups of 2.3x and 4.2x 'matches the number of SIMD units in each processor' is unclear and not substantiated; please replace it with a direct reference to the measured instruction throughput ratios or to Table 1.","section":"Section 7.5"},{"comment":"The performance results are reported as averages over at least 50 seconds, but no standard deviation or run-to-run variation is given; please report at least min/max or standard deviation for the end-to-end times, particularly on the 8-core SpacemiT K1 where thread/IME scheduling may vary.","section":"Section 7.1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope as a systems/performance study. The main risk is the unbenchmarked FP32 baseline; if the authors add the requested comparison and remove the unsupported energy numbers, I would be willing to reconsider. I would also ask the editor to have the authors verify the Table 6 baseline accuracy, which is unusually low for ResNet50v1.5 on ImageNet."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Practical take: this is a useful engineering paper, not a conceptual breakthrough. It gives detailed, ISA-by-ISA recipes for adapting GotoBLAS-style GEMM to INT8 dot-product and matrix-engine instructions, and it ships the first measurements I've seen on the SpacemiT K1's IME. The per-layer and end-to-end speedups (1.67x-2.32x on three edge CPUs, with roughly 1% accuracy loss) are plausible and internally consistent with the raw GOPS ratios in Table 1.\n\nWhat it does well: the ISA taxonomy table is a handy reference; the micro-kernel code for ARMv8.0 NEON, ARMv8.2 NEON, SVE2, and the K1 assembly are concrete enough to port; the accuracy reporting is straightforward; and the paper is honest that the K1's four IMEs are not better than all eight cores on these workloads.\n\nThe soft spots are real but not fatal. The headline ratios are MIP time divided by 'our FP32 GEMM that mimics GotoBLAS2' (Section 7.4), whose competitiveness on these boards is asserted via citation [33] and never demonstrated in-paper. If that baseline is 1.5x slower than OpenBLAS or ARMPL, the 'roughly 2x' story shrinks to 'modest gains.' The authors are credible HPC people, so I'd bet the baseline is decent, but I want to see a race against at least one vendor library on each board. A second issue: Section 8 says energy was 'not fully measured' yet quotes 0.98 vs 0.20 Joules and 0.79 vs 0.29 Joules with no methodology. Either describe the measurement or cut the numbers. Minor: no error bars, no code/artifact link, and no comparison to existing quantized GEMM libraries like Ruy or QNNPACK. Those are additions, not corrections.\n\nThis paper is for people who port GEMM kernels to new ISAs and care about CPU inference on edge devices. It deserves a serious referee. I'd send it to review and ask for external baselines and energy clarity; the core engineering is worth publishing.","headline":"Useful kernel porting guide with new K1 measurements, but the headline speedup ratios rest on an unvalidated FP32 baseline and an unmeasured energy claim.","tokens_in":25425,"tokens_out":3007,"would_cite":true,"duration_ms":28527,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By recasting GEMM's innermost loop as INT8 dot products accumulating into INT32, the paper shows quantized ResNet-50 and BERT-Large inference runs about 1.7–2.3× faster than tuned FP32 on three edge CPUs, losing at most about one accuracy…","keywords":["mixed-precision GEMM","INT8 quantized inference","dot-product micro-kernels","matrix engines","ARM NEON and SVE2","Intel AMX","RISC-V SpacemiT K1","edge CPU inference"],"falsifier":"Re-run the two workloads on the same three boards with the paper's FP32 GEMM, the paper's INT8+INT32 GEMM, and a well-tuned FP32 GEMM from a mainstream BLAS library such as OpenBLAS, BLIS, or the ARM Performance Library. If the vendor FP32 kernel materially outperforms the paper's FP32 baseline, the reported speedups shrink proportionally, and if a vendor INT8 GEMM matches or beats the MIP kernels, the claim that hand-tuned micro-kernels are required for these gains would need qualification.","tokens_in":24310,"feed_emoji":"⚡","tokens_out":11679,"duration_ms":100201,"temperature":0.7,"pith_summary":"The paper argues that the classic recipe for fast matrix multiplication on CPUs, the Goto–van de Geijn framework built around vector AXPY operations in FP32, no longer matches what modern hardware offers, and shows how to rebuild it for mixed-precision integer (MIP) arithmetic: INT8 inputs multiplied and accumulated into INT32, the format behind quantized deep-learning inference. It details micro-kernels and packing layouts for six instruction sets, ARMv8.0-A NEON, ARMv8.2 NEON, ARM SVE2, ARM SME, Intel AMX, and the RISC-V SpacemiT K1's integrated matrix engine, while keeping the outer loop structure of the framework intact. On three edge boards, the MIP kernels deliver end-to-end speedups of 1.67×, 2×, and 2.1× for ResNet-50 and 2.25×, 1.67×, and 2.32× for BERT-Large relative to a tuned FP32 GEMM, with accuracy drops of roughly one percentage point or less. If these numbers hold, INT8 GEMM is a practical way to run latency-sensitive inference on commodity CPUs, not only on GPUs or dedicated accelerators.","feed_headline":"INT8 GEMM nearly doubles quantized AI speed on edge CPUs","feed_subtitle":"ResNet-50 and BERT-Large keep within a point of accuracy while INT8 GEMM beats FP32 on ARM and RISC-V.","key_machinery":"The carrying object is the GEMM micro-kernel recast from AXPY to DOT form inside the fixed Goto–van de Geijn macro-kernel, a set of five nested tiling loops with two packing stages that reorganize blocks of A and B into contiguous buffers. The decisive change sits in the innermost loop and in the packer: instead of spreading a rank-1 update over vector FMA instructions on FP32, the micro-kernel issues dot-product instructions, for example vdotq_laneq_s32 in ARMv8.2 NEON (four 4-element INT8 dot products accumulated into INT32 per call), vmadot in the SpacemiT K1 (a 4×4×8 INT8-to-INT32 matrix multiply), tile dpbssd for Intel AMX (16×16×64), and smopa for ARM SME, and the packing routines lay out micro-panels so that each instruction's operands are read with stride-1 access in the native register or tile lane order. The mechanical consequence is that packing and compute are co-designed per ISA while the macro-kernel, the parallelism, and the blocking parameters are inherited from the classical framework.","core_discovery":"The discovery is that the Goto–van de Geijn GEMM blueprint survives the transition to quantized inference, but only if its core is inverted: instead of decomposing the micro-kernel's rank-1 update into AXPY operations on FP32 vectors, the micro-kernel must be expressed as DOT-product operations that feed INT8 operands into mixed-precision accumulators (INT8×INT8→INT32), and the packing routines must be redesigned so that the micro-panels of A and B arrive in the exact lane and tile layout each ISA's dot or matrix-engine instruction expects. The authors show this working across a deliberately heterogeneous set of ISAs, from the small ARM Cortex-A72 with ARMv8.0-A NEON to the ARM Cortex-A78AE with ARMv8.2 dot products, the RISC-V SpacemiT K1's vmadot matrix engine, Intel AMX tiles, and ARM SME's outer-product tiles, an era of diversification they call the Cambrian period of matrix multiplication. A hardware-native DOT instruction compresses the micro-kernel to a few vdotq_laneq_s32 calls, where the older ARMv8.0-A NEON needs multi-stage widening, multiply, and pairwise-add sequences. The empirical payoff is that MIP inference speeds up by roughly 1.7–2.3× end-to-end versus FP32 across both a vision model (ResNet-50 v1.5 on ImageNet) and a language model (BERT-Large on SST-2), quantized accuracy stays within about one percentage point of the FP32 model, and memory footprint drops to roughly a quarter to a third of the FP32 model.","pith_inferences":["The measured boards are early adopters of the new instructions; the peak ratios reported in the paper's Table 1 for newer matrix engines (Intel AMX near 65× and ARM SME near 34× the FP32 FMA rate) imply that on those chips the MIP-versus-FP32 gap could dwarf the roughly 2× observed here, provided packing and the memory pipeline keep the engines fed.","The paper quantizes only weight-to-activation GEMMs in BERT and uses dynamic quantization; applying the same MIP micro-kernels to the attention-score and softmax GEMMs, or to INT4 inputs, is a natural stress test of whether the DOT-centric packing generalizes beyond the reported accuracy results.","On the SpacemiT K1, the four IME accelerators did not beat the eight integer cores (one configuration was 1.2× slower), so the practical way to use the IMEs on RISC-V remains an open tuning question, likely resolved by larger matrices or by overlapping core and accelerator work rather than the static split tested here."],"forward_implications":["Quantized INT8 inference on CPUs reaches a performance tier that previously motivated dedicated accelerators: on the tested boards, the MIP GEMMs run ResNet-50 and BERT-Large end-to-end at 1.7–2.3× the FP32 rate with accuracy cost of about one point or less.","The Goto–van de Geijn framework remains the right scaffolding for high-performance GEMM; the portability burden has moved entirely into the micro-kernel and packing routines, which must be regenerated for each instruction set.","As mixed-precision instructions define the new peak throughput of CPUs, mainstream and vendor BLAS libraries will need MIP GEMM variants as first-class kernels to reach the hardware ceiling.","In the measured cases, energy on the Jetson platform drops 5.1× for ResNet-50 and 2.72× for BERT-Large, making INT8 GEMM a direct lever for battery- and thermally-constrained edge deployment."],"supporting_citations":[{"why":"Goto and van de Geijn's anatomy of high-performance matrix multiplication, the macro-kernel and AXPY micro-kernel blueprint that the paper adapts to MIP arithmetic.","marker":"[11]"},{"why":"Hubara et al. on quantized neural networks, the source of the INT8 weight-and-activation quantization scheme that the MIP GEMM implements.","marker":"[1]"},{"why":"LLM.int8(), which establishes 8-bit matrix multiplication for transformer inference and motivates the BERT-Large evaluation and the INT8-plus-INT32 accumulator choice.","marker":"[2]"},{"why":"Natesh and Kung on dot-product accumulation precision, cited as the justification for preferring DOT-centric MIP kernels over classic AXPY in quantized inference.","marker":"[24]"},{"why":"Chellapilla et al. on im2col lowering, which converts the ResNet-50 convolutions into the GEMMs the micro-kernels execute.","marker":"[29]"},{"why":"The authors' earlier Euro-Par study, cited as the only evidence that the FP32 baseline GEMM is competitive with tuned BLAS libraries on these platforms and therefore the anchor for the reported speedup ratios.","marker":"[33]"},{"why":"The SpacemiT K1 RISC-V IME extension specification, which defines the vmadot instruction the K1 micro-kernel is built on.","marker":"[16]"},{"why":"Remke and Breuer on ARM SME, which documents the ZA tile model and smopa outer-product instructions used in the ARMv9 micro-kernel design.","marker":"[17]"},{"why":"Kim et al. on exploiting Intel AMX for LLM inference, the context the tile_dpbssd AMX micro-kernel follows.","marker":"[13]"}],"fun_headline_variants":["Cambrian explosion of GEMM: INT8 beats FP32 on edge CPUs","INT8 GEMM gives 1.7-2.3x speedup on edge CPUs for quantized AI","Mixed-precision GEMM: 2x faster quantized inference on ARM and RISC-V","The Cambrian era of matrix multiplication for AI inference","INT8 dot products beat FP32 on ARM, x86, and RISC-V for quantized inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported speedups rest on the premise that the paper's FP32 GEMM is nearly as fast as the best possible FP32 GEMM on each of the three boards; the authors cite their earlier study for that competitiveness rather than benchmarking vendor libraries head-to-head here, so a slower-than-optimal FP32 baseline would inflate the gains.","fun_headline_variants_meta":{"raw":{"variants":["Cambrian explosion of GEMM: INT8 beats FP32 on edge CPUs","INT8 GEMM gives 1.7-2.3x speedup on edge CPUs for quantized AI","Mixed-precision GEMM: 2x faster quantized inference on ARM and RISC-V","The Cambrian era of matrix multiplication for AI inference","INT8 dot products beat FP32 on ARM, x86, and RISC-V for quantized inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001463,"raw_usage":{"total_tokens":6005,"prompt_tokens":1186,"completion_tokens":4819,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":802,"completion_tokens_details":{"reasoning_tokens":4705}},"tokens_in":802,"tokens_out":4819,"duration_ms":28268,"temperature":1.0,"reasoning_tokens":4705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:56.915513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the two workloads on the same three boards with the paper's FP32 GEMM, the paper's INT8+INT32 GEMM, and a well-tuned FP32 GEMM from a mainstream BLAS library such as OpenBLAS, BLIS, or the ARM Performance Library. If the vendor FP32 kernel materially outperforms the paper's FP32 baseline, the reported speedups shrink proportionally, and if a vendor INT8 GEMM matches or beats the MIP kernels, the claim that hand-tuned micro-kernels are required for these gains would need qualification.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Goto and van de Geijn's anatomy of high-performance matrix multiplication, the macro-kernel and AXPY micro-kernel blueprint that the paper adapts to MIP arithmetic."},{"cited_title":"Hubara, M","cited_arxiv_id":null,"evidence_quote":"Hubara et al. on quantized neural networks, the source of the INT8 weight-and-activation quantization scheme that the MIP GEMM implements."},{"cited_title":"Dettmers, M","cited_arxiv_id":null,"evidence_quote":"LLM.int8(), which establishes 8-bit matrix multiplication for transformer inference and motivates the BERT-Large evaluation and the INT8-plus-INT32 accumulator choice."},{"cited_title":"PQS (Prune, Quantize, and Sort): Low-Bitwidth Accumulation of Dot Products in Neural Network Computations","cited_arxiv_id":"2504.09064","evidence_quote":"Natesh and Kung on dot-product accumulation precision, cited as the justification for preferring DOT-centric MIP kernels over classic AXPY in quantized inference."},{"cited_title":"Chellapilla, S","cited_arxiv_id":null,"evidence_quote":"Chellapilla et al. on im2col lowering, which converts the ResNet-50 convolutions into the GEMMs the micro-kernels execute."},{"cited_title":"Mart ´ınez, F","cited_arxiv_id":null,"evidence_quote":"The authors' earlier Euro-Par study, cited as the only evidence that the FP32 baseline GEMM is competitive with tuned BLAS libraries on these platforms and therefore the anchor for the reported speedup ratios."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The SpacemiT K1 RISC-V IME extension specification, which defines the vmadot instruction the K1 micro-kernel is built on."},{"cited_title":"Hello SME! Generating Fast Matrix Multiplication Kernels Using the Scalable Matrix Extension","cited_arxiv_id":"2409.18779","evidence_quote":"Remke and Breuer on ARM SME, which documents the ZA tile model and smopa outer-product instructions used in the ARMv9 micro-kernel design."}],"review_version":1}