Pith. sign in

REVIEW 4 major objections 6 minor 24 references

SMM-Conv: Scalar Matrix Multiplication with Zero Packing for Accelerated Convolution

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SMM-Conv replaces im2col+GEMM with scalar-matrix multiplication into one reused buffer per thread, and reports end-to-end CPU speedups of 3.42x on AlexNet, 2.11x on VGG, and 2.00x on YOLO.

desk verdict A clean stride-1 convolution recipe with a reused buffer, but the claimed network speedups rest on an undocumented stride/padding extension and thin baselines. read the letter →

arxiv 2411.15659 v1 pith:T4RLC2FW submitted 2024-11-23 cs.CV

Amir Ofir , Gil Ben-Artzi This is my paper · ORCID
classification cs.CV
keywords convolutionaccelerationCPUinferenceim2colscalar-matrixmultiplicationmemory-efficientchannels-firstlayoutdeepneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a convolution can be computed on CPU without packing the image into a large matrix. The proposed method, SMM-Conv, slices each input channel into overlapping sub-matrices, shifts them down the kernel height, and accumulates scalar-weight products into a single reused buffer per thread. If the measurements are right, this simple recipe outperforms the standard im2col-plus-GEMM pipeline and a memory-efficient convolution baseline on AlexNet, VGG, and YOLO while using far less temporary memory. The practical payoff would be faster inference on mobile and low-power devices with existing, already-trained channels-first models.

What carries the argument

The load-bearing mechanism is scalar-matrix multiplication with zero packing, driven by a 'shifting' operation. For a kernel of height $k_h$ and width $k_w$, the input is sliced into $k_w$ sub-matrices of size $h \times w'$ (all rows, $w'$ consecutive columns); each slice is shifted down $k_h$ times so the rows align with the $k_h$ kernel rows, and each shifted $h' \times w'$ window is multiplied by a single scalar kernel weight and accumulated into the output. Because all scalar-matrix products read from one contiguous $h \times w'$ buffer that is reused across channels and offsets, the method needs only $d$ such buffers for $d$ threads and avoids the $c_i \cdot k_h \cdot k_w \cdot h' \cdot w'$ temporary matrix of im2col. The kernel is stored in $c_i \times k_w \times k_h \times c_o$ layout so the scalar weights are accessed in the same order as the shifts.

What would settle it

Take the first convolutional layer of AlexNet (11x11 kernel, stride 4, padding 2) or a VGG 3x3 stride-1 padding-1 layer, run SMM-Conv against a reference convolution implementation on random input, and compare outputs exactly; a mismatch would show the published algorithm does not cover the configurations claimed in Table 2.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the standard im2col+GEMM pipeline is not the best way to compute a channels-first convolution on a CPU. Instead of copying every kernel-sized image block into a column of a large matrix and then calling a matrix-matrix product, SMM-Conv keeps the input in place: for each input channel it extracts $k_w$ horizontal slices of width $w'$, shifts each slice down $k_h$ times, multiplies the shifted $h' \times w'$ windows by the corresponding scalar kernel weights, and accumulates into the output. The same $h \times w'$ buffer is overwritten for every slice, so temporary memory is about one output-sized matrix per thread. The authors report that this approach runs the convolutional layers of AlexNet, VGG, and YOLO in 0.1348 s, 1.3535 s, and 0.2889 s respectively, corresponding to 3.42x, 2.11x, and 2.00x speedups over im2col+GEMM and faster than the memory-efficient convolution baseline.

Load-bearing premise

The network-level speedups assume the undocumented extension of the stride-1, no-padding algorithm to the padding and stride settings of real network layers; if that extension is wrong, the end-to-end numbers do not hold.

Editorial extensions

If this is right

  • The convolution is exact: the number of multiply-accumulate operations is the same as direct convolution, so the reported speedups come without accuracy loss or retraining.
  • Temporary memory falls from $c_i k_h k_w h' w'$ to $h w'$ per thread, roughly a factor of $c_i k_h k_w$ when $h' \approx h$.
  • Memory use becomes independent of the number of input channels, which matters for low-power devices with tight memory budgets.
  • The method extends naturally to multi-threading: $d$ threads, each owning one $h \times w'$ buffer and $c_o/d$ output maps, run in parallel with a synchronization point per slice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the published description covers stride-1 valid convolution only; real networks use stride and padding, so either the implementation contains undocumented preprocessing or the Table 2 speedups apply only to the subset of layers that match the described setting.
  • Editorial extension: the speedup mechanism implies the largest wins occur when im2col duplication is worst, namely small kernels, many input channels, and spatial sizes where $h' \approx h$; the scalability plots support this, and a user should expect smaller gains for large-stride or heavily padded layers.
  • Editorial extension: the same buffer-reuse idea could be tested on depthwise convolutions and on transposed convolutions, where im2col-style packing is also memory-heavy; the paper does not address these.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SMM-Conv, a CPU convolution method for channels-first tensors that replaces im2col+GEMM with repeated scalar-matrix multiplications on shifted h×w' slices, reusing one memory buffer per thread. It claims to reduce memory overhead and to achieve end-to-end speedups of 3.42x, 2.11x, and 2.00x on AlexNet, VGG, and YOLO, respectively, compared with im2col+GEMM, and to run faster than MEC in all three networks. The core derivation for stride-1, no-padding valid convolution is straightforward, but the printed algorithms contain an output-channel indexing error, and the paper never specifies how padding and stride are handled in the benchmarked networks.

Significance. If the claims hold, SMM-Conv would be an attractive simple alternative for CPU inference: it avoids im2col's memory duplication, has no fitted parameters, and is derived directly from the definition of convolution. The memory-reduction argument around Eq. (1) is sound for the described valid-convolution setting. However, the significance is conditional because the central empirical claim depends on an undocumented extension to non-unit strides and padding, and because the pseudocode as printed is not correct as written. The paper would be strengthened substantially by releasing the implementation and by fixing the algorithm specification.

major comments (4)
  1. [Algorithm 1, line 11] The accumulation target is O[c,:,:] while c is the input-channel loop variable; the correct target is O[m,:,:], where m is the output-channel loop variable. As printed, the single-thread algorithm accumulates all output-channel contributions into the same input-channel-indexed slice, producing an incorrect output tensor or an out-of-bounds access when co < ci. This is a load-bearing error in the algorithm that defines the method.
  2. [Section 3.2 / Algorithms 1-2 / Table 2] The shifting step ShiftedMat = SlicedMat[k:h'+k,:] computes valid convolution with stride 1 and no padding only. The paper never gives the output-size formulas h'=(h+2p-kh)/s+1 and w'=(w+2p-kw)/s+1, nor does it explain how boundary zeros or output decimation are handled for stride s>1. Yet the benchmarked networks contain non-unit strides and padding: AlexNet conv1 uses stride 4 and pad 2, YOLO contains stride-2 convolutions, and VGG uses pad 1 throughout. Consequently, the end-to-end speedups in Table 2 are not supported by the described algorithm. If the implementation computes a dense stride-1 convolution and then subsamples, the MAC count grows by s^2, which contradicts the statement in Section 4.2 that the compared methods share the same number of multiplications and accumulations.
  3. [Algorithm 2, lines 15-17] The parallel indexing lambda*#n is not a valid assignment of output channels. With d threads, #n takes values 0,...,d-1, so K and O are accessed at indices 0, lambda, 2*lambda, ... for different lambda values. This causes overlapping writes for some channels and leaves other channels unwritten, so the parallel algorithm is incorrect as written. The intended indexing is presumably lambda + #n*(co/d) (or an equivalent block partition). Without a correct parallel description, the OpenMP-based speedups in Section 4 cannot be reproduced from the pseudocode.
  4. [Section 4.1 / 4.2] No code is released, and the paper does not state how padding and stride were treated in the implementations that produced Table 2 and Figure 3. Given that the printed algorithm covers only stride-1, no-padding convolution, the experimental section must be accompanied either by the implementation or by detailed per-layer configurations (padding, stride, input/output channel counts, kernel sizes, and how each is mapped to the slicing scheme). Without this, the claimed network-level speedups cannot be checked.
minor comments (6)
  1. [Section 3.2.1] The notation T^1_j is inconsistent with Table 1, which defines T^c_j; please use the latter consistently. Additionally, define h' and w' by explicit formulas that include padding and stride.
  2. [Title / Abstract / Section 3.2] The phrase 'zero packing' is never defined. If it refers to zero-initializing the output buffer, that is not the same as zero-padding the input; please clarify the terminology.
  3. [Section 4.1] Please report compiler flags, OpenMP thread count, CPU frequency/power settings, and whether timings are averaged over multiple runs. These details are important for interpreting the speedup numbers.
  4. [Section 3.4 / Eq. (1)] The sentence 'reduces the total temporary memory by ci*Kh*kw' is ambiguous; Eq. (1) shows a reduction by a factor of approximately ci*kh*kw, so the text should say 'by a factor of approximately ...'.
  5. [Section 4.2 / Reference [6]] The cited reference [6] is YOLO-Lite, not YOLOv3; please cite the original YOLOv3 paper (or clarify if YOLO-Lite is indeed the network used).
  6. [Figure 3] The figure should identify which curve corresponds to AlexNet, VGG, and YOLO, either directly in the plot or in the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SMM-Conv is derived directly from the definition of convolution and benchmarked against external baselines.

full rationale

The paper's derivation chain is self-contained. SMM-Conv is introduced in Section 3.2 as a decomposition of convolution into a summation of kh*kw shifted sub-matrix products, which is a direct algebraic reformulation of the convolution definition rather than a fitted or assumed result. No parameter is fitted to the benchmark data; the kernel weights are given network weights used to compute the convolution itself, and the reported speedups are measured against im2col+GEMM and MEC as external baselines. The memory-requirement comparison in Eq. (1) is an algebraic ratio between two buffer sizes and does not presuppose the claimed speedups. The self-citations in the paper (Refs. [1], [2], [5]) appear only in background statements about computational cost, denoising, and low-rank approximations, and none of them is load-bearing for the central algorithmic claim or the experimental results. Concerns such as the undocumented handling of stride and padding in the network-level benchmarks are correctness or reproducibility risks, not circularity: an undocumented implementation step is not the same as a prediction that reduces to its own input. The central algorithm and its performance comparison therefore do not rely on circular reasoning, self-citation chains, or fitted inputs disguised as predictions.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters or invented entities. The central claim rests on assumptions about baseline representativeness, compiler behavior, and undocumented handling of padding and stride.

assumptions (3)
  • domain assumption The compared baselines (PyTorch/MKL im2col+GEMM and MEC) are representative of state-of-the-art CPU convolution implementations.
    Section 4.1 defines baselines; no comparison to oneDNN or high-performance direct convolution is included, so the claim of outperforming existing methods depends on this assumption.
  • domain assumption The compiler and hardware will apply FMA and SIMD optimizations to the scalar-matrix multiply-accumulate loops, and the cache behavior of the h by w' buffer is beneficial.
    Section 3.4 lists 'assumed reasons' including FMA instructions and CPU caching without measurements.
  • ad hoc to paper All tested convolutional layers in AlexNet, VGG, and YOLO are compatible with the described algorithm under some unstated handling of padding and stride.
    Section 3.2 and Algorithms 1-2 describe only stride-1 valid convolution; Section 4.2 applies the method to layers that typically use padding and stride without explaining the extension.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMM-Conv: Scalar Matrix Multiplication with Zero Packing for Accelerated Convolution." pith.science (2026). https://pith.science/paper/T4RLC2FW

@misc{pith2026241115659,
  author       = {Pith},
  title        = {Pith review of: SMM-Conv: Scalar Matrix Multiplication with Zero Packing for Accelerated Convolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T4RLC2FW}},
  note         = {Machine review of arXiv:2411.15659}
}
read the original abstract

We present a novel approach for accelerating convolutions during inference for CPU-based architectures. The most common method of computation involves packing the image into the columns of a matrix (im2col) and performing general matrix multiplication (GEMM) with a matrix of weights. This results in two main drawbacks: (a) im2col requires a large memory buffer and can experience inefficient memory access, and (b) while GEMM is highly optimized for scientific matrices multiplications, it is not well suited for convolutions. We propose an approach that takes advantage of scalar-matrix multiplication and reduces memory overhead. Our experiments with commonly used network architectures demonstrate a significant speedup compared to existing indirect methods.

Figures

Figures reproduced from arXiv: 2411.15659 by the authors.

Figure 1
Figure 1. Im2col operation (the arrow on the right) with a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our approach. The result of convolutions of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Acceleration of convolutional layers in various neural networks. The x-axis is the depth of the layer and the y-axis is the speedup, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Acceleration of input channels. The x-axis is the number of input channels and the y-axis is the speedup, normalized to im2col [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: A comparison of the speedups of different squared input dimensions. The x-axis represents the first dimension of the input, and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Speedups of various kernel sizes. The x-axis represents the size of the kernels, and the y-axis represents the speedup, normalized [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Speedups of various number of output channels. The x-axis represents the number of output channels, and the y-axis represents [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages

  1. [1]

    Hypernetwork-based adap- tive image restoration

    Shai Aharon and Gil Ben-Artzi. Hypernetwork-based adap- tive image restoration. In ICASSP 2023 - 2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 1–5, 2023. 1

    work page 2023

  2. [2]

    The gray- code filter kernels

    Gil Ben-Artzi, Hagit Hel-Or, and Yacov Hel-Or. The gray- code filter kernels. IEEE transactions on pattern analysis and machine intelligence, 29(3):382–393, 2007. 3

    work page 2007

  3. [3]

    An updated set of basic linear algebra subprograms (blas)

    L Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Iain Duff, Sven Hammarling, Greg Henry, et al. An updated set of basic linear algebra subprograms (blas). ACM Trans- actions on Mathematical Software, 28(2):135–151, 2002. 1

    work page 2002

  4. [4]

    Mec: memory-efficient con- volution for deep neural network

    Minsik Cho and Daniel Brand. Mec: memory-efficient con- volution for deep neural network. In International Confer- ence on Machine Learning, pages 815–824. PMLR, 2017. 1, 2, 5

    work page 2017

  5. [5]

    The role of redundant bases and shrinkage functions in image denoising

    Yacov Hel-Or and Gil Ben-Artzi. The role of redundant bases and shrinkage functions in image denoising. IEEE Transactions on Image Processing, 30:3778–3792, 2021. 1

    work page 2021

  6. [6]

    Yolo- lite: a real-time object detection algorithm optimized for non-gpu computers

    Rachel Huang, Jonathan Pedoeem, and Cuixian Chen. Yolo- lite: a real-time object detection algorithm optimized for non-gpu computers. In 2018 IEEE International Conference on Big Data (Big Data), pages 2503–2510. IEEE, 2018. 4, 5

    work page 2018

  7. [7]

    Math kernel library https://software.intel.com/en- us/intel-mkl, 2015

    Intel. Math kernel library https://software.intel.com/en- us/intel-mkl, 2015. 5

    work page 2015

  8. [8]

    Speeding up convolutional neural networks with low rank expansions

    Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014. 3

    arXiv 2014

Show all 24 references
  1. [9]

    Caffe: Convolutional architecture for fast feature embedding

    Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM inter- national conference on Multimedia , pages 675–67...

    2014

  2. [10]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. In Advances in neural information processing sys- tems, pages 1097–1105, 2012. 1, 5

    2012

  3. [11]

    Fast algorithms for convo- lutional neural networks

    Andrew Lavin and Scott Gray. Fast algorithms for convo- lutional neural networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 4013–4021, 2016. 3

    2016

  4. [12]

    The fast fourier transform

    Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms , pages 80–

  5. [13]

    https : / / docs

    Nvidia: Deep learning performance documentation; ten- sor layouts. https : / / docs . nvidia . com / deeplearning/performance/dl-performance- convolutional/index.html. 1

  6. [14]

    OpenMP application program interface version 3.0, May 2008

    OpenMP Architecture Review Board. OpenMP application program interface version 3.0, May 2008. 5

    2008

  7. [15]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Rai- son, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, L...

    2019

  8. [16]

    https : / / pytorch

    Pytorch: Channels last memory format. https : / / pytorch . org / tutorials / intermediate / memory_format_tutorial.html. 1

  9. [17]

    Binary neural networks: A survey

    Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: A survey. Pattern Recognition, 105:107281, 2020. 3

    2020

  10. [18]

    You only look once: Unified, real-time object de- tection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object de- tection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016. 4

    2016

  11. [19]

    Learning separable filters

    Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pas- cal Fua. Learning separable filters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2013. 3

    2013

  12. [20]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. In In- ternational Conference on Learning Representations , 2015. 5

    2015

  13. [21]

    Improving the speed of neural networks on cpus.[(accessed on 1 may 2019)]; deep learning & unsupervised feature learning workshop nips

    V Vanhoucke and MZ Mao. Improving the speed of neural networks on cpus.[(accessed on 1 may 2019)]; deep learning & unsupervised feature learning workshop nips. 1

    2019

  14. [22]

    Arithmetic complexity of computations , volume 33

    Shmuel Winograd. Arithmetic complexity of computations , volume 33. Siam, 1980. 3

    1980

  15. [23]

    Effi- cient mobile implementation of a cnn-based object recogni- tion system

    Keiji Yanai, Ryosuke Tanno, and Koichi Okamoto. Effi- cient mobile implementation of a cnn-based object recogni- tion system. In Proceedings of the 24th ACM international conference on Multimedia, pages 362–366, 2016. 2

    2016

  16. [24]

    High performance zero-memory overhead direct convolutions

    Jiyuan Zhang, Franz Franchetti, and Tze Meng Low. High performance zero-memory overhead direct convolutions. In International Conference on Machine Learning , pages 5776–5785. PMLR, 2018. 1, 3, 4

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.