GPU Parallelization Strategies for Forward and Backward Propagation in Shallow Neural Networks: A CUDA-Based Comparative Study

Amel Sadoun; Fatma Salhi; Nadine Bousdjira; Rania Zitouni; Sarah Hasnaoui

arxiv: 2606.30497 · v1 · pith:O6UVQ33Jnew · submitted 2026-06-29 · 💻 cs.DC · cs.LG

GPU Parallelization Strategies for Forward and Backward Propagation in Shallow Neural Networks: A CUDA-Based Comparative Study

Rania Zitouni , Nadine Bousdjira , Sarah Hasnaoui , Amel Sadoun , Fatma Salhi This is my paper

Pith reviewed 2026-06-30 03:24 UTC · model grok-4.3

classification 💻 cs.DC cs.LG

keywords CUDA optimizationshallow neural networksforward propagationbackward propagationshared memoryGPU performancematrix multiplicationkernel fusion

0 comments

The pith

Three stacked CUDA optimizations deliver a 1.41x speedup for forward and backward passes in shallow neural networks on large datasets.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests three memory-focused improvements applied one after another to a basic graphics-card implementation of the calculations needed to train and run a simple neural network. The improvements target how data moves between slow global memory and fast on-chip memory, how weight matrices are laid out, and how multiple steps are combined into single operations. On the biggest test collection of 25,600 examples the best version finishes in 14.8 seconds instead of the baseline 21.0 seconds. Readers might care because the gains come from standard programming adjustments rather than changes to the learning algorithm itself and are measured against both ordinary processors and multi-threaded versions.

Core claim

Applying tiled shared memory with one-column padding to remove bank conflicts, storing weight matrices in transposed form for better global-memory access patterns, and merging the matrix-multiplication and ReLU steps into one kernel produces a fully optimized CUDA version that runs 1.41 times faster than the unoptimized baseline CUDA code for both forward and backward propagation steps.

What carries the argument

The three stacked CUDA optimizations consisting of tiled shared memory with padding, pre-transposed weight matrices, and a fused MatMul-plus-ReLU kernel.

If this is right

The complete set of optimizations reduces execution time on the 25,600-sample dataset from 21.0 s to 14.8 s.
The same optimized code outperforms both a sequential CPU implementation and an OpenMP version.
Memory-access tuning improves performance of the core linear-algebra steps used in neural-network training.
The relative gains hold across three different dataset sizes on an NVIDIA Tesla T4.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same memory-pattern changes could be tested on networks that use different activation functions to see whether the fusion benefit remains.
Repeating the measurements on other GPU models would show how much the 1.41x factor depends on the specific hardware memory hierarchy.
The work suggests that low-level memory tuning can still matter even when the network itself is shallow.

Load-bearing premise

The reported speedups rest on starting from an otherwise unoptimized baseline CUDA code and on the gains remaining stable when thread-block sizes, padding choices, or network layer widths change.

What would settle it

Running the identical experiment but with a baseline that already contains one or more of the three optimizations, or with a shallow network of different width, and checking whether the 1.41x factor still appears.

Figures

Figures reproduced from arXiv: 2606.30497 by Amel Sadoun, Fatma Salhi, Nadine Bousdjira, Rania Zitouni, Sarah Hasnaoui.

**Figure 4.** Figure 4: Speedup evolution with dataset size. Table II: Training Time vs Hidden Layer Size Hidden Neurons Time (s) Speedup vs 32 32 0.810 1.00× 64 0.921 0.88× 128 1.107 0.73× 256 1.551 0.52× 512 2.743 0.30× 1024 4.645 0.17× VII. DISCUSSION A. Interpretation of Results The experiments clearly show that the optimized version performs better, especially for large datasets. This confirms that the baseline was limited m… view at source ↗

read the original abstract

We present a comparative study of CUDA optimization strategies applied to forward and backward propagation in a shallow neural network. Three stacked optimizations are evaluated: (1) tiled shared memory with bank-conflict elimination via +1-column padding, (2) pre-transposed weight matrices for coalesced global memory access, and (3) a fused MatMul+ReLU kernel that eliminates intermediate global-memory round-trips. Experiments on an NVIDIA Tesla T4 (CUDA 13.0) across three dataset sizes show that the fully optimized implementation achieves a 1.41x speedup over the baseline CUDA version on the large dataset (25,600 samples), reducing execution time from 21.0s to 14.8s. Results are compared against a sequential CPU baseline and an OpenMP parallel implementation, demonstrating the effectiveness of memory-access optimization in GPU-accelerated deep learning primitives.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This is a standard CUDA benchmarking study on a shallow network that stacks three known memory optimizations for a 1.41x measured speedup, but leaves the baseline configuration unspecified.

read the letter

This paper benchmarks three common CUDA optimizations for forward and backward passes in a shallow neural network. The key result is a 1.41x speedup on an NVIDIA Tesla T4 for 25,600 samples, cutting time from 21.0s to 14.8s when all optimizations are stacked.

Nothing here is new. Tiled shared memory with padding to avoid bank conflicts, pre-transposing weights for better access patterns, and fusing matrix multiply with ReLU are all standard practices covered in NVIDIA's own documentation and prior GPU papers. The paper simply applies them together to this specific workload.

What it does reasonably is report measured times on a named GPU across three dataset sizes and include comparisons to a sequential CPU version and an OpenMP implementation. Those numbers give a sense of the practical gains from memory optimizations.

The weaknesses are in the experimental details. The abstract gives no error bars, no description of the baseline kernel's thread block configuration, and no mention of the hidden layer width or whether padding was applied consistently. The stress-test concern is on point: these choices directly affect the memory access patterns that the optimizations target, so the speedup cannot be cleanly attributed without that information. There is also no released code and no check that the fused kernel produces the same numerical results.

This kind of paper is aimed at practitioners who are implementing basic neural net layers on GPUs and want to see these techniques applied end to end. Someone already comfortable with CUDA will find little value.

It does not merit sending out for peer review. The work is too incremental and the reporting too thin to justify referee time.

Referee Report

2 major / 1 minor

Summary. The paper presents a comparative empirical study of three stacked CUDA optimizations (tiled shared memory with +1 padding for bank conflicts, pre-transposed weights for coalesced access, and fused MatMul+ReLU) applied to forward and backward passes in a shallow neural network. On an NVIDIA Tesla T4 with datasets up to 25,600 samples, the fully optimized version is reported to deliver a 1.41x speedup over a baseline CUDA implementation (21.0 s to 14.8 s), with additional comparisons to sequential CPU and OpenMP versions.

Significance. If the timing results prove reproducible and the optimizations preserve numerical correctness, the work supplies concrete, hardware-specific evidence that targeted memory-access changes can improve GPU throughput for basic neural-network primitives. The contribution is modest in scope (shallow networks only, single GPU model, no open artifacts) and does not introduce new algorithms or theoretical bounds.

major comments (2)

[Abstract] Abstract: The headline 1.41x speedup (21.0 s → 14.8 s on the 25 600-sample run) is presented without any description of the baseline kernel’s thread-block dimensions, the exact hidden-layer width, or whether input matrices were padded identically in both versions. Because the three optimizations directly target memory-access patterns that are sensitive to these parameters, the measured difference cannot be unambiguously attributed to the optimizations.
[Abstract] Abstract: No verification is reported that the fused MatMul+ReLU kernel produces numerically identical results to the unfused baseline, nor are error bars, number of timing repetitions, or warm-up procedures described. These omissions make it impossible to assess whether the reported times reflect stable performance differences.

minor comments (1)

[Abstract] The abstract and title refer to a “shallow neural network” but supply no concrete architecture parameters (input dimension, hidden width, output dimension) that would allow readers to reproduce the exact workload.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the abstract. We address each point below and have made revisions to improve clarity and completeness.

read point-by-point responses

Referee: [Abstract] Abstract: The headline 1.41x speedup (21.0 s → 14.8 s on the 25 600-sample run) is presented without any description of the baseline kernel’s thread-block dimensions, the exact hidden-layer width, or whether input matrices were padded identically in both versions. Because the three optimizations directly target memory-access patterns that are sensitive to these parameters, the measured difference cannot be unambiguously attributed to the optimizations.

Authors: We agree that the abstract should include these parameters to allow unambiguous interpretation of the speedup. The manuscript body specifies a thread-block size of 256 threads, a hidden-layer width of 1024, and identical +1 column padding applied to both baseline and optimized versions. We will revise the abstract to briefly state these experimental parameters. revision: yes
Referee: [Abstract] Abstract: No verification is reported that the fused MatMul+ReLU kernel produces numerically identical results to the unfused baseline, nor are error bars, number of timing repetitions, or warm-up procedures described. These omissions make it impossible to assess whether the reported times reflect stable performance differences.

Authors: We acknowledge the omission in the abstract. The full manuscript reports numerical equivalence (maximum absolute difference below 1e-6) between fused and unfused kernels, with timings averaged over 10 runs after 3 warm-up iterations and standard deviations provided. We will add a concise statement to the abstract summarizing the verification and timing methodology. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical timing comparison with no derivations

full rationale

The paper is a direct empirical benchmarking study that measures wall-clock execution times for different CUDA kernel implementations on fixed hardware. No equations, predictions, fitted parameters, or uniqueness theorems are present; the central claim (1.41x speedup) is a measured ratio between two code variants run on the same datasets. All load-bearing steps are external comparisons to CPU/OpenMP baselines and hardware timings, with no self-referential reductions or self-citation chains.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical performance study. No mathematical derivations, fitted constants, background axioms, or postulated entities are introduced.

pith-pipeline@v0.9.1-grok · 5698 in / 1068 out tokens · 38040 ms · 2026-06-30T03:24:00.146056+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 1 canonical work pages · 1 internal anchor

[1]

CUDA C Programming Guide,

NVIDIA Corporation, “CUDA C Programming Guide,” 2024. [Online]. Available: https://docs.nvidia.com/cuda/

2024
[2]

Goodfellow, Y

I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016

2016
[3]

Sanders and E

J. Sanders and E. Kandrot, CUDA by Example: An Introduction to General-Purpose GPU Programming . Boston, MA, USA: Addison - Wesley Professional, 2010

2010
[4]

cuDNN: Efficient Primitives for Deep Learning

S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cuDNN: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759, 2014

work page internal anchor Pith review Pith/arXiv arXiv 2014
[5]

Caffe: Convolutional architecture for fast feature embedding,

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in Proc. 22nd ACM Int. Conf. Multimedia , Orlando, FL, USA, Nov. 2014, pp. 675–678

2014
[6]

Benchmarking GPUs to tune dense linear algebra,

V. Volkov and J. W. Demmel, “Benchmarking GPUs to tune dense linear algebra,” in Proc. 2008 ACM/IEEE Conf. Supercomputing (SC’08), Austin, TX, USA, Nov. 2008, pp. 1–11

2008
[7]

Neural network acceleration study with ReRAM: Opportunities and challenges,

S. Li, A. Mishra, J. J. Doherty, M. Beadon, and S. Cadambi, “Neural network acceleration study with ReRAM: Opportunities and challenges,” in Proc. IEEE Int. Symp. Performance Anal. Syst. Softw. (ISPASS), Uppsala, Sweden, Apr. 2016, pp. 197 –198

2016
[8]

TensorFlow: A system for large-scale machine learning,

M. Abadi et al., “TensorFlow: A system for large-scale machine learning,” in Proc. 12th USENIX Symp. Operating Syst. Design Implementation (OSDI), Savannah, GA, USA, Nov. 2016, pp. 265 –283

2016

[1] [1]

CUDA C Programming Guide,

NVIDIA Corporation, “CUDA C Programming Guide,” 2024. [Online]. Available: https://docs.nvidia.com/cuda/

2024

[2] [2]

Goodfellow, Y

I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016

2016

[3] [3]

Sanders and E

J. Sanders and E. Kandrot, CUDA by Example: An Introduction to General-Purpose GPU Programming . Boston, MA, USA: Addison - Wesley Professional, 2010

2010

[4] [4]

cuDNN: Efficient Primitives for Deep Learning

S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cuDNN: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759, 2014

work page internal anchor Pith review Pith/arXiv arXiv 2014

[5] [5]

Caffe: Convolutional architecture for fast feature embedding,

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in Proc. 22nd ACM Int. Conf. Multimedia , Orlando, FL, USA, Nov. 2014, pp. 675–678

2014

[6] [6]

Benchmarking GPUs to tune dense linear algebra,

V. Volkov and J. W. Demmel, “Benchmarking GPUs to tune dense linear algebra,” in Proc. 2008 ACM/IEEE Conf. Supercomputing (SC’08), Austin, TX, USA, Nov. 2008, pp. 1–11

2008

[7] [7]

Neural network acceleration study with ReRAM: Opportunities and challenges,

S. Li, A. Mishra, J. J. Doherty, M. Beadon, and S. Cadambi, “Neural network acceleration study with ReRAM: Opportunities and challenges,” in Proc. IEEE Int. Symp. Performance Anal. Syst. Softw. (ISPASS), Uppsala, Sweden, Apr. 2016, pp. 197 –198

2016

[8] [8]

TensorFlow: A system for large-scale machine learning,

M. Abadi et al., “TensorFlow: A system for large-scale machine learning,” in Proc. 12th USENIX Symp. Operating Syst. Design Implementation (OSDI), Savannah, GA, USA, Nov. 2016, pp. 265 –283

2016