Portable Lattice QCD implementation based on OpenCL

Chik Him Wong; Jana N. Guenther; Piyush Kumar; Szabolcs Borsanyi

REVIEW 3 major objections 4 minor 10 references

Portable Lattice QCD implementation based on OpenCL

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a portable OpenCL backend for an existing lattice QCD code matches the performance of the CUDA backend on a single GPU across lattice sizes up to $64^4$, on both NVIDIA and AMD hardware.

desk verdict Honest engineering paper with a real parity claim that needs qualification at large lattice sizes; worth refereeing after minor fixes. read the letter →

arxiv 2502.03249 v1 pith:GY7K6EW3 submitted 2025-02-05 hep-lat

Piyush Kumar , Szabolcs Borsanyi , Jana N. Guenther , Chik Him Wong This is my paper · ORCID

classification hep-lat PACS 11.15.Ha12.38.Gc

keywords OpenCLlatticeQCDGPUportabilityDiracoperatorconjugategradientCUDAstaggeredfermionsperformancebenchmarking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses the cost of hardware diversity in lattice QCD: simulations need GPUs, but NVIDIA and AMD GPUs traditionally require different vendor programming interfaces. The authors built an OpenCL backend into their existing production code so that the same source runs on both, and benchmarked the most expensive operation, the inversion of the staggered Dirac operator via conjugate gradient. Their central claim is that on a single GPU the OpenCL implementation performs on par with the CUDA backend across a range of lattice sizes. If that holds, a lattice QCD code does not need separate vendor-specific backends for the single-GPU case, which is the regime of finite-temperature simulations on moderate lattices.

What carries the argument

The carrying mechanism is the OpenCL execution model applied to the staggered Dirac kernel: the host enqueues kernels with explicit work sizes and work-group decompositions, and the runtime compiles device code just-in-time. The backend stores lattice fields in a structure-of-arrays layout with even sites before odd sites, and it uses gauge fixing to store only the first two rows of each $SU(3)$ link, reconstructing the third row on the fly to cut memory traffic to 126 floats per site. This yields an arithmetic intensity of about 0.57 FLOP/Byte in double precision and 1.13 in single precision; applying the operator to multiple vectors at once (the NMULTI parameter) raises the intensity and improves throughput until about NMULTI=4, after which performance saturates.

What would settle it

Rerun the same staggered-Dslash and CG benchmarks on an NVIDIA A100 with the OpenCL and CUDA backends, using at least ten runs per lattice size with the same compiler and data layout, and report medians with uncertainty; if OpenCL is consistently more than 10% slower than CUDA on any size from $16^4$ to $64^4$, the central parity claim would be false.

Watch

Extended reading notes

Core claim

The paper's claim is that one OpenCL codebase can replace a vendor-specific CUDA backend without a performance penalty for the dominant cost in a lattice QCD simulation. Concretely, for the staggered Dirac operator and its inversion by conjugate gradient, OpenCL on one NVIDIA A100 reaches about the same throughput as the CUDA backend, and on one AMD MI250X compute die it delivers roughly two-thirds of the A100 rate, consistent with the hardware's memory-bandwidth difference. The benchmarks cover lattice sizes from $16^4$ to $64^4$ in single and double precision. The paper also reports that the CG update routine on a $48^4$ lattice sustains about 85% of peak memory bandwidth on the A100 and about 75% on the MI250X.

Load-bearing premise

The load-bearing premise is that the Figure 3 benchmarks are representative: the paper gives no number of runs, error bars, compiler or runtime versions, or timing methodology, so the OpenCL/CUDA parity rests on those point measurements being typical.

Editorial extensions

If this is right

A single source tree can target both NVIDIA and AMD GPUs for single-GPU lattice QCD runs, removing the need to maintain separate CUDA and HIP backends at the current performance level.
Finite-temperature QCD studies on lattices that fit in one GPU can already use the OpenCL backend; the paper states it is in production for equation-of-state work via the density-of-states method.
Multi-GPU scaling is not covered: OpenCL lacks direct GPU-to-GPU communication, so the authors identify SYCL with GPU-aware MPI as the path to portability beyond one device.
For these bandwidth-bound kernels, memory traffic rather than floating-point peak is the binding constraint; the measured saturation past NMULTI=4 marks where the benefit of adding vectors stops.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the parity claim generalizes beyond the Dirac operator to gauge-field updates, force terms, and other kernels, the practical value is larger than the paper states: the entire HMC evolution could run from one portable codebase on either vendor's hardware.
The benchmarks compare throughput but not energy or power; on large systems, energy per inversion is often the real constraint, so an equal-FLOP/s result does not yet tell an operator which backend is cheaper to run.
A testable extension is to run the same NMULTI roofline analysis on the planned SYCL backend: if SYCL recovers the same arithmetic-intensity curve, the portability argument extends naturally to multi-GPU systems.
The absence of reported variance in the benchmarks means the on-par claim should be read as a point estimate; a small follow-up with error bars would turn it into a quantitative parity statement.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is an honest engineering paper that does what it says — an OpenCL backend for a single-GPU lattice QCD code, benchmarked on A100 and MI250X. The headline claim of parity with CUDA is mostly supported by the A100 curves, but the paper qualifies it in Sec. 4.2 in a way that the conclusion doesn't carry over, and that mismatch is worth fixing.

The genuinely new part is the implementation and the benchmarks: a working OpenCL path for the Wuppertal-Budapest code, with SoA layout, even-odd site ordering, and NMULTI vector blocking. The roofline analysis and NMULTI sweep are sensible and the numbers look believable. The authors also get credit for being explicit that multi-GPU is not yet supported and that OpenCL lacks direct GPU-GPU communication. That is the right kind of limitation to state.

The soft spots are real but not fatal. First, the Sec. 5 conclusion says 'on par with CUDA across a range of lattice sizes,' but Sec. 4.2 says CUDA's memory layout is better for larger lattices, giving higher performance there. The plots show the gap growing at the largest sizes. So the conclusion should be qualified to 'small and moderate lattices' or the word 'on par' needs a tolerance. It is a wording problem in a short proceedings paper, but it is the central claim, so it should be fixed. Second, there are no error bars, no timing methodology, and no statement of how many runs each point used. For a benchmark paper that makes a performance-equivalence claim, this is a transparency issue. Third, no HIP baseline on MI250X, so the portability claim on AMD is demonstrated only by OpenCL running, not by comparison with a vendor-optimized backend. Fourth, no code or data artifact. All of these are fixable in a revision.

Who is this for? Lattice practitioners thinking about porting to AMD or looking for portable single-GPU paths. It is a legitimate proceedings contribution, and the underlying measurements can be checked once the methodology and data are supplied. It deserves review, not a desk reject.

Referee Report

3 major / 4 minor

Summary. The paper describes an OpenCL backend added to the existing Wuppertal-Budapest lattice QCD code and benchmarks it against the CUDA backend on a single NVIDIA A100 (JUWELS), with additional OpenCL results on a single AMD MI250X GCD (LUMI). The benchmarks cover the staggered Dirac operator and its inversion via conjugate gradient in single and double precision for lattice sizes up to 64^4, together with a roofline analysis and a multi-vector (NMULTI) optimization sweep. The central claim, stated in Section 5, is that the portable OpenCL implementation achieves performance on par with CUDA across a range of lattice sizes.

Significance. If substantiated, the result is practically significant: a single portable OpenCL codebase could serve both NVIDIA and AMD GPUs without the performance penalty usually associated with portability layers, and the authors report that the backend is already used in production for QCD equation-of-state studies. The paper's strengths are its direct measurements on two leadership-class supercomputers, the concrete roofline calculation, and the NMULTI sweep showing the effect of arithmetic-intensity tuning. However, the benchmark reporting lacks statistical and methodological detail, and the text's own qualification about CUDA's better large-lattice memory layout conflicts with the unqualified conclusion.

major comments (3)

[Sec. 4.2 vs Sec. 5] The concluding sentence of Sec. 5 ('performance on par with CUDA across a range of lattice sizes') is not reconciled with Sec. 4.2's statement that 'The CUDA backend uses a different memory layout which is better suited for larger lattice sizes, thus, giving higher performance than OpenCL for such cases.' Since CUDA is benchmarked only on the A100, this sentence appears to exclude the larger-lattice part of the tested range from the parity claim. Please define 'on par' quantitatively (e.g., a percentage threshold) and either restrict the conclusion to the sizes where parity holds or show that the large-lattice gap is within that threshold.
[Sec. 4.2, Fig. 3] The benchmark methodology is under-specified: the number of runs per point, run-to-run variance, timing procedure (e.g., repeated kernel launches, warm-up, clock source), compiler and OpenCL/CUDA runtime versions, and the CG stopping criterion are not reported. Since the parity claim depends on the relative separation of the OpenCL and CUDA curves, the absence of error bars makes it impossible to judge whether the observed differences are statistically meaningful.
[Sec. 4.3, Table 2 and Fig. 3] The value of NMULTI used for the benchmarks in Fig. 3 is not stated. Table 2 shows that Dslash performance on A100 varies from 2.244 to 3.439 TFLOPS (FP32, 32^4) depending on NMULTI, so the comparison in Fig. 3 is under-specified unless the NMULTI value is given and shown to be the same for the OpenCL and CUDA measurements.

minor comments (4)

[Sec. 4.3, Eq. (2)] Equation (2) as printed, min(peak performance / (arithmetic intensity × max bandwidth)), has incorrect dimensions; it should read min(peak performance, arithmetic intensity × max bandwidth). Please correct the formula.
[Sec. 4.2] The sentence 'For both Dslash and CG, the performance on one A100 is roughly 1.5 times that of a single gcd of MI250x' is too broad: Fig. 3 shows the ratio varying with lattice size and precision. Please state the range or give specific lattice sizes for which this factor holds.
[Table 1] Please specify which routine is meant by 'CG update' and how the sustained bandwidth was measured, since the same kernel could yield different values depending on data layout and vectorization.
[Fig. 3] The lattice-size labels on the horizontal axis, e.g. '16^3x32' and '32^2x64^2', appear without superscript formatting in the rendered figure; please use proper typography or a textual notation such as '16^3 x 32' for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims rest on direct benchmark measurements rather than on fitted inputs or self-cited premises.

full rationale

The paper's central claim is an empirical performance comparison, not a derived prediction. The OpenCL and CUDA results in Fig. 3 and Table 1 are direct measurements on A100 and MI250X hardware. The NMULTI variation in Table 2 and Fig. 4 is an optimization scan, not a parameter fitted to force a conclusion; it is presented as a search for the best multi-vector count. The roofline calculation in Eq. (2) and the arithmetic-intensity estimate are standard textbook formulas used to interpret the measurements, not to manufacture the headline result. The paper cites prior work for context (e.g., Ref. [7] for OpenCL lattice QCD), but the present benchmark comparison is self-contained and does not depend on any cited result. The only substantive weakness is internal consistency: Sec. 4.2 states that the CUDA backend's memory layout gives higher performance for larger lattices, while Sec. 5 concludes parity 'across a range of lattice sizes'; this is a scope/interpretation issue, not circularity. No step in the paper reduces by definition to its own inputs, and no load-bearing premise is imported from a self-citation. Score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical benchmark. It relies on standard lattice QCD formulas, hardware vendor specifications, and the representativeness of the measurements. No free parameters are fit to force the result, and no new theoretical entities are introduced. The only tuned quantity, NMULTI, is swept in Table 2 and does not enter the parity claim.

assumptions (3)

domain assumption Temporal link variables are gauge-fixed to the identity for all sites with t != 0, reducing link storage to the first two rows of each SU(3) matrix (Sec. 4.3).
The memory layout and arithmetic intensity quoted in Sec. 4.3 assume this gauge choice; the benchmarks therefore cover only gauge-fixed configurations.
standard math The third row of each SU(3) link is reconstructable on the fly from the stored first two rows using unitarity.
Valid property of SU(3) matrices, used to cut memory traffic; no error is introduced in exact arithmetic.
domain assumption The measured benchmark points are representative of production performance.
No error bars, run counts, or timing methodology are given in Sec. 4.2, so the parity claim depends on this unstated assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Portable Lattice QCD implementation based on OpenCL." pith.science (2026). https://pith.science/paper/GY7K6EW3

@misc{pith2026250203249,
  author       = {Pith},
  title        = {Pith review of: Portable Lattice QCD implementation based on OpenCL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GY7K6EW3}},
  note         = {Machine review of arXiv:2502.03249}
}

read the original abstract

The presence of GPU from different vendors demands the Lattice QCD codes to support multiple architectures. To this end, Open Computing Language (OpenCL) is one of the viable frameworks for writing a portable code. It is of interest to find out how the OpenCL implementation performs as compared to the code based on a dedicated programming interface such as CUDA for Nvidia GPUs. We have developed an OpenCL backend for our already existing code of the Wuppertal-Budapest collaboration. In this contribution, we show benchmarks of the most time consuming part of the numerical simulation, namely, the inversion of the Dirac operator. We present the code performance on the JUWELS and LUMI Supercomputers based on Nvidia and AMD graphics cards, respectively, and compare with the CUDA backend implementation.

Figures

Figures reproduced from arXiv: 2502.03249 by the authors.

**Figure 2.** Diagram illustrating the code hierarchy by providing some example classes 3 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Comparison of benchmarks for 𝐷/ and its inversion via CG in single and double precision. 4.3 Dslash Roofline The roofline plot gives a bound on attainable performance based on the device capabilities such as peak performance and memory bandwidths. The peak performance (Flop/s) of a kernel is defined as follows: Attainable performance = min ( peak performance arithmetic intensity × max bandwidth ) (2) We need to comp… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Performance of Dslash with varying number of vectors on one A100 and a single gcd of MI250x on a 324 lattice NMULTI A100 MI250x FP32 FP64 FP32 FP64 1 2.244 1.245 1.512 0.759 2 2.532 1.530 2.166 1.081 3 2.953 1.929 2.550 1.250 4 3.205 2.199 2.643 1.254 6 3.090 2.039 2.5…

**Figure 5.** Figure 5: Roofline plots for staggered Dslash operator with varying number of vectors on a 324 lattice on one A100 and a single gcd of MI250x. 5. Conclusions and Outlook Our portable OpenCL implementation for a single GPU achieves performance on par with CUDA across a range of l…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 6 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION blank.sep after.quote 'output.state := FUNCTION fin.entry output.state after.quoted.block = 'skip 'add.period if write newline FUNCTION new.block output.state before.all = 'skip output.state after.quote = after.quoted.block 'output.state := after.block 'output.state := if if FUNCTION new.sentence out...
[2]

G.I. Egri, Z. Fodor, C. Hoelbling, S.D. Katz, D. Nogradi and K.K. Szabo, Lattice QCD as a video game , https://doi.org/10.1016/j.cpc.2007.06.005 Comput. Phys. Commun. 177 (2007) 631 [ https://arxiv.org/abs/hep-lat/0611022 hep-lat/0611022 ]

work page Pith review arXiv 2007
[3]

QUDA collaboration, Solving Lattice QCD systems of equations using mixed precision solvers on GPUs , https://doi.org/10.1016/j.cpc.2010.05.002 Comput. Phys. Commun. 181 (2010) 1517 [ https://arxiv.org/abs/0911.3191 0911.3191 ]

arXiv 2010
[4]

Boyle, A

P. Boyle, A. Yamaguchi, G. Cossu and A. Portelli, Grid: A next generation data parallel C++ QCD library , https://arxiv.org/abs/1512.03487 1512.03487

arXiv
[5]

HotQCD collaboration, SIMULATeQCD: A simple multi-GPU lattice code for QCD calculations , https://doi.org/10.1016/j.cpc.2024.109164 Comput. Phys. Commun. 300 (2024) 109164 [ https://arxiv.org/abs/2306.01098 2306.01098 ]

work page arXiv 2024
[6]

Stone, D

J.E. Stone, D. Gohara and G. Shi, Opencl: A parallel programming standard for heterogeneous computing systems, https://doi.org/10.1109/MCSE.2010.69 Computing in Science & Engineering 12 (2010) 66

doi:10.1109/mcse.2010.69 2010
[7]

Khronos Group , ``Sycl 2020 specification.'' https://www.khronos.org/registry/SYCL/specs/sycl-2020/pdf/sycl-2020.pdf, 2021

work page 2020
[8]

M. Bach, V. Lindenstruth, O. Philipsen and C. Pinke, Lattice QCD based on OpenCL , https://doi.org/10.1016/j.cpc.2013.03.020 Comput. Phys. Commun. 184 (2013) 2042 [ https://arxiv.org/abs/1209.5942 1209.5942 ]

work page Pith review arXiv 2013

Show all 10 references

[9]

LUMI , `` LUMI Supercomputer .'' https://www.lumi-supercomputer.eu/, 2025

2025
[10]

Jülich Wizard for European Leadership Science (JUWELS) , `` JUWELS Supercomputer .'' https://www.fz-juelich.de/en/ias/jsc/systems/supercomputers/juwels, 2025

2025

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION blank.sep after.quote 'output.state := FUNCTION fin.entry output.state after.quoted.block = 'skip 'add.period if write newline FUNCTION new.block output.state before.all = 'skip output.state after.quote = after.quoted.block 'output.state := after.block 'output.state := if if FUNCTION new.sentence out...

[2] [2]

G.I. Egri, Z. Fodor, C. Hoelbling, S.D. Katz, D. Nogradi and K.K. Szabo, Lattice QCD as a video game , https://doi.org/10.1016/j.cpc.2007.06.005 Comput. Phys. Commun. 177 (2007) 631 [ https://arxiv.org/abs/hep-lat/0611022 hep-lat/0611022 ]

work page Pith review arXiv 2007

[3] [3]

QUDA collaboration, Solving Lattice QCD systems of equations using mixed precision solvers on GPUs , https://doi.org/10.1016/j.cpc.2010.05.002 Comput. Phys. Commun. 181 (2010) 1517 [ https://arxiv.org/abs/0911.3191 0911.3191 ]

arXiv 2010

[4] [4]

Boyle, A

P. Boyle, A. Yamaguchi, G. Cossu and A. Portelli, Grid: A next generation data parallel C++ QCD library , https://arxiv.org/abs/1512.03487 1512.03487

arXiv

[5] [5]

HotQCD collaboration, SIMULATeQCD: A simple multi-GPU lattice code for QCD calculations , https://doi.org/10.1016/j.cpc.2024.109164 Comput. Phys. Commun. 300 (2024) 109164 [ https://arxiv.org/abs/2306.01098 2306.01098 ]

work page arXiv 2024

[6] [6]

Stone, D

J.E. Stone, D. Gohara and G. Shi, Opencl: A parallel programming standard for heterogeneous computing systems, https://doi.org/10.1109/MCSE.2010.69 Computing in Science & Engineering 12 (2010) 66

doi:10.1109/mcse.2010.69 2010

[7] [7]

Khronos Group , ``Sycl 2020 specification.'' https://www.khronos.org/registry/SYCL/specs/sycl-2020/pdf/sycl-2020.pdf, 2021

work page 2020

[8] [8]

M. Bach, V. Lindenstruth, O. Philipsen and C. Pinke, Lattice QCD based on OpenCL , https://doi.org/10.1016/j.cpc.2013.03.020 Comput. Phys. Commun. 184 (2013) 2042 [ https://arxiv.org/abs/1209.5942 1209.5942 ]

work page Pith review arXiv 2013