Scaling SU(2) to 1000 GPUs using HiRep

Antonio Rago; Claudio Pica; Emiliano Molinaro; Erik Kjellgren; Sofie Martins

REVIEW 2 major objections 5 minor 1 cited by

Scaling SU(2) to 1000 GPUs using HiRep

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The HiRep lattice code reaches near-optimal weak scaling on AMD GPUs, with 679 TB/s on 1024 compute dies and 82% of theoretical peak bandwidth on a single die.

desk verdict Solid, honest scaling study of HiRep on AMD GPUs, but the headline numbers cover only the bare Wilson-Dirac kernel, not the production HMC path. read the letter →

arxiv 2411.18511 v2 pith:TL5P3MW4 submitted 2024-11-27 hep-lat

Sofie Martins , Erik Kjellgren , Emiliano Molinaro , Claudio Pica , Antonio Rago This is my paper · ORCID

classification hep-lat

keywords HiReplatticefieldtheoryWilsonfermionsGPUscalingweakAMDGPUshigherrepresentationscloverimprovement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a large-scale performance measurement of the HiRep lattice field theory code on AMD GPUs. For the Wilson-Dirac operator, the code reaches 82% of the theoretical memory bandwidth of a single MI250X GPU compute die, and its weak scaling stays essentially linear out to 1024 compute dies, where it sustains 679 TB/s. The result matters because HiRep supports Wilson fermions in higher representations of SU(N) with arbitrary numbers of colors and flavors, which is what beyond-standard-model lattice predictions need. The paper also shows the clover-improvement term adds little extra time and that strong scaling, unlike weak scaling, degrades once the inner computation becomes too fast to hide communication.

What carries the argument

The load-bearing mechanism is the kernel decomposition of the Wilson-Dirac operator into local inner-point computations and boundary computations. The inner kernel fills the GPU completely, while a separate POSIX thread performs send-buffer synchronization, asynchronous copies, and the boundary kernel, so communication happens behind the compute. Performance is measured as achieved memory bandwidth rather than floating-point throughput, because lattice Dirac operators are memory-bound; AMD and NVIDIA profiler traces are used to check that the communication and compute streams really do overlap.

What would settle it

Run a full Hybrid Monte Carlo update (or at least a preconditioned Wilson-Dirac inverter) at a fixed local lattice of $48^4$ on 1024 AMD MI250X compute dies and compare the aggregate bandwidth or time per trajectory with a single-die run; if the aggregate throughput falls clearly below the $679$ TB/s reported for the isolated operator, the near-optimal weak scaling claim does not carry over to production.

Watch

Extended reading notes

Core claim

The paper's central claim is that the GPU port of HiRep's Wilson-Dirac operator scales close to optimally on AMD hardware: one compute die (GCD) of an MI250X delivers 1.34 TB/s (82% of the 1.6384 TB/s theoretical peak), and 1024 compute dies together deliver 679 TB/s at a fixed local lattice of $48^4$ with all four dimensions parallelized. This is achieved by splitting the operator into an inner kernel that needs no communicated data and a boundary kernel that does, with all communication issued from a separate POSIX thread so that message passing overlaps with the heavy inner kernel. The reported figures are for the unimproved Wilson-Dirac operator; strong scaling on a fixed $128^4$ global lattice falls from 41% efficiency at 32 GCDs to 16% at 1024 GCDs, and the authors advise against local lattices smaller than $16^4$ for SU(2) gauge groups.

Load-bearing premise

The 82% and 679 TB/s numbers come from the bare Wilson-Dirac operator alone, and the paper assumes that this near-optimal behavior carries over to the full production code path, including inverters, integrators, even-odd preconditioning, and clover terms, which have different communication patterns and kernel mixes.

Editorial extensions

If this is right

Weak scaling is close to perfect up to 1024 GCDs, so fixed-size local volumes such as $48^4$ can be run on full machine partitions without paying an efficiency penalty.
A single GCD runs the Wilson-Dirac operator at 82% of theoretical memory bandwidth, meaning the CUDA-originated code is close to the hardware limit on AMD as well.
Strong scaling loses efficiency beyond about 128 GCDs, so users who want small local lattices should expect lower per-GPU bandwidth; local volumes below $16^4$ are advised against for SU(2).
The clover-improvement term adds only modest extra time, so improved actions, including exponential clover, can be used without sacrificing the scaling picture.
The scaling behaviour makes high-statistics, full-machine lattice calculations with higher-representation Wilson fermions practical for beyond-standard-model phenomenology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper benchmarks only the isolated Dirac operator and the clover term, so the 82% and 679 TB/s figures are not end-to-end Hybrid Monte Carlo performance; a fairer test would be measuring the full inverter or trajectory time.
A natural next benchmark is weak scaling at larger local volumes such as $64^4$, which should give the inner kernel more work to hide communication and may extend the near-perfect curve.
Because the observed overhead comes from boundary kernels and buffer synchronization, communication optimisations such as GPU-aware MPI could plausibly improve strong scaling without changing the lattice mathematics.
The $16^4$ local-volume guideline is established only for SU(2); for larger color groups the same threshold would need its own scaling test.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a good, honest engineering paper, and the scaling numbers are probably real—but they are for the bare Wilson-Dirac operator in isolation, not for the full HiRep production path. The paper says so if you read carefully, but the abstract and conclusion generalize to "the code," which overstates what is measured.

What's new: scaling measurements up to 1024 GCDs on AMD MI250X (LUMI-G), with a 679 TB/s aggregate bandwidth at 40% of theoretical peak in weak scaling, and 82% (1.34 TB/s) of peak on a single GCD. Also useful ROCm tracing showing kernel overlap of boundary and communication with the inner Dirac application. This extends the authors' earlier EuroPLEx2023 work from NVIDIA to AMD and to larger jobs. The code is on GitHub, which is a plus.

The paper is honest about its limitations. Section 2.1 states that the even-odd preconditioned Dirac operator used mainly in HMC cannot exploit the parallel streaming of even and odd parts. Section 2.2 benchmarks the clover term separately. Table 1 is explicitly for the unimproved Wilson-Dirac operator. So the concern about extrapolating to production is real—the 82% and 679 TB/s do not directly say anything about HMC performance, which uses inverters, global reductions, and a different communication pattern. The paper is careful to say "weak scaling only" and to disclose the strong scaling collapse to 16% at 1024 GCDs. That is good.

Soft spots: no error bars or repeated-run statistics, and timings come from profiler sampling with acknowledged variability. For an engineering performance claim, that is a reproducibility gap. A commit hash and raw data would help. Also, the comparison to theoretical peak should be taken with a grain of salt (they exceed it on A100 due to L2 reuse). That doesn't invalidate the results, but it means "percent of peak" is a loose metric.

Bottom line: this is a useful proceedings paper for lattice practitioners who run on AMD GPUs, and the authors know their code base. The central scaling claim is supported for the Dirac kernel, and the paper is transparent about the gap between that and full production. It deserves a serious referee if submitted as a full paper, with requests for reproducibility artifacts and a clearer statement that the headline numbers are kernel-level, not end-to-end HiRep.

Recommendation: engage with it; the work is sound and the limitations are disclosed.

Referee Report

2 major / 5 minor

Summary. This proceedings paper reports GPU porting progress for the HiRep lattice field theory code, focusing on scaling of the Wilson-Dirac operator on AMD MI250X GPUs (LUMI-G). The authors present profiler traces of kernel execution, benchmark the unimproved Wilson-Dirac operator and the clover term, and report weak and strong scaling up to 1024 GCDs. They claim a single-GCD memory bandwidth of 1.34 TB/s (about 82% of the theoretical peak of 1.6384 TB/s) and a peak aggregate bandwidth of 679 TB/s at 1024 GCDs, with weak scaling staying at about 40% of theoretical peak beyond 128 GCDs. The paper also compares performance against NVIDIA A100 and H100 results.

Significance. If the scaling result holds for the production code path, this is a useful data point for the lattice community: it demonstrates that a CUDA-originated code can reach near-optimal single-GCD bandwidth and flat weak scaling on AMD GPUs at scale for a Wilson-Dirac kernel. The paper is honest about strong-scaling degradation (16% efficiency at 1024 GCDs, local lattice 128^4) and makes the code available open-source. The main strength is that the measurements are direct and compared against the theoretical device peak. The main weakness is that the benchmark covers only the unimproved Wilson-Dirac operator in isolation, and the manuscript generalizes the conclusions to the full production code used in HMC, which includes even-odd preconditioning, clover terms, inverters, and integrators with different communication and reduction patterns.

major comments (2)

[Section 3 / Table 1 / Conclusion] The scaling data and headline bandwidths in Section 3 and Table 1 are explicitly for the unimproved Wilson-Dirac operator in isolation (Table 1 caption), but the abstract and conclusion generalize these results to 'the code' and to enabling BSM production runs. The paper itself notes in Section 2.1 that the even-odd preconditioned Dirac operator, used mainly in the HMC, cannot use the same overlapping of even/odd inner-point work, and Section 2.2 treats the clover term only as a separate application. Inverters and integrators introduce global reductions (dot products, all-reduces) and different communication patterns that are absent from this kernel-only benchmark. The 82% single-GCD efficiency and the flat ~40%-of-peak weak-scaling plateau are therefore not direct evidence that the full production HiRep path scales near-optimally to 1024 GCDs. Please either restrict the claims to the Dirac-operator kernel or add measurements of the even-odd preconditioned operator and at least one representative inverter/solver path.
[Section 3 / Table 1] The scaling numbers in Table 1 and the cited peak bandwidth values (679 TB/s and 1.34 TB/s) appear to come from single profiler runs; the paper acknowledges 'inevitable variability from one Dirac execution to another' and profiler sampling precision. Without repeated-run statistics or error estimates, the reader cannot judge whether the small efficiency differences in the weak-scaling column (49%, 44%, 40%, 41%, 40%) are significant or whether the plateau is truly flat as claimed. Please report the run-to-run spread for representative GCD counts (e.g., 16, 128, 1024) or explicitly state that these are single measurements and characterize the associated uncertainty.

minor comments (5)

[Sections 2.2 and 3 (typography)] The text renders local lattice sizes as '484' and '164'; these should be 48^4 and 16^4 with proper superscripts, since the meaning is otherwise unclear.
[Table 1 caption] The '%' column in Table 1 is not defined in the caption; specify that it is the percentage of the theoretical aggregate peak memory bandwidth of the GCDs.
[Section 1] The claim that HiRep is 'the highest performing option' for Wilson fermions in higher representations is unsupported by a comparative reference or benchmark; consider softening this or providing a citation.
[Section 2.1.1] The discussion of blocking versus non-blocking communications concludes that the choice 'needs to be tested on an individual basis,' but no such test is reported; briefly state which mode is used in the presented benchmarks.
[Figure 1 caption] The caption notes that the right ROCm profile shows only one of the four MPI processes; consider adding that this trace therefore does not display the node-level communication pattern.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the performance claims are direct measurements against external hardware peak-bandwidth specifications, with no fitted parameter or self-citation used as load-bearing evidence.

full rationale

This is a measurement and porting-progress paper, not a derivation. The central quantitative claims — 82% (1.34 TB/s) of a single MI250X GCD's theoretical peak bandwidth and 679 TB/s across 1024 GCDs — are obtained from profiler and benchmark runs reported in Section 3, Figure 4, and Table 1. The comparison point, 1.6384 TB/s, is the vendor-specified theoretical peak bandwidth of the MI250X GCD, an external hardware specification that does not depend on any outcome of the paper. There is no fitted parameter that is later renamed as a prediction, and no equation in the paper transforms an input assumption into the claimed scaling result. The only self-citation, reference [4] (Martins et al., EuroPLEx2023), describes the prior CUDA-oriented GPU port of HiRep; it is used as background for the code's origin, not to justify the new AMD scaling numbers, and the new measurements are presented directly. A scope limitation does exist and is explicitly acknowledged in the paper itself: Table 1's caption states that the scaling data are for the unimproved Wilson-Dirac operator, and Section 2.1 notes that even-odd preconditioned Dirac operator, used mainly in HMC, cannot exploit parallel streaming of even and odd inner-point work. This means the 82% and 679 TB/s figures may not directly describe full production HMC paths involving inverters, clover terms, and preconditioning. However, that is a correctness or generalization risk, not circular reasoning: the benchmark is what it claims to be, and the paper does not derive the production-path efficiency from the kernel-only measurement by construction. No circular step can be exhibited with a specific equation, definition, or self-citation chain, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is an engineering benchmark paper with no fitted constants and no new theoretical constructs. The central claims rest on hardware specifications and the assumption that kernel-level bandwidth measurements reflect production performance. The axioms listed are standard domain assumptions, not free parameters or invented entities.

assumptions (3)

domain assumption The theoretical peak memory bandwidth of the MI250X GCD is 1.6384 TB/s.
The paper uses this hardware specification to compute the 82% efficiency on a single GCD. It is a manufacturer-provided number, not derived in the paper.
domain assumption Memory bandwidth is the correct performance metric for lattice simulations of this type.
The paper states lattice simulations are memory-bound. This is a standard assumption for Wilson-Dirac kernels, but it is not validated for the full HMC chain.
domain assumption The profiler traces shown are representative of typical execution.
The authors explicitly note variability and state the shown example is typical and not the best found. All timing comparisons rely on this representativeness.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scaling SU(2) to 1000 GPUs using HiRep." pith.science (2026). https://pith.science/paper/TL5P3MW4

@misc{pith2026241118511,
  author       = {Pith},
  title        = {Pith review of: Scaling SU(2) to 1000 GPUs using HiRep},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TL5P3MW4}},
  note         = {Machine review of arXiv:2411.18511}
}

read the original abstract

HiRep allows flexible simulations of higher representations of Wilson Fermions with various actions and gauge groups and a range of inverters and integrators. This is particularly important for enabling evaluations of observables relevant to phenomenological inputs for Beyond-the-Standard-Model physics from lattice field theory. We present progress on the GPU porting of available features, especially in terms of scaling to large jobs on AMD GPUs.

Figures

Figures reproduced from arXiv: 2411.18511 by the authors.

**Figure 1.** ROCm profiles for single and multiple Dirac operator executions on LUMI-G in a 4 GCD/single-node job on a 484 local lattice parallelized in two dimensions. The vertical direction is arranged first by kernel type and then by queue ID, showing the parallelization of different kernel types in different streams. In this case, the profile on the right shows only one of the four MPI processes. For comparison, we show in … view at source ↗

**Figure 2.** Profile of repeated Dirac operator applications on four NVIDIA H100 GPUs using a 484 local lattice parallelized in two dimensions. The tracing plot shows on the left side a single application of the Dirac operator as part of this profile and on the right side the repeated Dirac executions on all four MPI processes. The vertical direction is arranged first by process, then kernel type, and finally by stream ID. In co… view at source ↗

**Figure 3.** A single Dirac operator application including the diagonal clover term executed on AMD MI250X GCDs on LUMI-G using a 484 local lattice with two parallelized dimensions, both for regular clover improvement (left) and exponential clover improvement (right). The vertical direction is organized first by kernel type and second by queue ID. synchronization. If the inner Dirac kernel execution is becoming too fast to mask … view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Weak and strong scaling from 16 to 1024 GCDs on LUMI-G with all four dimensions parallelized on a local lattice of 484 (weak scaling) and global lattice of 1284 (strong scaling) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The singlet scalar state in a chiral ensemble in $SU(2)$ with two fundamental flavours
hep-lat 2025-02 conditional novelty 4.0 of 10

The first lattice measurement of the sigma mass with exponential clover fermions in SU(2) with two fundamental flavours finds the effective mass agrees with the two-pion threshold, 2 m_PS.

Reference graph

Works this paper leans on

7 extracted references · 5 canonical work pages · cited by 1 Pith paper

[1]

Boyle, Guido Cossu, Azusa Yamaguchi, and Antonin Portelli

Peter A. Boyle, Guido Cossu, Azusa Yamaguchi, and Antonin Portelli. Grid: A next generation data parallel C++ QCD library.PoS, LATTICE2015:023, 2016

work page 2016
[2]

Symplectic lattice gauge theories in the grid framework: Approaching the conformal window.Phys

Ed Bennett et al. Symplectic lattice gauge theories in the grid framework: Approaching the conformal window.Phys. Rev. D, 108(9):094508, 2023

work page 2023
[3]

Higher representations on the lattice: Numerical simulations

Luigi Del Debbio, Agostino Patella, and Claudio Pica. Higher representations on the lattice: Numerical simulations. SU(2) with adjoint fermions.Phys. Rev. D, 81:094503, 2010

2010
[4]

GPU- accelerated Higher Representations of Wilson Fermions with HiRep.PoS, EuroPLEx2023:035, 2024

Sofie Martins, Erik Kjellgren, Emiliano Molinaro, Claudio Pica, and Antonio Rago. GPU- accelerated Higher Representations of Wilson Fermions with HiRep.PoS, EuroPLEx2023:035, 2024

work page 2024
[5]

Sheikholeslami and R

B. Sheikholeslami and R. Wohlert. Improved Continuum Limit Lattice Action for QCD with Wilson Fermions.Nucl. Phys. B, 259:572, 1985

work page 1985
[6]

Master-field simulations of O(𝑎)-improved lattice QCD: Algorithms, stability and exactness.Comput

Anthony Francis, Patrick Fritzsch, Martin Lüscher, and Antonio Rago. Master-field simulations of O(𝑎)-improved lattice QCD: Algorithms, stability and exactness.Comput. Phys. Commun., 255:107355, 2020

2020
[7]

Speeding up the hybrid Monte Carlo algorithm for dynamical fermions.Phys

Martin Hasenbusch. Speeding up the hybrid Monte Carlo algorithm for dynamical fermions.Phys. Lett. B, 519:177–182, 2001. 8

work page 2001

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Boyle, Guido Cossu, Azusa Yamaguchi, and Antonin Portelli

Peter A. Boyle, Guido Cossu, Azusa Yamaguchi, and Antonin Portelli. Grid: A next generation data parallel C++ QCD library.PoS, LATTICE2015:023, 2016

work page 2016

[2] [2]

Symplectic lattice gauge theories in the grid framework: Approaching the conformal window.Phys

Ed Bennett et al. Symplectic lattice gauge theories in the grid framework: Approaching the conformal window.Phys. Rev. D, 108(9):094508, 2023

work page 2023

[3] [3]

Higher representations on the lattice: Numerical simulations

Luigi Del Debbio, Agostino Patella, and Claudio Pica. Higher representations on the lattice: Numerical simulations. SU(2) with adjoint fermions.Phys. Rev. D, 81:094503, 2010

2010

[4] [4]

GPU- accelerated Higher Representations of Wilson Fermions with HiRep.PoS, EuroPLEx2023:035, 2024

Sofie Martins, Erik Kjellgren, Emiliano Molinaro, Claudio Pica, and Antonio Rago. GPU- accelerated Higher Representations of Wilson Fermions with HiRep.PoS, EuroPLEx2023:035, 2024

work page 2024

[5] [5]

Sheikholeslami and R

B. Sheikholeslami and R. Wohlert. Improved Continuum Limit Lattice Action for QCD with Wilson Fermions.Nucl. Phys. B, 259:572, 1985

work page 1985

[6] [6]

Master-field simulations of O(𝑎)-improved lattice QCD: Algorithms, stability and exactness.Comput

Anthony Francis, Patrick Fritzsch, Martin Lüscher, and Antonio Rago. Master-field simulations of O(𝑎)-improved lattice QCD: Algorithms, stability and exactness.Comput. Phys. Commun., 255:107355, 2020

2020

[7] [7]

Speeding up the hybrid Monte Carlo algorithm for dynamical fermions.Phys

Martin Hasenbusch. Speeding up the hybrid Monte Carlo algorithm for dynamical fermions.Phys. Lett. B, 519:177–182, 2001. 8

work page 2001