REVIEW 3 major objections 6 minor 1 cited by
TempLat lets you write lattice field equations almost as math and compiles them into fast kernels that scale on CPUs and GPUs in any dimension.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 05:59 UTC pith:ZNMRP37G
load-bearing objection Solid open-source lattice engine with real GPU CosmoLattice path and honest HILA/cuFFTMp baselines; “excellent multi-node GPU scaling” is real in the tested regime but geometry-limited at high rank. the 3 major comments →
TempLat: a versatile C++ engine for lattice field theories
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
TempLat is a versatile, dimension-agnostic C++ lattice engine: expression templates map mathematical field equations almost verbatim into C++ while fusing them into highly optimized device kernels, and a Kokkos-backed host/device model delivers performance-portable classical-statistical and related lattice simulations with excellent strong and weak scaling on CPUs and on NVIDIA and AMD GPUs, accompanied by the standalone arbitrary-dimension parallel DFT library ParaFaFT.
What carries the argument
Expression templates over lattice fields: an abstract expression tree is built at compile time, pruned (e.g. by ZeroType/OneType), then fused into a single kernel and dispatched via a device iteration/reduce API, with automatic halo exchanges and DFTs inferred from the tree.
Load-bearing premise
The claim of excellent scaling at large GPU counts assumes that the pencil domain split and current halo-exchange strategy stay adequate even when each device holds little volume and network traffic dominates.
What would settle it
Repeat the paper’s SU(2) and scalar leapfrog weak/strong scaling on the same A100-class nodes but with a full multi-dimensional block decomposition and non-blocking halo overlap; if per-site time then rises sharply or falls behind a block-decomposed baseline at high GPU count while kernels stay flat, the scaling claim is geometry-limited as the paper’s own comparison already hints.
If this is right
- The same source can target CPU clusters and NVIDIA/AMD GPUs without rewriting lattice physics expressions.
- Cosmology and other classical lattice apps built on this engine gain a GPU path with automatic FFTs and ghost updates.
- New composite types (e.g. further gauge groups) can be added by supplying their algebra, reusing the same evaluation and device layer.
- ParaFaFT fills a gap for distributed, arbitrary-dimension DFTs across the supported hardware backends.
- Cross-checks against an independent portable lattice engine become practical on identical workloads.
Where Pith is reading between the lines
- Decoupling real-space domain layout from FFT layout, as the authors flag, is likely the highest-leverage fix when FFTs or high-rank halos dominate.
- The compile-time algebra and device split are a natural base for a Monte Carlo or semiclassical-fermion layer once those algebras land.
- Accessibility for non-specialists will hinge as much on docs and model packs as on kernel speed once GPUs are routine.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents TempLat, an open-source C++ lattice field theory framework built on expression templates with a Kokkos-based device abstraction (CPU via OpenMP/pthreads, NVIDIA via CUDA, AMD via HIP), together with ParaFaFT, a standalone distributed DFT library supporting arbitrary dimensions via pencil decomposition. The interface and architecture are described in §§2–3; §4 reports strong and weak scaling for an SU(2) Yang–Mills leapfrog benchmark on the Otus (EPYC) and Noctua2 (A100) clusters, including per-phase (halo/kick/drift) timing breakdowns, and a cross-validation against HILA on a scalar φ⁴ kernel. Appendix A benchmarks ParaFaFT against FFTW3-MPI and cuFFTMp. The manuscript is clearly written, the code and tests are public, and the limitations (pencil-decomposition halo growth, GPU-count-dependent competitiveness vs HILA and cuFFTMp) are reported frankly. My concerns are local: the headline scaling claim needs scoping, benchmark reproducibility details are missing, and one scaling argument in §4 is asserted rather than shown.
Significance. If the results hold as reported — and there is little reason to doubt the measurements — TempLat is a useful contribution: an open-source, expression-template lattice engine with Kokkos-based CPU/NVIDIA/AMD portability, plus ParaFaFT, which to my knowledge is the only distributed DFT library combining arbitrary dimensionality with cross-vendor performance portability. Specific strengths worth naming: (i) both codes are open source with unit-test coverage and reproducible counter-based RNG; (ii) the performance claims are substantiated with strong/weak scaling on two real clusters with per-phase (halo/kick/drift) breakdowns; (iii) an independent cross-check against HILA on an identical φ⁴ workload is rare and valuable practice; (iv) limitations (halo cost growing ~P^{1/2} vs the ideal P^{1/6}; ParaFaFT trailing cuFFTMp at small GPU counts) are disclosed and analyzed rather than hidden. The gap between pencil and block decompositions is honestly identified as the main scaling risk.
major comments (3)
- [Abstract and §5, vs. Fig. 6 and Appendix A] The abstract and §5 state 'excellent strong and weak scaling on both CPUs and GPUs' without qualification, but the paper's own results bound the regime in which this holds: Fig. 6 (left) shows HILA overtaking TempLat in strong scaling at high GPU counts (attributed to HILA's d-dimensional block vs TempLat's pencil decomposition), and Fig. 7 (right) shows ParaFaFT behind cuFFTMp at smaller GPU lattices. The body text discusses these limitations honestly and well; the problem is only that the headline claim does not reflect them. Please qualify the abstract/conclusion claim to the demonstrated regime (workload-rich runs, up to 32 A100s / 3072 CPU ranks) rather than asserting scaling 'excellence' generally.
- [§4 and Appendix A] The performance claims of §4 are the paper's central deliverable, but the manuscript gives no compiler versions, Kokkos/CUDA/HIP/MPI versions, or build flags, and does not state whether the benchmark harness (run scripts, job configurations) is shipped in the GitHub repositories. For a software/performance paper, these are needed for the results to be independently checkable. Relatedly, the abstract claims AMD GPU (HIP) support as part of performance portability, yet no AMD benchmark or even a validation result is shown — all GPU data are A100. Please either add an AMD data point or state explicitly that HIP support is validated by unit tests only, with performance numbers deferred.
- [§4, Fig. 5 discussion] The text derives an ideal weak-scaling halo cost per site of O(P^{1/6}) but reports measured growth 'closer to O(P^{1/2})', attributed to 'non-square decompositions and additional synchronization overhead'. This is a large gap asserted without quantitative support: over only 8 points spanning 1–32 GPUs (with node-boundary effects at 4), a P^{1/2} fit is weakly constrained, and the residual communication model in the HILA discussion (4N²P^{1/2} at fixed N) is not obviously consistent with the weak-scaling per-site statement. Please either provide a quantitative decomposition (e.g., measured surface-to-volume scaling for the actual rank grids used, or a sync-vs-transfer split) or present the P^{1/2} statement as an empirical observation over the tested range only.
minor comments (6)
- [§2, Listing 1 and Eq. (1)] Listing 1: the noise is rescaled by 1/sqrt(dt) 'to get the correct dt->0 limit', but the fluctuation-dissipation normalization (factor of 2T in <eta eta>) is silently absorbed. One sentence on the assumed temperature/normalization would help readers reuse the example.
- [Fig. 3 and Fig. 4] The labels '78%', '75%', '61%' (and Fig. 4's 96/87/82/78%) appear to annotate parallel efficiency at selected points, but this is never stated in the captions. Please define them.
- [Listing 2] 'SessionGuard' appears without explanation; a half-sentence (RAII initialization/finalization of MPI and Kokkos) would suffice.
- [References] Reference [21] is a bare GitHub URL dated 2026. If a citable HILA publication or release note exists it should be used; at minimum cite a tagged release/commit so the comparison version is pinned. The HILA compiler/flags used for Fig. 6 should also be stated.
- [Appendix A] Fig. 7, right panel: 'we equate one CPU node with a single GPU' for the shared x-axis is easy to misread as a performance equivalence claim; please clarify in the caption that it is only an axis alignment.
- [§2, Table 1] Table 1 describes U1Field as an alias of ComplexField; it would be worth one sentence noting whether U(1) assignments are projected to unit modulus or left to the user, since this affects physical correctness in gauge applications.
Circularity Check
No circularity: software/benchmark paper with measured timings and external baselines, not definitional or fitted predictions.
full rationale
TempLat is a methods/software paper. Its central claims are (i) an expression-template + Kokkos device abstraction that compiles mathematical lattice expressions to portable kernels, and (ii) measured strong/weak scaling on CPUs and GPUs, plus ParaFaFT DFT scaling versus FFTW-MPI and cuFFTMp. Performance numbers are wall-clock timings of fixed leapfrog workloads (SU(2) Yang–Mills and φ⁴), not quantities derived from fitted parameters or from definitions that already encode the result. Self-references to CosmoLattice establish lineage (TempLat as its former engine) and do not load-bear the scaling claims. Cross-checks against independent HILA and vendor FFT libraries are external evidence. Domain decomposition inherited from ParaFaFT affects absolute halo cost and is disclosed analytically (O(P^{1/6}) ideal vs measured growth), but that is a limitation of the design choice, not a circular reduction of a claimed derivation. No self-definitional loop, fitted-input-as-prediction, uniqueness import, or renamed empirical law appears in the derivation chain.
Axiom & Free-Parameter Ledger
axioms (5)
- domain assumption Kokkos correctly and performantly maps the TempLat device iteration/memory API onto CUDA, HIP, OpenMP, pthreads, and serial backends.
- ad hoc to paper Pencil domain decomposition (keep one axis local, segment d−1) is an acceptable default for both real-space halo exchange and distributed DFTs across the reported rank counts.
- domain assumption Classical leapfrog SU(2) Yang–Mills and φ⁴ kernels with one ghost layer and double precision are representative workloads for claiming general lattice-engine efficiency.
- domain assumption Expression-template full-tree fusion at assignment yields a single device kernel without harmful register/spilling regressions relative to hand-written kernels on target GPUs.
- standard math Standard lattice finite differences, periodic boundaries, and MPI halo semantics correctly implement the continuum-intended operators used in the examples.
invented entities (2)
-
TempLat framework (expression-template lattice algebra + device abstraction)
independent evidence
-
ParaFaFT parallel DFT library
independent evidence
read the original abstract
We present TempLat, a C++ framework for lattice field theory simulations in arbitrary dimensions. Its symbolic language, built on expression templates, translates mathematical expressions almost verbatim into C++ while compiling to highly optimized kernels, in contexts ranging from classical-statistical to Monte Carlo simulations. TempLat is performance-portable: building on Kokkos, it supports large CPU clusters, as well as NVIDIA and AMD GPUs. Its abstraction of hardware into a device concept makes TempLat extensible to future architectures. We demonstrate excellent strong and weak scaling on both CPUs and GPUs. We also present ParaFaFT, a standalone parallel discrete Fourier transform library supporting arbitrary dimensions on all of the above hardware, and benchmark its scaling. Both are open source and available on GitHub, and power a new release of CosmoLattice, a widely used early-universe simulation library now available on GPUs.
Figures
Forward citations
Cited by 1 Pith paper
-
The art of simulating the early Universe. Part III: Scalar-Gauge-Fluid Dynamics
Detailed continuum-to-lattice schemes are given for perfect/imperfect fluids alone or coupled to scalars/gauges in FLRW, enabling self-consistent CosmoLattice simulations of early-Universe plasma dynamics and GWs.
Reference graph
Works this paper leans on
-
[1]
G. I. Egri, Z. Fodor, C. Hoelbling, S. D. Katz, D. Nogradi, K. K. Szabo, Lattice QCD as a video game, Comput. Phys. Commun. 177 (2007)631–639.arXiv:hep-lat/0611022,doi:10.1016/j.cpc.2007.06. 005
Pith/arXiv arXiv 2007
-
[2]
M. A. Clark, R. Babich, K. Barros, R. C. Brower, C. Rebbi, Solving Lattice QCD systems of equations using mixed precision solvers on GPUs, Comput. Phys. Commun. 181 (2010) 1517–1528.arXiv: 0911.3191,doi:10.1016/j.cpc.2010.05.002
Pith/arXiv arXiv 2010
-
[3]
Lüescher, S
M. Lüescher, S. Schaefer, Openqcd,https://luscher.web.cern.ch/ luscher/openQCD/
-
[4]
Sexty, Simulating full QCD at nonzero density using the complex Langevin equation, Phys
D. Sexty, Simulating full QCD at nonzero density using the complex Langevin equation, Phys. Lett. B 729 (2014) 108–111.arXiv:1307. 7748,doi:10.1016/j.physletb.2014.01.019
-
[5]
arXiv:1512.03487,doi:10.22323/1.251.0023
P.Boyle,G.Cossu,A.Yamaguchi,A.Portelli,Grid:Anextgeneration data parallel C++ QCD library, PoS LATTICE2015 (2016) 023. arXiv:1512.03487,doi:10.22323/1.251.0023
Pith/arXiv arXiv 2016
-
[6]
L. Altenkort, D. Bollweg, D. A. Clarke, O. Kaczmarek, L. Mazur, C. Schmidt, P. Scior, H.-T. Shu, HotQCD on multi-GPU Systems, PoS LATTICE2021 (2022) 196.arXiv:2111.10354,doi:10.22323/1. 396.0196
Pith/arXiv arXiv 2022
-
[7]
Mazur, et al., SIMULATeQCD: A simple multi-GPU lattice code for QCD calculations, Comput
L. Mazur, et al., SIMULATeQCD: A simple multi-GPU lattice code for QCD calculations, Comput. Phys. Commun. 300 (2024) 109164. arXiv:2306.01098,doi:10.1016/j.cpc.2024.109164
Pith/arXiv arXiv 2024
-
[8]
S. Laizet, N. Li, Incompact3d: A powerful tool to tackle turbu- lence problems with up to o(105) computational cores, International Journal for Numerical Methods in Fluids 67 (11) (2011) 1735– 1757.arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/fld. 2480,doi:https://doi.org/10.1002/fld.2480. URLhttps://onlinelibrary.wiley.com/doi/abs/10.1002/fld.2480
work page doi:10.1002/fld 2011
-
[9]
A.Brandenburg,etal.,ThePencilCode,amodularMPIcodeforpar- tial differential equations and particles: multipurpose and multiuser- maintained, J. Open Source Softw. 6 (58) (2021) 2807.arXiv:2009. 08231,doi:10.21105/joss.02807
-
[10]
J. M. Stone, K. Tomida, C. J. White, K. G. Felker, The athena++ adaptive mesh refinement framework: Design and magnetohydrody- namic solvers, The Astrophysical Journal Supplement Series 249 (1) (2020) 4.doi:10.3847/1538-4365/ab929b. URLhttps://doi.org/10.3847%2F1538-4365%2Fab929b
-
[11]
F. Bruckner, S. Koraltan, C. Abert, D. Suess, magnum.np – a py- torchbasedgpuenhancedfinitedifferencemicromagneticsimulation framework for high level development and inverse design (2023). arXiv:2302.08843. URLhttps://arxiv.org/abs/2302.08843
Pith/arXiv arXiv 2023
-
[12]
A. Vansteenkiste, J. Leliaert, M. Dvornik, M. Helsen, F. Garcia-Sanchez, B. Van Waeyenberge, The design and verification of mumax3, AIP Advances 4 (10) (2014) 107133. arXiv:https://pubs.aip.org/aip/adv/article-pdf/doi/10.1063/1. 4899186/12878560/107133_1_online.pdf,doi:10.1063/1.4899186. URLhttps://doi.org/10.1063/1.4899186
doi:10.1063/1 2014
-
[13]
A. Piñeiro Orioli, K. Boguslavski, J. Berges, Universal self-similar dynamics of relativistic and nonrelativistic field theories near non- thermal fixed points, Phys. Rev. D 92 (2) (2015) 025041.arXiv: 1503.02498,doi:10.1103/PhysRevD.92.025041
Pith/arXiv arXiv 2015
-
[14]
D. G. Figueroa, A. Florio, F. Torrenti, W. Valkenburg, The art of simulating the early Universe – Part I: Integration techniques and canonicalcases,JCAP04(2021)035.arXiv:2006.15122,doi:10.1088/ 1475-7516/2021/04/035
Pith/arXiv arXiv 2021
-
[15]
J. Baeza-Ballesteros, D. G. Figueroa, A. Florio, J. Lizarraga, N. Loayza, K. Marschall, T. Opferkuch, B. A. Stefanek, F. Torrentí, A. Urio, The art of simulating the early Universe. Part II. Non- canonical cases & gravitational waves (12 2025).arXiv:2512.15627
arXiv 2025
-
[16]
D.G.Figueroa,A.Florio,F.Torrenti,W.Valkenburg,CosmoLattice: A modern code for lattice simulations of scalar and gauge field dynamics in an expanding universe, Comput. Phys. Commun. 283 (2023) 108586.arXiv:2102.01031,doi:10.1016/j.cpc.2022.108586
Pith/arXiv arXiv 2023
-
[17]
D. G. Figueroa, A. Florio, F. Torrenti, Present and future ofosmo attice, Rept. Prog. Phys. 87 (9) (2024) 094901.arXiv:2312.15056, doi:10.1088/1361-6633/ad616a
Pith/arXiv arXiv 2024
-
[18]
H. C. Edwards, C. R. Trott, D. Sunderland, Kokkos: Enabling manycore performance portability through polymorphic memory access patterns, Journal of Parallel and Distributed Computing 74 (12) (2014) 3202 – 3216, domain-Specific Languages and High-Level Frameworks for High-Performance Computing. doi:https://doi.org/10.1016/j.jpdc.2014.07.003. URLhttp://www....
-
[19]
C. Trott, L. Berger-Vergiat, D. Poliakoff, S. Rajamanickam, D.Lebrun-Grandie,J.Madsen,N.AlAwar,M.Gligoric,G.Shipman, G.Womeldorff,Thekokkosecosystem:Comprehensiveperformance portability for high performance computing, Computing in Science Engineering 23 (5) (2021) 10–18.doi:10.1109/MCSE.2021.3098509
arXiv 2021
-
[20]
C. R. Trott, D. Lebrun-Grandié, D. Arndt, J. Ciesko, V. Dang, N. Ellingwood, R. Gayatri, E. Harvey, D. S. Hollman, D. Ibanez, N. Liber, J. Madsen, J. Miles, D. Poliakoff, A. Powell, S. Rajaman- ickam, M. Simberg, D. Sunderland, B. Turcksin, J. Wilke, Kokkos 3: Programming model extensions for the exascale era, IEEE Trans- actions on Parallel and Distribut...
arXiv 2022
-
[21]
HILA, Hila,https://github.com/CFT-HY/HILA(2026)
2026
-
[22]
J. Correia, M. Hindmarsh, J. Lizarraga, A. Lopez-Eiguren, K. Rum- mukainen, J. Urrestilla, Scaling density of axion strings in terasite simulations, Phys. Rev. D 111 (6) (2025) 063532.arXiv:2410.18064, doi:10.1103/PhysRevD.111.063532
Pith/arXiv arXiv 2025
-
[23]
Rummukainen, private communication (2026)
K. Rummukainen, private communication (2026)
2026
-
[24]
URLhttps://doi.org/10.1145/2063384.2063405
J.K.Salmon,M.A.Moraes,R.O.Dror,D.E.Shaw,Parallelrandom numbers: as easy as 1, 2, 3, in: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’11, Association for Computing Machinery, New York, NY, USA, 2011.doi:10.1145/2063384.2063405. URLhttps://doi.org/10.1145/2063384.2063405
arXiv 2011
-
[25]
A.Ayala,S.Tomov,A.Haidar,J.Dongarra,heffte:Highlyefficientfft for exascale, in: V. V. Krzhizhanovskaya, G. Závodszky, M. H. Lees, J. J. Dongarra, P. M. A. Sloot, S. Brissos, J. Teixeira (Eds.), Com- putational Science – ICCS 2020, Springer International Publishing, Cham, 2020, pp. 262–275
2020
-
[26]
URLhttp://arxiv.org/abs/1804.09536
L.Dalcín,M.Mortensen,D.E.Keyes,Fastparallelmultidimensional FFT using advanced MPI, CoRR abs/1804.09536 (2018).arXiv: 1804.09536. URLhttp://arxiv.org/abs/1804.09536
Pith/arXiv arXiv 2018
-
[27]
Program Generation, Optimization, and Platform Adaptation
M. Frigo, S. G. Johnson, The design and implementation of FFTW3, Proceedings of the IEEE 93 (2) (2005) 216–231, special issue on “Program Generation, Optimization, and Platform Adaptation”
2005
-
[28]
NVIDIA Corporation, cufft,https://developer.nvidia.com/cufft, nVIDIA CUDA Toolkit (2024)
2024
-
[29]
AMD Corporation, hipfft,https://github.com/ROCm/hipFFT, aMD ROCm HIP FFT Library (2024)
2024
-
[30]
S. Ehtesabi, M. Hossain, T. Kenter, A. Krawinkel, H. Nitsche, L. Os- termann, C. Plessl, H. Riebler, S. Rohde, R. Schade, M. Schwarz, J. Simon, N. Winnwa, A. Wiens, X. Wu, Otus Supercomputer, PC2 Technical Report PC2TR-2025-1, Paderborn Center for Par- allel Computing (PC2), Paderborn, Germany, available at pc2.uni- paderborn.de/research/pc2-technical-rep...
-
[31]
Bauer, T
C. Bauer, T. Kenter, M. Lass, L. Mazur, M. Meyer, H. Nitsche, H. Riebler, R. Schade, M. Schwarz, N. Winnwa, A. Wiens, X. Wu, C. Plessl, J. Simon, Noctua 2 supercomputer, Journal of large-scale research facilities JLSRF 9 (2024).doi:https://doi.org/10.17815/ jlsrf-8-187
2024
-
[32]
G.Aarts,J.Smit,Realtimedynamicswithfermionsonalattice,Nucl. Phys. B 555 (1999) 355–394.arXiv:hep-ph/9812413,doi:10.1016/ S0550-3213(99)00320-X. A. Florio and F. R. Sattler:Preprint submitted to ElsevierPage 12 of 12
Pith/arXiv arXiv 1999
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.