{"id":"ca2ca17f-817c-47e7-b848-5312d2f2ff2d","arxiv_id":"2411.18511","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"HiRep, a lattice field theory code, scales to 1024 AMD GPUs, reaching 82% of peak bandwidth on a single GPU and 679 TB/s aggregate bandwidth.","lead":"The HiRep lattice simulation code, originally written for NVIDIA GPUs, now also runs efficiently on AMD GPUs and can scale to 1024 of them. This makes large-scale simulations of beyond-Standard-Model physics feasible on European supercomputers like LUMI.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scaling claim rests on the bare Wilson-Dirac operator only; the paper itself notes even-odd preconditioning cannot use the same overlap, so 82% and 679 TB/s may not describe production HiRep.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the measured scaling of the bare Wilson-Dirac operator is treated as representative of the full HiRep production path, even though even-odd preconditioning, inverters, integrators, and clover terms have different communication and kernel mixes. This is not a minor issue because the paper's stated value proposition is enabling large BSM lattice calculations, not merely benchmarking a single kernel. The manuscript itself flags that the even-odd preconditioned Dirac operator used mainly in HMC cannot exploit the same parallel structure, which is direct in-text evidence that the scaling results may not generalize. I do not think this warrants rejection: the paper is a proceedings-style engineering report and reports operator-level data transparently, and the weak-scaling plateau from 32 to 1024 GCDs is a useful result in its own right. However, the conditional verdict is appropriate, and the concrete test would determine whether the extrapolation to production HiRep holds. I considered whether the more striking arithmetic issue (per-GCD bandwidth at 1024 GCDs is about 49% of the stated single-GCD value, not near 100%) should be the primary concern. That is a real characterization issue, but it is visible in the table and does not invalidate the scaling measurement itself. The representativeness gap is more load-bearing for the paper's central claim and is exactly where the argument is least secure.","tokens_in":4458,"tokens_out":12724,"duration_ms":118688,"concrete_test":"On LUMI-G, run the even-odd preconditioned Wilson-Dirac operator (or the CG solver used in HMC) with the same 48^4 local lattice and MPI decompositions as Table 1, from 32 to 1024 GCDs. Compare per-GCD bandwidth to the unimproved-Dirac rows. If per-GCD bandwidth falls more than ~20% from 32 to 1024 GCDs, or if the single-GCD baseline drops below ~60%, the headline scaling claim does not transfer to the production path.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 and Table 1 benchmark only the unimproved Wilson-Dirac operator in isolation, while the abstract and conclusion generalize to 'the code' and to enabling BSM production runs. The missing step is the even-odd preconditioned operator and the inverter/integrator path actually used in HMC. Section 2.1 states that the parallel streaming of even and odd inner-point work is impossible in the even-odd preconditioned Dirac operator used mainly in HMC; Section 2.2 shows the clover term only separately. Inverters add global reductions and different communication patterns (e.g., dot products, all-reduces, Schur complement operations) that are absent from the kernel-only benchmark. Therefore the 82% single-GCD efficiency and the flat ~40%-of-peak weak-scaling plateau are not direct evidence that the full production code scales near-optimally to 1024 GCDs. The central inference is an extrapolation from a single kernel to a full production path, and the manuscript contains an explicit limitation supporting this concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This proceedings paper reports GPU porting progress for the HiRep lattice field theory code, focusing on scaling of the Wilson-Dirac operator on AMD MI250X GPUs (LUMI-G). The authors present profiler traces of kernel execution, benchmark the unimproved Wilson-Dirac operator and the clover term, and report weak and strong scaling up to 1024 GCDs. They claim a single-GCD memory bandwidth of 1.34 TB/s (about 82% of the theoretical peak of 1.6384 TB/s) and a peak aggregate bandwidth of 679 TB/s at 1024 GCDs, with weak scaling staying at about 40% of theoretical peak beyond 128 GCDs. The paper also compares performance against NVIDIA A100 and H100 results.","tokens_in":4616,"tokens_out":5305,"duration_ms":46829,"significance":"If the scaling result holds for the production code path, this is a useful data point for the lattice community: it demonstrates that a CUDA-originated code can reach near-optimal single-GCD bandwidth and flat weak scaling on AMD GPUs at scale for a Wilson-Dirac kernel. The paper is honest about strong-scaling degradation (16% efficiency at 1024 GCDs, local lattice 128^4) and makes the code available open-source. The main strength is that the measurements are direct and compared against the theoretical device peak. The main weakness is that the benchmark covers only the unimproved Wilson-Dirac operator in isolation, and the manuscript generalizes the conclusions to the full production code used in HMC, which includes even-odd preconditioning, clover terms, inverters, and integrators with different communication and reduction patterns.","major_comments":[{"comment":"The scaling data and headline bandwidths in Section 3 and Table 1 are explicitly for the unimproved Wilson-Dirac operator in isolation (Table 1 caption), but the abstract and conclusion generalize these results to 'the code' and to enabling BSM production runs. The paper itself notes in Section 2.1 that the even-odd preconditioned Dirac operator, used mainly in the HMC, cannot use the same overlapping of even/odd inner-point work, and Section 2.2 treats the clover term only as a separate application. Inverters and integrators introduce global reductions (dot products, all-reduces) and different communication patterns that are absent from this kernel-only benchmark. The 82% single-GCD efficiency and the flat ~40%-of-peak weak-scaling plateau are therefore not direct evidence that the full production HiRep path scales near-optimally to 1024 GCDs. Please either restrict the claims to the Dirac-operator kernel or add measurements of the even-odd preconditioned operator and at least one representative inverter/solver path.","section":"Section 3 / Table 1 / Conclusion"},{"comment":"The scaling numbers in Table 1 and the cited peak bandwidth values (679 TB/s and 1.34 TB/s) appear to come from single profiler runs; the paper acknowledges 'inevitable variability from one Dirac execution to another' and profiler sampling precision. Without repeated-run statistics or error estimates, the reader cannot judge whether the small efficiency differences in the weak-scaling column (49%, 44%, 40%, 41%, 40%) are significant or whether the plateau is truly flat as claimed. Please report the run-to-run spread for representative GCD counts (e.g., 16, 128, 1024) or explicitly state that these are single measurements and characterize the associated uncertainty.","section":"Section 3 / Table 1"}],"minor_comments":[{"comment":"The text renders local lattice sizes as '484' and '164'; these should be 48^4 and 16^4 with proper superscripts, since the meaning is otherwise unclear.","section":"Sections 2.2 and 3 (typography)"},{"comment":"The '%' column in Table 1 is not defined in the caption; specify that it is the percentage of the theoretical aggregate peak memory bandwidth of the GCDs.","section":"Table 1 caption"},{"comment":"The claim that HiRep is 'the highest performing option' for Wilson fermions in higher representations is unsupported by a comparative reference or benchmark; consider softening this or providing a citation.","section":"Section 1"},{"comment":"The discussion of blocking versus non-blocking communications concludes that the choice 'needs to be tested on an individual basis,' but no such test is reported; briefly state which mode is used in the presented benchmarks.","section":"Section 2.1.1"},{"comment":"The caption notes that the right ROCm profile shows only one of the four MPI processes; consider adding that this trace therefore does not display the node-level communication pattern.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"This is a proceedings-style contribution, and the kernel-level scaling data are a legitimate result. The central issue is the gap between the benchmarked Dirac operator and the production HMC path; this is fixable either by rewording the claims to be kernel-specific or by adding a solver-level benchmark. Given the paper's title and abstract, I would encourage the latter."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a good, honest engineering paper, and the scaling numbers are probably real—but they are for the bare Wilson-Dirac operator in isolation, not for the full HiRep production path. The paper says so if you read carefully, but the abstract and conclusion generalize to \"the code,\" which overstates what is measured.\n\nWhat's new: scaling measurements up to 1024 GCDs on AMD MI250X (LUMI-G), with a 679 TB/s aggregate bandwidth at 40% of theoretical peak in weak scaling, and 82% (1.34 TB/s) of peak on a single GCD. Also useful ROCm tracing showing kernel overlap of boundary and communication with the inner Dirac application. This extends the authors' earlier EuroPLEx2023 work from NVIDIA to AMD and to larger jobs. The code is on GitHub, which is a plus.\n\nThe paper is honest about its limitations. Section 2.1 states that the even-odd preconditioned Dirac operator used mainly in HMC cannot exploit the parallel streaming of even and odd parts. Section 2.2 benchmarks the clover term separately. Table 1 is explicitly for the unimproved Wilson-Dirac operator. So the concern about extrapolating to production is real—the 82% and 679 TB/s do not directly say anything about HMC performance, which uses inverters, global reductions, and a different communication pattern. The paper is careful to say \"weak scaling only\" and to disclose the strong scaling collapse to 16% at 1024 GCDs. That is good.\n\nSoft spots: no error bars or repeated-run statistics, and timings come from profiler sampling with acknowledged variability. For an engineering performance claim, that is a reproducibility gap. A commit hash and raw data would help. Also, the comparison to theoretical peak should be taken with a grain of salt (they exceed it on A100 due to L2 reuse). That doesn't invalidate the results, but it means \"percent of peak\" is a loose metric.\n\nBottom line: this is a useful proceedings paper for lattice practitioners who run on AMD GPUs, and the authors know their code base. The central scaling claim is supported for the Dirac kernel, and the paper is transparent about the gap between that and full production. It deserves a serious referee if submitted as a full paper, with requests for reproducibility artifacts and a clearer statement that the headline numbers are kernel-level, not end-to-end HiRep.\n\nRecommendation: engage with it; the work is sound and the limitations are disclosed.","headline":"Solid, honest scaling study of HiRep on AMD GPUs, but the headline numbers cover only the bare Wilson-Dirac kernel, not the production HMC path.","tokens_in":5176,"tokens_out":2188,"would_cite":true,"duration_ms":19377,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The HiRep lattice code reaches near-optimal weak scaling on AMD GPUs, with 679 TB/s on 1024 compute dies and 82% of theoretical peak bandwidth on a single die.","keywords":["HiRep","lattice field theory","Wilson fermions","GPU scaling","weak scaling","AMD GPUs","higher representations","clover improvement"],"falsifier":"Run a full Hybrid Monte Carlo update (or at least a preconditioned Wilson-Dirac inverter) at a fixed local lattice of $48^4$ on 1024 AMD MI250X compute dies and compare the aggregate bandwidth or time per trajectory with a single-die run; if the aggregate throughput falls clearly below the $679$ TB/s reported for the isolated operator, the near-optimal weak scaling claim does not carry over to production.","tokens_in":4250,"feed_emoji":"⚡","tokens_out":12762,"duration_ms":101751,"temperature":0.7,"pith_summary":"This paper reports a large-scale performance measurement of the HiRep lattice field theory code on AMD GPUs. For the Wilson-Dirac operator, the code reaches 82% of the theoretical memory bandwidth of a single MI250X GPU compute die, and its weak scaling stays essentially linear out to 1024 compute dies, where it sustains 679 TB/s. The result matters because HiRep supports Wilson fermions in higher representations of SU(N) with arbitrary numbers of colors and flavors, which is what beyond-standard-model lattice predictions need. The paper also shows the clover-improvement term adds little extra time and that strong scaling, unlike weak scaling, degrades once the inner computation becomes too fast to hide communication.","feed_headline":"Lattice code hits 82% of peak bandwidth on one AMD GPU","feed_subtitle":"It weak-scales to 1024 GPUs at 679 TB/s, enabling large beyond-standard-model runs.","key_machinery":"The load-bearing mechanism is the kernel decomposition of the Wilson-Dirac operator into local inner-point computations and boundary computations. The inner kernel fills the GPU completely, while a separate POSIX thread performs send-buffer synchronization, asynchronous copies, and the boundary kernel, so communication happens behind the compute. Performance is measured as achieved memory bandwidth rather than floating-point throughput, because lattice Dirac operators are memory-bound; AMD and NVIDIA profiler traces are used to check that the communication and compute streams really do overlap.","core_discovery":"The paper's central claim is that the GPU port of HiRep's Wilson-Dirac operator scales close to optimally on AMD hardware: one compute die (GCD) of an MI250X delivers 1.34 TB/s (82% of the 1.6384 TB/s theoretical peak), and 1024 compute dies together deliver 679 TB/s at a fixed local lattice of $48^4$ with all four dimensions parallelized. This is achieved by splitting the operator into an inner kernel that needs no communicated data and a boundary kernel that does, with all communication issued from a separate POSIX thread so that message passing overlaps with the heavy inner kernel. The reported figures are for the unimproved Wilson-Dirac operator; strong scaling on a fixed $128^4$ global lattice falls from 41% efficiency at 32 GCDs to 16% at 1024 GCDs, and the authors advise against local lattices smaller than $16^4$ for SU(2) gauge groups.","pith_inferences":["The paper benchmarks only the isolated Dirac operator and the clover term, so the 82% and 679 TB/s figures are not end-to-end Hybrid Monte Carlo performance; a fairer test would be measuring the full inverter or trajectory time.","A natural next benchmark is weak scaling at larger local volumes such as $64^4$, which should give the inner kernel more work to hide communication and may extend the near-perfect curve.","Because the observed overhead comes from boundary kernels and buffer synchronization, communication optimisations such as GPU-aware MPI could plausibly improve strong scaling without changing the lattice mathematics.","The $16^4$ local-volume guideline is established only for SU(2); for larger color groups the same threshold would need its own scaling test."],"forward_implications":["Weak scaling is close to perfect up to 1024 GCDs, so fixed-size local volumes such as $48^4$ can be run on full machine partitions without paying an efficiency penalty.","A single GCD runs the Wilson-Dirac operator at 82% of theoretical memory bandwidth, meaning the CUDA-originated code is close to the hardware limit on AMD as well.","Strong scaling loses efficiency beyond about 128 GCDs, so users who want small local lattices should expect lower per-GPU bandwidth; local volumes below $16^4$ are advised against for SU(2).","The clover-improvement term adds only modest extra time, so improved actions, including exponential clover, can be used without sacrificing the scaling picture.","The scaling behaviour makes high-statistics, full-machine lattice calculations with higher-representation Wilson fermions practical for beyond-standard-model phenomenology."],"supporting_citations":[{"why":"introduces the HiRep code with higher-representation Wilson fermions that this paper scales to AMD GPUs","marker":"[3]"},{"why":"the prior GPU port of HiRep that this work extends and benchmarks on AMD hardware","marker":"[4]"},{"why":"defines the standard clover improvement whose application overhead is measured","marker":"[5]"},{"why":"defines the exponential clover improvement whose symmetric Dirac/inverse timing is exploited for Hasenbusch acceleration","marker":"[6]"},{"why":"introduces Hasenbusch acceleration, the integrator technique that benefits from the exponential clover timing","marker":"[7]"}],"fun_headline_variants":["HiRep GPU port hits 82% of peak bandwidth on AMD","HiRep weak-scales to 1024 GPUs at 679 TB/s","82% peak bandwidth on AMD, 679 TB/s on 1024 GPUs","Wilson-Dirac kernel on AMD: 82% peak, scales to 1024 GPUs","HiRep on AMD MI250X: 1.34 TB/s single GCD, 679 TB/s at 1024"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 82% and 679 TB/s numbers come from the bare Wilson-Dirac operator alone, and the paper assumes that this near-optimal behavior carries over to the full production code path, including inverters, integrators, even-odd preconditioning, and clover terms, which have different communication patterns and kernel mixes.","fun_headline_variants_meta":{"raw":{"variants":["HiRep GPU port hits 82% of peak bandwidth on AMD","HiRep weak-scales to 1024 GPUs at 679 TB/s","82% peak bandwidth on AMD, 679 TB/s on 1024 GPUs","Wilson-Dirac kernel on AMD: 82% peak, scales to 1024 GPUs","HiRep on AMD MI250X: 1.34 TB/s single GCD, 679 TB/s at 1024"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000546,"raw_usage":{"total_tokens":2539,"prompt_tokens":803,"completion_tokens":1736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":1619}},"tokens_in":419,"tokens_out":1736,"duration_ms":10506,"temperature":1.0,"reasoning_tokens":1619,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:06:40.850384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a full Hybrid Monte Carlo update (or at least a preconditioned Wilson-Dirac inverter) at a fixed local lattice of $48^4$ on 1024 AMD MI250X compute dies and compare the aggregate bandwidth or time per trajectory with a single-die run; if the aggregate throughput falls clearly below the $679$ TB/s reported for the isolated operator, the near-optimal weak scaling claim does not carry over to production.","supporting_citations":[{"cited_title":"GPU- accelerated Higher Representations of Wilson Fermions with HiRep.PoS, EuroPLEx2023:035, 2024","cited_arxiv_id":null,"evidence_quote":"the prior GPU port of HiRep that this work extends and benchmarks on AMD hardware"},{"cited_title":"Sheikholeslami and R","cited_arxiv_id":null,"evidence_quote":"defines the standard clover improvement whose application overhead is measured"},{"cited_title":"Speeding up the hybrid Monte Carlo algorithm for dynamical fermions.Phys","cited_arxiv_id":null,"evidence_quote":"introduces Hasenbusch acceleration, the integrator technique that benefits from the exponential clover timing"}],"review_version":1}