REVIEW 2 major objections 4 minor 2 cited by
Usage of GPUs for online and offline Reconstruction in ALICE in Run 3
T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read ALICE's GPU farm speeds offline reconstruction 2.5x, aiming for 5x.
desk verdict Useful ALICE status report with solid measured speedups; the 5x offline projection is an unstated Amdahl upper bound, not a guarantee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the O2 framework's unified data processing graph (DPL), in which all GPU code is written in generic C++ and can dispatch to CUDA, ROCm, or OpenCL hardware. The EPN farm, 350 servers each carrying 8 AMD GPUs, supplies 90% of the online computing power through highly optimized TPC tracking, clustering, and compression, which is 99% of the online workload and 50% to 60% of the asynchronous workload. DPL runs multiple round-robin instances of slower CPU tasks and uses a publishing-rate smoothing heuristic to keep CPU utilization above 90% during offline processing. The paper's speedup expectations are computed from relative processing-time tables for the reconstruction steps.
What would settle it
Measure the relative CPU fractions for the full central-barrel chain on a high-multiplicity Pb-Pb dataset with calorimeters in the run; if the offloadable fraction falls below 80% or the GPU-to-CPU speedup ratio differs from the tables, the paper's 5x prediction would be contradicted. A direct wall-clock comparison of the same offline campaign with 60% versus 80% GPU offload on the EPN farm would also settle the predicted 5x.
Extended reading notes
Core claim
The central discovery is that the same O2 software and generic GPU code can serve both real-time online compression and calibration and asynchronous offline reconstruction, and that GPU offload changes which resource is the bottleneck. Online, TPC processing dominates at 99% of the reconstruction workload, so the farm is GPU-bound and 99% of online reconstruction runs on GPUs. Offline, TPC processing is still the largest item but is only 50% to 60% of the total, so the asynchronous processing is CPU-bound; offloading that TPC fraction plus other tasks yields measured speedups of 2.5x for 650 kHz pp and 2x for 47 kHz Pb-Pb, closely matching the expectation from the relative workload fractions. The paper predicts that offloading the full central-barrel global tracking chain, about 80% of the offline workload, will give a total speedup of 5x.
Load-bearing premise
The predicted 5x speedup assumes the relative processing-time fractions measured for the 2022 pp and 2023 Pb-Pb datasets stay representative across data-taking conditions such as interaction rate, event multiplicity, and detector configuration.
Editorial extensions
If this is right
- If the 80% offload target is reached, asynchronous reconstruction on the EPN farm should run about 5x faster than CPU-only processing.
- A CPU-only online farm would need over 3000 servers with 64 physical cores each, so the GPU-enabled EPN farm is what makes 50 kHz Pb-Pb data taking affordable.
- Because the GPU code is portable across CUDA, ROCm, and OpenCL, the same offloaded reconstruction can run at GPU-equipped GRID sites.
- Online processing stays GPU-bound with 99% of reconstruction on GPUs, so offloading the remaining 1% would add complexity without benefit.
Reading between the lines
- If the 5x speedup holds in routine production, ALICE's EPN farm could become a primary offline reconstruction resource, reducing reliance on external CPU-only GRID capacity during LHC downtimes.
- The 5x prediction rests on workload fractions measured for specific 2022 pp and 2023 Pb-Pb datasets; re-measuring those fractions at different interaction rates or with calorimeters in the run would show how stable the speedup is.
- The pattern of regular dips in CPU utilization during rate smoothing suggests a closed-loop controller could push average utilization above the reported 90% and shorten offline processing campaigns.
- The principle that even an inefficient GPU port is worthwhile when the processor is CPU-bound extends naturally to simulation and analysis workloads, not just reconstruction.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This proceedings paper describes the use of GPUs in the ALICE online and offline reconstruction during LHC Run 3. The online EPN farm, consisting of 350 servers with 8 GPUs each, processes TPC data for calibration and compression, with 99% of the online reconstruction workload on GPUs. When there is no beam, the same farm runs offline reconstruction. The paper presents relative CPU processing-time tables for the synchronous (online) reconstruction and for the asynchronous (offline) reconstruction of pp and Pb-Pb data, and reports measured speedups of 2.5x (pp) and 2.0x (Pb-Pb) from offloading the dominant TPC processing to GPUs. It also discusses scheduling and rate-limiting strategies to keep CPU utilization above 90%, and states that offloading 80% of the offline workload to GPUs should yield an expected speedup of 5x. The paper concludes that GPUs are essential for the online farm, estimating that a CPU-only farm would require over 3000 servers.
Significance. The paper provides a concise, quantitative update on ALICE's production use of GPUs in Run 3, which is useful for the HEP computing community. Its strengths include the presentation of measured relative workloads (Tables 1-3), the consistency check of the current speedups against Amdahl's law (Section 2), and the concrete description of the DPL scheduling heuristics with before/after CPU utilization figures. The claims about the essential role of GPUs for the online farm are supported by the data. However, the future projection of a 5x speedup from 80% offload is an idealized Amdahl bound and needs qualification, and there is a minor numerical inconsistency in the offload fraction for Pb-Pb data.
major comments (2)
- [Section 4] The predicted 'expected total speedup of 5×' from offloading 80% of the offline workload is the Amdahl upper bound 1/(1-0.8) = 5, which assumes that the offloaded work executes in zero time on the GPU. The paper does not provide measured or estimated GPU execution times for the central-barrel tracking tasks (ITS, TRD, TOF, secondary vertexing) that are planned to be offloaded, nor does it account for the scheduling and multiplicity overheads described in Section 3. The measured current speedups (2.5x for pp, 2.0x for Pb-Pb) are slightly below the Amdahl limits for the reported TPC fractions (2.59x and 2.10x, respectively), indicating that GPU time for the offloaded TPC processing is small but not zero; for the other tasks, the GPU speedups may be lower. I recommend either providing an estimate of the expected speedup that includes finite GPU time, or explicitly stating that 5x is an upper bound rather than the expected speedup.
- [Section 4, Table 3] The paper states that the full central-barrel global tracking chain (ITS, TPC, TRD, and TOF tracking and matching, secondary vertexing, and track refit) 'amounts to 80% of the total workload' in both offline cases. Summing the relevant rows of Table 3 for Pb-Pb gives 52.39% + 12.65% + 8.97% + 4.39% + 2.85% + 2.64% = 83.89%, not 80%. This numerical inconsistency should be corrected (or the set of included tasks clarified), as it is the basis for the 80% offload claim and the associated speedup projection.
minor comments (4)
- [Tables 2 and 3] The relative processing times are quoted without any uncertainty or indication of how they were measured (e.g., average over a run, median, or representative values). Adding a note on the stability of these fractions across data-taking conditions would strengthen the extrapolation to the 5x projection.
- [Section 2] The sentence 'offloading 50% or 60% should achieve a speedup of 2× or 2.5×' is ambiguous regarding which fraction corresponds to pp and which to Pb-Pb; Table 2 shows a TPC fraction of 61.41% for pp and Table 3 shows 52.39% for Pb-Pb, so the text could specify that pp corresponds to the 60% case and Pb-Pb to the 50% case.
- [Figures 1 and 2] The captions are minimal; adding the average CPU utilization values and the time window to the captions would make the improvement easier to quantify.
- [Section 2] Reference [4] is cited for the Amdahl expectation; since the speedup formula is central to the argument, a brief derivation or explicit formula in Section 2 would improve self-containedness.
Circularity Check
No significant circularity: the paper reports measured GPU speedups and uses Amdahl-law expectations as consistency checks, not as fitted predictions.
full rationale
The paper is a performance report for the ALICE Run 3 reconstruction. The central quantitative statements are measurements: the 2.5x pp and 2x Pb-Pb offline speedups are presented as achieved values, and the tables list relative processing times for specific datasets. The expected 5x speedup from offloading 80% of the workload is an Amdahl-limit estimate (1/(1-0.8)=5) made from the stated target fraction; it is not derived by fitting a parameter to the measured speedups and then claiming those speedups as a prediction. The statement that the achieved speedup matches the expectation for offloading 50-60% is an independent consistency check, and the paper explicitly notes that other algorithms may have different GPU speedups, showing awareness that the estimate is an upper bound rather than a forced identity. The self-citations ([3], [4], [6], [7]) refer to prior ALICE software and GPU-framework work, but no load-bearing mathematical claim is imported from those citations; the uniqueness-type argument is absent. The CPU-only farm estimate is an extrapolation from the farm design, not a circular definition. Overall, the derivation chain is self-contained: measured inputs, standard speedup formulas, and measured outputs.
Assumptions & free parameters
assumptions (2)
- standard math Amdahl's law applies to the ALICE reconstruction pipeline when estimating speedups from GPU offload.
- domain assumption The relative processing times in Tables 2 and 3 are representative of the actual run conditions and software versions.
Cite this review
Pith. "Pith review of Usage of GPUs for online and offline Reconstruction in ALICE in Run 3." pith.science (2026). https://pith.science/paper/AAXX4KHB
@misc{pith2026250209138,
author = {Pith},
title = {Pith review of: Usage of GPUs for online and offline Reconstruction in ALICE in Run 3},
year = {2026},
howpublished = {\url{https://pith.science/paper/AAXX4KHB}},
note = {Machine review of arXiv:2502.09138}
}
read the original abstract
ALICE records Pb-Pb collisions in Run 3 at an unprecedented rate of 50 kHz, storing all data in continuous readout (triggerless) mode. The main purpose of the ALICE online computing farm is the calibration of the detectors and the compression of the recorded data. The detector with the largest data volume by far is the TPC, and the online farm is thus optimized for fast and efficient processing of TPC data during data taking. For this, ALICE leverages heavily the compute power of GPUs. When there is no beam in the LHC, the GPU-equipped farm performs the offline reconstruction of the recorded data, in addition to the GRID. Since the majority of the compute capacity of the farm is in the GPUs, and meanwhile also some GRID sites begin to offer GPU resources, ALICE has started to offload other parts of the offline reconstruction to GPUs as well. The talk will present the experience and processing performance with GPUs in the Run 3 Pb-Pb and pp online and offline processing in ALICE.
Figures
Forward citations
Cited by 2 Pith papers
-
Real-Time Analysis of Unstructured Data with Machine Learning on Heterogeneous Architectures
A graph neural network (ETX4VELO) reconstructs LHCb VELO tracks with performance comparable to the production 'search by triplet' algorithm while running end to end in the GPU-based first-level trigger, with additiona...
-
Usage of GPUs for ALICE Run 3 Offline Reconstruction on the GRID
ALICE reports a 29% throughput gain from porting track model decoding and ITS tracking to GPU, and the first LHC offline reconstruction jobs on external GRID GPUs.
Reference graph
Works this paper leans on
-
[1]
TheALICEexperimentattheCERNLHC
ALICECollaboration,“TheALICEexperimentattheCERNLHC”,J.Inst. 3S08002(2008)
work page 2008
-
[2]
Technical Design Report for the Upgrade of the Online-Offline Com- puting System
ALICE Collaboration, “Technical Design Report for the Upgrade of the Online-Offline Com- puting System”, CERN-LHCC-2015-006, ALICE-TDR-019 (2015)
work page 2015
-
[3]
Global Track Reconstruction and Data Compression Strategy in ALICE for LHC Run 3
D. Rohr for the ALICE Collaboration, “Global Track Reconstruction and Data Compression Strategy in ALICE for LHC Run 3”, Proceedings of CTD2019 (2019) arXiv:1910.12214
work page Pith review arXiv 2019
-
[4]
The O2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3
G. Eulisse and D. Rohr for the ALICE Collaboration, “The O2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3”, EPJ Web of Conf., vol. 295, pp. 05022, Proceedings of CHEP 2023 (2024) arXiv:2402.01205
arXiv 2024
-
[5]
ALICE Collaboration, Real-time data processing in the ALICE High Level Trigger at the LHC, CPC242 25 (2019), arXiv:1812.08036
work page Pith review arXiv 2019
-
[6]
D. Rohr, V. Lindenstruth, “Portable and Vendor-Independent Low-Level Programming and Performance Benchmarking for Graphics Cards and Processors”, Proceedings of 2017 IEEE 19th HPCCWS (2017)
work page 2017
-
[7]
GPU performance in Run 3 ALICE online/offline reconstruction
G. Cimador for the ALICE Collaboration, “GPU performance in Run 3 ALICE online/offline reconstruction”, Proceedings of 12th Large Hadron Collider Physics Conference (2024). 6
work page 2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.