Pith. sign in

REVIEW 3 major objections 5 minor 14 references

Usage of GPUs for ALICE Run 3 Offline Reconstruction on the GRID

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ALICE puts ITS tracking on GPU, lifting offline throughput 29%.

desk verdict A credible operational milestone with a real but caveated 29% throughput claim; worth refereeing, though the benchmarking methodology needs a sensitivity analysis. read the letter →

arxiv 2608.11819 v1 pith:XGNTU2PJ submitted 2026-08-12 physics.ins-det

David Rohr for the ALICE Collaboration This is my paper · ORCID
classification physics.ins-det
keywords ALICELHCheavy-ionexperimentGPUreconstructionofflineGRIDcomputingO2frameworkITStrackingTPCRun3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ALICE, the heavy-ion experiment at the LHC, already runs nearly all of its online reconstruction on GPUs. This paper reports extending the same accelerated tracking chain to offline reconstruction, where the aggregate bottleneck is the CPU: because most offline work still runs on CPUs, the throughput gain comes from moving a larger fraction of the work off the CPU, not from the raw GPU speedup. Porting TPC track-model decoding and ITS tracking to GPU raised offline time-frame reconstruction throughput by 29%, on top of the earlier 2.5x gain from putting TPC tracking on GPU. The paper also reports the first use of GPUs for LHC offline reconstruction on the GRID, run in 2026 on NERSC's Perlmutter cluster, using a single software build that serves AMD, NVIDIA, and OpenCL devices and a two-setting user interface for GPU job submission. The stated target is full barrel tracking on GPU, projected to give a 5x total offline throughput gain.

What carries the argument

The load-bearing mechanism is the ALICE offline reconstruction as a directed graph of parallel OS processes in the O2 Data Processing Layer, where all tasks must sustain the same global throughput and slower tasks are replicated through process multiplicities. Because the workflow is CPU-bound, global throughput is set by the remaining CPU load, so the relevant quantity is the fraction of workload moved to GPU rather than the GPU speedup; the paper quantifies this as a 2.5x gain from offloading TPC reconstruction (60% of the workload) and a projected 5x from offloading the full barrel tracking chain (about 80%). The enabling infrastructure is a plugin-based GPU backend framework with run-time compilation, allowing one central CVMFS build to target AMD MI50/MI100, NVIDIA A100, and OpenCL devices across GRID sites.

What would settle it

A decisive test would be to run the identical ALICE offline workload at a fixed CPU allocation with one GPU versus two GPUs, and then with multiplicities set by an automated tuner: the CPU-bound model predicts no throughput change from the extra GPU, and an automated-tuned benchmark would reveal whether the 29% gain survives outside the developers' manual settings.

Watch

Extended reading notes

Core claim

The central claim is that the ALICE offline reconstruction workflow, though CPU-bound in aggregate, can be sped up by offloading selected tracking steps to GPU, and that the speedup is determined by the fraction of CPU workload removed rather than by raw GPU performance. The paper reports a 29% throughput increase from porting TPC track-model decoding (about 3%) and ITS tracking (about 26%) to GPU, as part of a pipeline whose remaining CPU tasks must be re-tuned with manually adjusted process multiplicities. It further claims a first for LHC computing: in 2026 ALICE ran offline reconstruction jobs on the GPUs of the NERSC Perlmutter cluster on the GRID, using a vendor-neutral plugin architecture and run-time compilation of GPU code so that a single distributed software build serves CPU-only sites and all GPU vendors. From the CPU-bound model, the paper projects that offloading the full barrel tracking chain, roughly 80% of the offline workload, would yield about a 5x total throughput gain.

Load-bearing premise

The 29% gain assumes that the manually tuned CPU task multiplicities used in the benchmarks are fair to both the CPU-only and GPU configurations; if tuning favored the GPU case, the production gain could be smaller.

Editorial extensions

If this is right

  • Offline reconstruction on the EPN farm remains CPU-bound, so further GPU offload raises throughput only as the offloaded workload fraction grows; the immediately next steps are the barrel-tracking tasks the paper lists as in commissioning or work in progress.
  • A GRID user can request GPU reconstruction by specifying only the GPU count and a site or device mapping; all workflow and backend configuration happens behind the scenes.
  • One central software build replaces site-specific builds for GPU reconstruction, since vendor libraries are loaded through plugins and GPU code is compiled at run time to the local architecture.
  • If the full barrel tracking chain, about 80% of the offline workload, is offloaded, total offline throughput should approach 5 times the CPU-only baseline, provided the pipeline does not become GPU-bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The manual tuning burden described in Section 3.1 could erode the 29% gain in routine production if data-taking conditions drift; automated tuning, which the paper says is under consideration, would make the gain reproducible outside the developers' benchmark setup.
  • If the CPU-bound model is right, the 29% gain should transfer across GRID sites even when the GPUs are slower, because the workflow-level gain depends on the offloaded workload fraction, not on GPU speed.
  • The plugin-and-run-time-compilation design suggests a path for other LHC experiments to offer heterogeneous GPU resources without maintaining per-vendor builds, though ALICE's CPU-bound offline balance may not carry over to workflows with different CPU/GPU splits.
  • A practical next test would be to measure whether the two-setting user interface lets outside users reproduce the 29% gain on demand, rather than only in the ALICE team's tuned benchmark runs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports on the ALICE collaboration's use of GPUs for Run 3 offline reconstruction on the GRID. The authors describe the GPU-based online processing on the EPN farm, the subsequent deployment of TPC offline tracking on GPUs since 2023, and the recent porting of TPC track-model decoding and ITS tracking to GPU, which they state improved total offline reconstruction throughput by 29%. They also report the first use of GPUs for LHC offline reconstruction on the GRID at the NERSC Perlmutter cluster in 2026, supported by a vendor-agnostic plugin architecture, runtime compilation (RTC), and transparent job submission with only GPU count and type specified by the user. The paper includes figures comparing GPU processing times and throughput between EPN MI50 GPUs and NERSC A100 GPUs, and discusses the CPU-bound nature of offline processing and the need to tune CPU process multiplicities.

Significance. If the claimed 29% throughput improvement is robust, this is a practically important incremental result for HEP offline computing: it increases the case for using existing GPU infrastructure for LHC offline reconstruction and demonstrates a working model for heterogeneous GRID sites. The paper also documents genuinely useful systems work: the CVMFS-based central software build with vendor-independent GPU backends, runtime compilation to local architecture, and the simplification of user-side GPU job submission to two settings. The framing of offline processing as CPU-bound, with throughput determined by the fraction of workload offloaded, is a clear and falsifiable statement of the performance model. The main weakness is that the central quantitative claim rests on manually tuned process-multiplicity settings, and the paper provides no sensitivity analysis, no error bars, and no end-to-end wall-clock comparison with the GPU offload toggled on and off; for a conference-proceedings contribution this may be acceptable, but as reported the 29% figure is not yet established beyond the specific tuning choices made.

major comments (3)
  1. [Section 3.1] The central claim of a 29% throughput improvement is not supported by any quantitative description of the manual tuning on which the benchmark comparisons depend. The text states that 'all benchmark comparisons require multiple iterations of the manual tuning, to enable an apples-to-apples comparison, and to prevent one of the CPU tasks from becoming a bottleneck,' and that 12 different multiplicity settings are required. Since offline processing is explicitly CPU-bound (Sections 3 and 4), the global throughput is set by the slowest residual CPU task; a different manual tuning choice for the CPU-only versus GPU-offloaded configurations could therefore change the measured 29% by an amount comparable to the effect being claimed. Please report the actual multiplicity values used for each benchmark configuration, the number of iterations performed, and a sensitivity scan over at least a subset of these settings so that the reader can assess whether the reported gain is robust to tuning choices.
  2. [Section 4] The paper does not show the end-to-end measurement from which the 29% improvement is derived. Figures 3 and 4 report GPU-side metrics (per-time-frame processing time on a single GPU, aggregate GPU throughput), and Figure 5 reports GPU idle time; none of these is a wall-clock comparison of the full offline reconstruction job with the two newly offloaded tasks (TPC track-model decoding and ITS tracking) enabled versus disabled. Given the statement in Section 3 that offline processing is CPU-bound and that 'the throughput on the GPU does not propagate to the global throughput,' the global throughput must be measured directly. Please provide a table or figure of the global offline reconstruction throughput (e.g., time frames per second per job) for the configurations with and without each GPU offload, including the number of repeated runs and the run-to-run spread. Without this, the claimed 3% + 26% = 29% decomposition cannot be verified from the presented data.
  3. [Section 3] The paper reports 'a ∼3% increase' from TPC track-model decoding and 'an additional 26% increase' from ITS tracking, yielding 29% in total, but does not specify whether the 3% and 26% were measured as independent separate benchmarks or obtained as a decomposition of a single joint measurement. These are different protocols and can give different results in a CPU-bound pipeline with nonlinear task interactions. Please clarify the measurement protocol: were the two porting efforts benchmarked independently and then combined, or measured together? If the latter, how were the individual contributions determined? This matters for the credibility of the additive claim and for the interpretation of Amdahl's-law-style reasoning used elsewhere in the paper.
minor comments (5)
  1. [Section 4, Figure 3] The caption and accompanying text state that the A100 is approximately twice as fast as the MI50, but the red curve (TPC-only on MI50) visibly flattens for large time frames; the text mentions this but does not explain the mechanism. Please add a brief explanation of why the MI50 becomes relatively more efficient at large datasets, and indicate whether the outliers in the green curve are excluded from any averaging.
  2. [Section 4, Figure 5] The horizontal axis is labeled 'Time [TF Id]' but the tick labels are 5000, 10000, 15000, 20000, 25000, 30000. This should be labeled 'Time-frame ID' (or 'TF number') for clarity. The text states that 'the full production takes longer, but the behavior just continues in the same way as on the right side of the figure'; it would be clearer to show a representative later portion or state explicitly that the steady-state behavior is identical beyond the shown range.
  3. [Section 5] The conclusion says 'with a total throughput increase of up to 2.5x' while Section 3 says the speedup is 'exactly the expected factor 2.5x'. Please reconcile these statements: is 2.5x the measured speedup, the theoretical expected speedup, or an upper bound? If measured, cite the relevant measurement; if theoretical, state so explicitly.
  4. [Section 3.1] The phrase '12 different multiplicity settings (with factors between 2 and 12)' is ambiguous: does 'factors' refer to the multiplicative increase in the number of process instances, or to the absolute multiplicity values? Please clarify the terminology (e.g., 'multiplicity values between 2 and 12').
  5. [References] Reference [13] is listed as 'Proceedings of CHEP 2026, to appear in this volume' and [14] as 'to appear in this volume'. If these are contributions to the same proceedings volume in which the paper appears, the bibliographic format should be consistent with the other references; if not, specify the actual publication venue.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 29% gain is a measured benchmark, not derived from its own inputs.

full rationale

The paper's central quantitative claims are empirical benchmarks rather than derivations. The 29% throughput gain from offloading TPC track-model decoding and ITS tracking is a measured comparison supported by timing and GPU-throughput data (Figs. 3–5), and the earlier 2.5x figure is checked against Amdahl's law from a workload fraction that is cited as measured, not fitted or defined by the claimed result. Citations to preceding ALICE papers for the component gains (3% and 26%) are normal incremental reporting of measured performance, not load-bearing logical reduction: no equation in the paper defines a predicted quantity in terms of the quantity it predicts. The paper's own admission in Sec. 3.1 that benchmark comparisons require multiple iterations of manual tuning of process multiplicities is a genuine benchmarking-robustness concern, but it is not circularity, because the claimed speedup is not equivalent by construction to the tuning inputs. The paper is self-contained against external benchmarks in the sense that the GPU speedups and wall-clock comparisons are measured and, where analytic speedups are quoted, they are computed from independently stated workload fractions. Therefore no significant circularity is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities or fitted constants. The only hand-chosen parameters are the process multiplicity settings, which are not disclosed and directly affect the reported speedups. The key assumptions are standard (Amdahl's law) or domain-specific (representative workload fractions and fair manual tuning).

free parameters (1)
  • CPU process multiplicity settings = Not disclosed (12 settings, factors 2 to 12 for pp and Pb-Pb)
    These hand-tuned parameters directly determine measured throughput; the paper admits benchmark comparisons require multiple tuning iterations, so undisclosed tuning could bias the reported 29% improvement.
assumptions (3)
  • standard math Amdahl's law models offline processing speedup as a function of the fraction of workload offloaded to GPU.
    Used in Section 3 to compute the expected 2.5x speedup from offloading 60% of TPC workload, and in Section 5 for the 5x projection.
  • domain assumption The measured workload fractions (e.g., TPC 50-60% of offline workload) are representative of production conditions.
    The speedup estimates depend on these fractions; they are stated as typical values without uncertainty in Section 3.
  • domain assumption Manual tuning of process multiplicities yields a fair (apples-to-apples) performance comparison.
    The authors themselves note in Section 3.1 that benchmark comparisons require multiple tuning iterations; the fairness of this procedure is load-bearing for the 29% improvement claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Usage of GPUs for ALICE Run 3 Offline Reconstruction on the GRID." pith.science (2026). https://pith.science/paper/XGNTU2PJ

@misc{pith2026260811819,
  author       = {Pith},
  title        = {Pith review of: Usage of GPUs for ALICE Run 3 Offline Reconstruction on the GRID},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGNTU2PJ}},
  note         = {Machine review of arXiv:2608.11819}
}
read the original abstract

ALICE, the dedicated heavy ion experiment at the LHC, records Pb-Pb collisions at an interaction rate of up to 50 kHz. It was the first LHC experiment to leverage GPUs for online data processing during LHC Runs 1 and 2, and its Run 3 online data processing is now fully based on GPUs with more than 90% of the compute load offloaded to the accelerator. In order to use its online processing server farm also for offline processing in an efficient way while the LHC is not operating, ALICE has been running the offline TPC tracking on GPUs since 2023. Since then, ALICE has been conducting an ongoing effort to offload more offline reconstruction steps to GPUs, and to use the GPUs at other GRID sites beyond the ALICE online computing farm for offline reconstruction. In particular, porting the track model decoding and ITS tracking to GPU has improved the throughput by 29%. In 2026, ALICE ran the first GPU reconstruction jobs on the NVIDIA GPUs of the NERSC Perlmutter cluster, marking the first use of GPUs for LHC offline reconstruction on the GRID. Users can submit GPU reconstruction jobs by specifying only the required GPU count and type, while the underlying infrastructure handles all remaining configurations transparently.

Figures

Figures reproduced from arXiv: 2608.11819 by the authors.

Figure 1
Figure 1. Processing graph of the ALICE barrel tracking chain. White labels indicate tasks required in the baseline scenario running only TPC online processing on GPUs. Black labels indicate the additional tasks intended for GPU offload in the optimistic scenario to offload the full barrel tracking. Colors indicate the status of the GPU implementation of the tasks, with green indicating the tasks that can run on GPU today. No… view at source ↗
Figure 2
Figure 2. ALICE O2 Pb-Pb Online Reconstruction Workflow Graph. Both NUMA domains run a copy of the same workflow each to avoid cross-NUMA traffic. Four tasks are running with process multiplicities, the TPC reconstruction for the four GPUs per NUMA domain and three CPU tasks. A critical aspect is to guarantee that all of the CPU processing steps are fast enough. In the ALICE DPL (Data Processing Layer) processing framework, e… view at source ↗
Figure 3
Figure 3. Processing time per time frame on a single AMD MI50 on an EPN versus an NVIDIA A100 GPU at NERSC, measured in TPC-only tracking and in TPC + ITS tracking on GPU [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Aggregate GPU processing throughput in time frames of jobs running on EPN (4 MI50 GPUs) and at NERSC (1 A100 GPU) performing TPC and TPC + ITS tracking on GPU versus time frame size Beyond improving the GPU usage in the EPN farm for offline processing, ALICE also aims …
Figure 5
Figure 5. Figure 5: GPU idle time of offline reconstruction jobs on EPN and at NERSC against the runtime in the job This is visible as significant GPU idle time, with an onset once the time frame in flight limit is reached. The job on the EPN using 4 GPUs has much faster aggregate GPU pro…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 10 canonical work pages

  1. [1]

    The ALICE experiment at the CERN LHC

    ALICE Collaboration, “The ALICE experiment at the CERN LHC”, JINST, vol. 3, pp. S08002 (2008)

    work page 2008

  2. [2]

    ALICE upgrades during the LHC Long Shutdown 2

    ALICE Collaboration, “ALICE upgrades during the LHC Long Shutdown 2”, JINST, vol. 19, pp. P05062 (2015)

    work page 2015

  3. [3]

    Technical Design Report for the Upgrade of the Online-Offline Computing System

    ALICE Collaboration, “Technical Design Report for the Upgrade of the Online-Offline Computing System”, CERN-LHCC-2015-006, ALICE-TDR-019 (2015)

    work page 2015

  4. [4]

    ALICE Collaboration, Real-time data processing in the ALICE High Level Trig- ger at the LHC, Computer Physics Communications, vol. 242, pp. 25 (2019), arXiv:1812.08036

    arXiv 2019

  5. [5]

    Ronchetti et al., Efficient high performance computing with the ALICE event processing nodes GPU-based farm, Front

    F. Ronchetti et al., Efficient high performance computing with the ALICE event processing nodes GPU-based farm, Front. Phys. vol. 13, pp. 1541854, doi: 10.3389/fphy.2025.1541854

    work page arXiv 2025

  6. [6]

    The O 2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3

    G. Eulisse and D. Rohr for the ALICE Collaboration, “The O 2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3”, EPJ Web of Conf., vol. 295, pp. 05022, Proceedings of CHEP 2023 (2024), arXiv:2402.01205

    arXiv 2024

  7. [7]

    Global Track Reconstruction and Data Com- pression Strategy in ALICE for LHC Run 3

    D. Rohr for the ALICE Collaboration, “Global Track Reconstruction and Data Com- pression Strategy in ALICE for LHC Run 3”, Proceedings of CTD2019 (2019), arXiv:1910.12214

    arXiv 2019

  8. [8]

    GPU performance in Run3 ALICE online/offline reconstruction

    G. Cimador for the ALICE Collaboration, “GPU performance in Run 3 ALICE on- line/offline reconstruction”, Proceedings of 12th Large Hadron Collider Physics Con- ference — PoS(LHCP2024), vol. 478, pp. 153 (2025), arXiv:2411.02206

    work page Pith review arXiv 2025

Show all 14 references
  1. [9]

    Extending ALICE’s GPU tracking capabilities: Towards a comprehensive accelerated barrel reconstruction

    M. Concas, D. Rohr for the ALICE Collaboration, “Extending ALICE’s GPU tracking capabilities: Towards a comprehensive accelerated barrel reconstruction”, EPJ Web of Conf., vol. 337, pp. 01019, Proceedings of CHEP 2024 (2025)

    2025

  2. [10]

    Usage of GPUs for online and offline reconstruction in ALICE in Run 3

    D. Rohr for the ALICE Collaboration, “Usage of GPUs for online and offline reconstruction in ALICE in Run 3”, PoS, vol. ICHEP2024, pp. 1012, (2024), arXiv:2502.09138

    2024 arXiv

  3. [11]

    Portable and Vendor-Independent Low-Level Programming and Performance Benchmarking for Graphics Cards and Processors

    D. Rohr, V . Lindenstruth, “Portable and Vendor-Independent Low-Level Programming and Performance Benchmarking for Graphics Cards and Processors”, Proceedings of 2017 IEEE 19th HPCCWS (2017), doi: 10.1109/HPCCWS.2017.00007

    2017

  4. [12]

    Improvements of the GPU Processing Frame- work for ALICE

    D. Rohr for the ALICE Collaboration, “Improvements of the GPU Processing Frame- work for ALICE”, EPJ Web of Conf., vol. 337, pp. 01362, Proceedings of CHEP 2024 (2025), arXiv:2511.17018

    2025

  5. [13]

    Automated tuning of GPU kernel parameters via RunTime Compilation for the ALICE Online reconstruction

    G. Cimador„ “Automated tuning of GPU kernel parameters via RunTime Compilation for the ALICE Online reconstruction ”, Proceedings of CHEP 2026, to appear in this volume

    2026

  6. [14]

    The end of the x86 dominance - orchestrating the heterogeneous Grid

    M. Storetvedt, “The end of the x86 dominance - orchestrating the heterogeneous Grid”, Proceedings of CHEP 2024, to appear in this volume

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.