{"id":"53105ce0-12c9-4885-bf23-b62054385119","arxiv_id":"2607.26674","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"HORSES3D was GPU-ported with OpenACC, achieving near-ideal scaling above ~16–20k elements per GPU and running a 2,048-GPU, 10.7B-DOF High-Lift Common Research Model simulation.","lead":"Researchers ported the open-source HORSES3D CFD solver to run on NVIDIA GPUs using OpenACC directives. On 2,048 H100 GPUs it ran a 10.7-billion-degree-of-freedom wing-body simulation, demonstrating that high-order aerodynamic solvers can scale to modern supercomputers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2048-GPU / 10.7B-DOF flagship run is absent from the scaling curves; \"efficient to 2048\" rests on baseline-relative extrapolation.","rationale":"The reader's weakest assumption identifies the multi-GPU baseline normalization, which is a real and related issue. I agree that normalizing strong scaling to a multi-GPU baseline absorbs baseline overhead and can make the reported efficiencies optimistic. However, the more direct gap in the evidence for the headline claim is that the specific largest simulation cited in the abstract - h3, P=7, 2048 GPUs, 10.7e9 DOF - is not included in any strong-scaling curve; the h3 panel of Figure 11 only displays low-order data, and the text gives only a P=2 efficiency at 2048 GPUs. This means the central 'scales efficiently to 2048 GPUs' sentence is supported by extrapolation from smaller/low-order configurations rather than by a measured efficiency for the flagship run. The TGV section does provide a reasonable single-GPU-to-multi-GPU comparison for periodic structured problems, which strengthens the overall porting claim, and the verification/validation sections show the GPU port is numerically sound. The OpenACC implementation is concrete and the repository is referenced. These factors keep the paper at CONDITIONAL rather than REJECT: the scaling claim is plausible but the strongest form is not yet supported by the released evidence. The proposed table-based reanalysis would settle whether the 2048-GPU efficiency is actually in the claimed range.","tokens_in":25148,"tokens_out":11370,"duration_ms":119615,"concrete_test":"Publish a table of walltimes/PIDs for all h3 runs used in Figure 11, including the h3/P=7 run at 1024 and 2048 GPUs. Recompute the strong-scaling efficiency for the 1024->2048 segment relative to the lowest feasible GPU count for that specific P=7 case, and also compute each point's absolute PID relative to the single-GPU-equivalent PID plateau from Figure 10/Section 5.2.1 (using the CRM-appropriate reference). If the P=7/2048 efficiency falls materially below the values implied by the figure (e.g., below ~70%) or the baseline-relative and absolute-relative efficiencies diverge by more than about 10 percentage points, the \"efficiently to 2048\" claim should be qualified or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that HORSES3D scales efficiently to 2048 GPUs on the 10.7e9-DOF CRM case. The paper's own data for that flagship case are incomplete. Section 5.2.2 states the h3/P=7/2048 run was performed, but Figure 11 shows h3 strong-scaling points only for low polynomial orders (P=1-3 in the panel), and the reported 2048-GPU efficiency for h3 is ~53% for P=2. The P=7/2048 point does not appear in any scaling curve, so its efficiency is never actually measured. In addition, every strong-scaling curve is normalized to its smallest multi-GPU configuration, so the leftmost point is 100% by construction and any communication/load-imbalance overhead present at the baseline is invisible. The TGV single-GPU comparison in Section 4.2.2 partially mitigates this for structured periodic meshes, but the CRM case uses unstructured partitions and complex boundary treatment, so the baseline overhead can differ. Without either the missing P=7/2048 data point or a single-GPU-equivalent PID reference for the CRM case, \"scales efficiently to 2048 GPUs\" is an extrapolation rather than a directly demonstrated result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HORSES3D-GPU, an OpenACC-based multi-GPU implementation of the high-order discontinuous Galerkin solver HORSES3D. It verifies the port with a manufactured-solutions convergence study and validates it on the Taylor–Green vortex and a wall-modeled turbulent channel flow. Performance is assessed on MareNostrum 5's NVIDIA H100 partition. For the TGV benchmark, the authors report single-GPU PID data, identify a practical regime of 16,000–20,000 elements per GPU, and show strong/weak scaling on up to 256 GPUs. For the CRM-HL wing–body configuration, they report PID across four meshes and polynomial orders P=1–7, strong scaling on up to 2048 GPUs, and a flagship simulation of the h3 mesh at P=7 with approximately 10.7e9 DOF. The central claims are that HORSES3D preserves its single-GPU performance characteristics on industrial geometries and scales efficiently to 2048 GPUs.","tokens_in":25348,"tokens_out":9834,"duration_ms":98591,"significance":"If fully established, the paper would be a useful contribution to GPU-based high-order CFD: it demonstrates that an OpenACC port can retain a single Fortran source, provides an open-source implementation, uses a clearly defined performance metric, and reports data on a production-scale high-lift configuration. The authors are also explicit about several limitations, including the multi-GPU baseline normalization and the memory-bound nature of the kernels. However, the headline scaling claim rests on a data gap: the 2048-GPU P=7 flagship run is mentioned but no measured speedup or efficiency is reported for it, and the CRM strong-scaling curves are normalized to a multi-GPU baseline, so the reported percentages are not absolute parallel efficiencies.","major_comments":[{"comment":"The claimed 2048-GPU result for the largest simulation is not present in the scaling data. The h3 panel of Figure 11 shows strong-scaling curves only for P=1, P=2, and P=3, while the text states that the h3/P=7/2048 run (≈10.7e9 DOF) was performed. No wall-time, speedup, PID, or efficiency is reported for this run; the only 2048-GPU efficiency given is about 53% for P=2. Consequently, the abstract's statement that the solver 'scale[s] efficiently to 2,048 GPUs' is an extrapolation from lower-order curves rather than a directly measured property of the flagship run. Please either report the timing/speedup/PID for the h3/P=7/2048 point or restrict the scaling claim to the polynomial orders actually shown and describe the P=7/2048 run as a capability demonstration.","section":"Section 5.2.2 / Figure 11"},{"comment":"The strong-scaling speedups in Figure 11 are normalized to the smallest multi-GPU configuration that can fit each mesh, so the leftmost point is 100% by construction and any communication or load-imbalance overhead present at the baseline is invisible. The paper acknowledges this in Section 5.2.1, but Section 5.2.2 then interprets the ratios as 'efficiency' (e.g., 53% at 2048 GPUs for h3/P=2). For the CRM case there is no single-GPU anchor; the TGV single-GPU comparison in Section 4.2.2 is for a structured periodic mesh and cannot be transferred directly to the unstructured CRM with complex boundary treatment. Please provide a single-GPU-equivalent reference for CRM, or an independent estimate of the baseline parallel efficiency, or consistently label the values as 'speedup relative to the multi-GPU baseline' rather than as absolute efficiency.","section":"Section 5.2.1 / Eq. (10)"},{"comment":"The comparison of CRM PID values with the single-GPU TGV PID values is used to support the claim that the solver 'preserves its performance characteristics.' Because the CRM PID is obtained from multi-GPU runs and rescaled under an ideal-strong-scaling assumption, it cannot separate the cost of complex boundary treatment from parallel-efficiency losses due to MPI communication, load imbalance, or network contention. The close agreement between CRM and TGV PID therefore establishes only that the combined effect remains comparable, not that the single-GPU performance characteristics are preserved. Please state this inferential limitation explicitly when drawing this conclusion.","section":"Section 5.2.1 / Figure 10"}],"minor_comments":[{"comment":"The weak-scaling baseline is described as a single 4-GPU node, yet the text and figure also include points for 1 and 2 GPUs and report efficiencies above 1 for those points. Please clarify the reference configuration and state whether the problem size is scaled from 1 GPU or from 4 GPUs.","section":"Section 4.2.2 / Figure 7"},{"comment":"The opening sentence says the figure shows 'exponential convergence, for h- and p-refinement.' h-refinement with fixed polynomial order gives algebraic convergence; the following sentences correctly distinguish the two behaviors. The opening should be reworded to avoid this imprecision.","section":"Section 3.1 / Figure 1"},{"comment":"The wall-model appendix states that in some MPI configurations the wall-model input may correspond to the previous Runge–Kutta stage, 'at the cost of a slight temporal lag.' This is an acknowledged approximation in the wall-modeled LES validation; please quantify the lag or at least discuss its expected effect on the reported mean-velocity profiles.","section":"Appendix B"},{"comment":"The performance measurements appear to be single runs on a shared HPC system, with no repeated runs or variance estimates. A brief statement about run-to-run variability and whether the reported times are representative would improve reproducibility.","section":"Sections 4.2 and 5.2"},{"comment":"Some efficiency percentages quoted in the text do not obviously match the annotations in Figure 11. For example, the text reports 56% at 1024 GPUs for h1/P=4, while the figure panel appears to contain different annotation values. Please reconcile the text and figure.","section":"Figure 11 / Section 5.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is close to being a solid contribution. The main gap is that the headline 2048-GPU/P=7 run is not accompanied by any measured performance figure; if the authors can supply that data point and tighten the baseline-normalization language, I would support acceptance. The wall-model temporal lag noted in Appendix B also deserves more attention during revision, though it is not central to the scaling claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, honest OpenACC port of HORSES3D to GPUs, with clean verification (MMS, TGV, channel flow) and a useful performance dataset on MareNostrum 5. The central claim—efficient scaling to 2048 GPUs and 10.7B DOF on the CRM-HL—is plausible but not fully demonstrated as stated. The P=7/2048 run is described in the text but never appears in the strong-scaling curves; the 2048 efficiency number quoted in the text (~53%) is for P=2 on h3. So the flagship point is an extrapolation from lower-order data, not a directly measured efficiency.\n\nCredit where due: the porting strategy (gang per element, vector per node, fused advective/viscous kernels, avoiding atomics in the split-form) is described concretely and matches what other GPU DGSEM solvers do; the paper says so. The threshold finding—≥~16–20k elements/GPU for near-optimal efficiency—is consistent across TGV, single-GPU, and CRM data, and it is an observed regularity, not a fitted parameter. The paper also explicitly acknowledges the multi-GPU baseline normalization in Section 5.2.1, which mitigates the worry that they are hiding it. The clustering-based weak-scaling analysis (grouping runs by elements/GPU) is a genuinely nice way to summarize a large dataset.\n\nSoft spots, in proportion: (1) strong-scaling speedups are all normalized to a multi-GPU baseline, so the leftmost point is 100% by construction and any overhead at that baseline is invisible. The single-GPU TGV comparison partially fixes this for periodic structured meshes, but the CRM case is unstructured and has more complex boundary treatment, so the baseline overhead could differ. (2) No error bars—all timings appear to be single runs. (3) The raw data and exact code version are not released, so the numbers can't be independently reproduced. (4) As above, the missing P=7/2048 data point. None of these are load-bearing flaws; they are gaps in evidence that a revision can close.\n\nWho this is for: CFD/HPC researchers porting DG or spectral-element solvers to GPUs, and anyone planning large-scale GPU runs and wanting a sanity-check range for elements-per-GPU. It deserves a serious referee. I'd send it to review with a request to include the P=7/2048 point (or explicitly note it was run but omitted from the curves), add a sentence or two on run-to-run variability, and soften the phrasing so claims match the measured data.","headline":"Honest, workmanlike OpenACC port with a real scaling claim that at the flagship 2048-GPU point is partly an extrapolation from lower-order data.","tokens_in":26018,"tokens_out":2430,"would_cite":true,"duration_ms":23312,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65M60","65Y05","76M10","76F65"],"pacs":[],"model":"deepseek-v4-flash","headline":"A directive-based OpenACC port of the HORSES3D discontinuous Galerkin solver scales near-ideally to 2,048 GPUs and reaches 10.7 billion degrees of freedom on a realistic high-lift wing-body configuration.","keywords":["high-order methods","discontinuous Galerkin","GPU acceleration","OpenACC","multi-GPU scaling","compressible Navier-Stokes equations","billion-degree-of-freedom simulations","DGSEM"],"falsifier":"Take a benchmark case that fits on one GPU, such as 16,384 hexahedral elements at polynomial order 5, measure the efficiency metric for 1, 2, 4, 8, up to 256 GPUs, and compute strong-scaling efficiency using the single-GPU run as the reference; if efficiency at 256 GPUs is below about 80% even though each GPU still has more than 16,000 elements, then the near-ideal scaling reported in the paper is an artifact of measuring relative to a multi-GPU baseline rather than to a single GPU.","tokens_in":24976,"feed_emoji":"🖥️","tokens_out":6146,"duration_ms":61264,"temperature":0.7,"pith_summary":"The paper reports a GPU port of HORSES3D, an open-source high-order discontinuous Galerkin solver for compressible flow, achieved by adding OpenACC directives to the existing Fortran code rather than rewriting it. The port maps each mesh element to a GPU thread block and nodal operations to vector lanes, keeping data resident on the GPU. On a canonical transition-to-turbulence benchmark and on a realistic high-lift wing-body configuration with up to 20.8 million hexahedral elements, the solver shows near-ideal strong and weak scaling up to 2,048 GPUs, with the largest run reaching about 10.7 billion degrees of freedom. The central practical claim is that performance is controlled by the number of elements per GPU: once each device holds roughly 16,000–20,000 elements, efficiency saturates and scaling becomes near-linear, even for complex geometries.","feed_headline":"High-order CFD solver scales to 2,048 GPUs","feed_subtitle":"Directive-based GPU port keeps near-ideal speedups once each GPU holds at least 16,000–20,000 elements.","key_machinery":"The central mechanism is the element-local structure of nodal DGSEM, exploited by mapping the element loop to OpenACC gangs (thread blocks) and the nodal quadrature loop to vector lanes, with register-resident accumulation to avoid global-memory round trips. For split-form two-point-flux schemes, the paper deliberately abandons CPU-side flux-symmetry reuse: because scattering one flux to two nodes would require atomic updates across lanes, it recomputes each two-point flux for both orderings and keeps accumulation private per lane, fusing advective and viscous contributions into one kernel. The measured 'elements per GPU' threshold then acts as a universal load parameter that predicts when c","core_discovery":"The discovery is that a high-order DGSEM CFD code can be made GPU-resident by a relatively low-touch OpenACC port, without changing the numerical formulation, and still scale to thousands of GPUs. The key evidence is that solver efficiency, measured as wall-time per degree of freedom per Runge-Kutta stage, improves with polynomial order, saturates when the load exceeds about 16,000 to 20,000 elements per GPU, and remains within a factor of roughly two of single-GPU performance on an industrial high-lift wing-body configuration. At 2,048 GPUs with polynomial order 7 and a 20.8-million-element mesh, the solver sustains strong-scaling efficiencies around 80% or better while the load stays above","pith_inferences":["Because strong-scaling baselines are multi-GPU runs, true single-GPU-relative speedups could be lower than reported; measuring against a single GPU would yield the honest efficiency curve.","The elements-per-GPU rule could be turned into a portable cost model: runtime per step might be predicted from element count divided by GPU count, independent of total problem size, which would help scheduler-aware resource allocation.","The threshold is likely to shift with hardware generations: accelerators with more memory bandwidth or more compute per device may lower or raise it, and the same methodology could re-measure it on other architectures.","The paper's positioning as a data engine for AI-era CFD would be strengthened by a throughput comparison against CPU baselines, including energy cost; the paper reports efficiency metric but not power or CPU speedup."],"forward_implications":["At polynomial orders 5–7 and loads above about 20,000 elements per GPU, near-ideal strong scaling is maintained up to roughly 1,024 GPUs; 2,048 GPUs remain usable with lower efficiency.","Higher polynomial order yields better GPU efficiency for a fixed number of degrees of freedom, so p-refinement is computationally cheaper than h-refinement on this architecture.","The same OpenACC codebase can target CPU and GPU with identical numerical results, verified by manufactured-solution convergence and turbulent-flow validation.","The elements-per-GPU threshold is a practical configuration guideline: below roughly 16,000 elements per device, communication dominates and efficiency degrades sharply.","GPU-resident execution, with all data on the device and minimal host-device transfers, is sufficient to reach billion-degree-of-freedom industrial-scale LES within current HPC allocations."],"fun_headline_variants":["OpenACC port takes DG CFD to 2,048 GPUs","Near-ideal scaling past 16k elements per GPU","Billion-DOF high-order CFD on 2,048 GPUs","DG solver sustains ~80% efficiency on 2k GPUs","Low-touch GPU port scales high-order CFD"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim of near-ideal scaling rests on assuming that the smallest multi-GPU run used as the speedup baseline already achieves perfect parallel efficiency; if that baseline carries significant communication or load-imbalance costs, reported speedups overstate true scalability.","fun_headline_variants_meta":{"raw":{"variants":["OpenACC port takes DG CFD to 2,048 GPUs","Near-ideal scaling past 16k elements per GPU","Billion-DOF high-order CFD on 2,048 GPUs","DG solver sustains ~80% efficiency on 2k GPUs","Low-touch GPU port scales high-order CFD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2653,"prompt_tokens":827,"completion_tokens":1826,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1739}},"tokens_in":571,"tokens_out":1826,"duration_ms":13750,"temperature":1.0,"reasoning_tokens":1739,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:00:32.154610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark case that fits on one GPU, such as 16,384 hexahedral elements at polynomial order 5, measure the efficiency metric for 1, 2, 4, 8, up to 256 GPUs, and compute strong-scaling efficiency using the single-GPU run as the reference; if efficiency at 256 GPUs is below about 80% even though each GPU still has more than 16,000 elements, then the near-ideal scaling reported in the paper is an artifact of measuring relative to a multi-GPU baseline rather than to a single GPU.","supporting_citations":[],"review_version":1}