Pith. sign in

REVIEW 3 major objections 4 minor 16 references

Leveraging Caliper and Benchpark to Analyze MPI Communication Patterns: Insights from AMG2023, Kripke, and Laghos

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding communication regions gives Caliper per-phase MPI message statistics it previously could not capture, and applying them to three HPC benchmarks exposes architecture-dependent bottlenecks.

desk verdict The paper adds a genuinely useful capability to Caliper—per-region MPI message statistics—but the lack of validation and shipped artifacts is the soft spot. read the letter →

arxiv 2507.22372 v1 pith:7BNLNBHR submitted 2025-07-30 cs.DC

Grace Nansamba , Evelyn Namugwanya , David Boehme , Dewi Yokelson , Riley Shipley , Derek Schafer , Michael McKinsey , Olga Pearce
show 1 more author
Anthony Skjellum
This is my paper · ORCID
classification cs.DC
keywords MPIcommunicationpatternsCaliperregionshaloexchangeperformanceprofilingscalabilityanalysisAMG2023Benchpark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces communication regions into the Caliper profiling tool: source-code markers that bracket the MPI calls belonging to one logical communication phase, such as a halo exchange, together with a profiler that records statistics about the messages and the participating ranks at the end of each region. The authors claim that Caliper previously could not capture metrics about the data being communicated or the MPI processes involved, and that this addition fills that gap without requiring full trace collection. They demonstrate the feature by annotating three Benchpark benchmarks, AMG2023, Kripke, and Laghos, and running weak- and strong-scaling studies on CPU and GPU systems. The reported data show that Kripke's sweep communication is light relative to its solve loop, that AMG2023's finest multigrid levels carry the largest byte volumes while its coarse levels involve many more ranks on CPUs than on GPUs, and that Laghos's message rate rises steeply under strong scaling while per-process throughput falls. If the approach works as described, performance engineers can attribute MPI traffic to named phases in complex applications and compare communication behavior across architectures.

What carries the argument

The central object is the communication region, an annotated span of code delimited by CALI_MARK_COMM_REGION_BEGIN and CALI_MARK_COMM_REGION_END that groups MPI calls forming one logical communication phase, such as a halo exchange, a transport sweep, or a matrix-vector-product exchange. A communication pattern profiler runs at region end and aggregates, per region instance, min/max counts of sends and receives, counts of distinct source and destination ranks, min/max bytes sent and received, and collective-call counts, using Caliper's existing MPI call interception via PMPI or GOTCHA. This machinery's work is attribution: it lets an analyst separate one logical phase's MPI traffic from another and from unrelated communication, so scaling trends and architecture differences can be read off directly from per-region aggregates.

What would settle it

Run the same annotated benchmarks while also recording a complete MPI trace, or counting every MPI call with the MPI profiling interface, then compare the trace's per-phase send counts and byte totals with Caliper's per-region statistics; any mismatch shows that the marker boundaries do not align with the intended MPI calls.

Watch

Extended reading notes

Core claim

At the core of the paper is the claim that logical communication patterns can be turned into first-class profiling objects. By placing CALI_MARK_COMM_REGION_BEGIN and CALI_MARK_COMM_REGION_END around a group of MPI calls, an application defines a named communication region, and Caliper's new communication pattern profiler computes per-region statistics when the region closes: min/max send and receive counts, counts of distinct destination and source ranks, min/max bytes sent and received, and collective-call counts. Because the region markers are placed by the application developer around the calls that constitute a phase, the resulting metrics describe that phase rather than individual functions or the whole run. The paper's demonstration on AMG2023, Kripke, and Laghos shows that this isolation reveals communication behaviors that were previously invisible at profile granularity; for example, AMG2023's coarsest multigrid levels can involve more than one hundred source ranks at 512 processes on the CPU system even though the byte volume at those levels is small, and Laghos's halo exchange cost stays roughly flat while its message rate climbs as the problem is strong-scaled to 896 processes.

Load-bearing premise

The load-bearing premise is that the manually inserted BEGIN/END markers in AMG2023, Kripke, and Laghos enclose exactly the MPI calls that form the named region, so the per-region aggregates describe the intended phase and nothing else.

Editorial extensions

If this is right

  • Caliper users can now obtain per-region MPI message statistics, including message counts, byte volumes, and source/destination rank counts, from ordinary profiled runs rather than from traces.
  • Annotated halo exchanges and other named phases can be compared across process counts and systems, letting scaling studies attribute bottlenecks to a specific communication phase instead of the whole application.
  • AMG2023's multigrid levels can be analyzed separately, showing that the finest levels dominate byte volume while coarse levels can involve unexpectedly many ranks on CPUs at high process counts.
  • Kripke's sweep communication is a small fraction of runtime relative to its solve loop, suggesting that optimizing Kripke should focus on computation on CPUs and on hiding message latency on GPUs.
  • Laghos under strong scaling shows a rising message rate and a falling per-process throughput, with a sharp bandwidth drop at 896 processes, indicating communication overhead that does not shrink with problem size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If communication-region markers become widespread, per-phase MPI message counts and byte totals could serve as regression-test metrics in continuous benchmarking, flagging code changes that alter communication behavior without requiring trace collection.
  • Editorial inference: Because the profiler intercepts MPI calls rather than assuming point-to-point send/receive behavior, the same region mechanism could plausibly capture one-sided MPI and GPU-aware communication, though the paper does not test those cases.
  • Editorial inference: The AMG2023 CPU result that coarse levels involve many source ranks suggests a testable optimization hypothesis: modifying coarse-grid aggregation or using communication-avoiding coarse solves should reduce coarse-level rank participation and improve CPU weak scaling.
  • Editorial inference: Combining communication regions with compute-region annotations could quantify communication-computation overlap or its absence, which the paper mentions as an optimization target but does not measure directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces 'communication regions' into the Caliper HPC profiling tool: new markers (CALI_MARK_COMM_REGION_BEGIN/END) that can be placed around groups of MPI calls forming a logical communication phase, plus a communication-pattern profiler that computes per-region statistics on message counts, sizes, and source/destination ranks. The authors manually annotate three Benchpark benchmarks (AMG2023, Kripke, Laghos), run weak- and strong-scaling experiments on two HPC systems (Dane CPU and Tioga GPU), and use Caliper with Thicket to produce per-region, per-multigrid-level, and bandwidth/message-rate analyses. The central claim is that these new regions make it possible to capture previously unavailable MPI communication statistics for logical communication phases and to reveal application-specific communication bottlenecks and scaling behaviors.

Significance. If the new Caliper feature and the region annotations are correct, the paper would make a practical contribution: it gives performance engineers a lightweight way to measure message counts, data volumes, and source/destination-rank structure for logical communication phases such as halo exchanges, rather than only aggregate MPI call times. The paper's strengths are its use of established, widely used tooling (Caliper, Thicket, Benchpark), its concrete cross-architecture scaling comparison, and the potentially useful per-multigrid-level view in AMG2023. However, the central experimental support is conditional: the paper does not currently provide validation of the new profiler or of the manual marker placements, and the experimental conclusions are drawn from single runs without error characterization. The reported insights are therefore plausible but not yet fully substantiated.

major comments (3)
  1. [III-B, Table IV] The paper's central claim—that the new communication-region markers and profiler produce correct per-region MPI statistics—is not validated. Section III-B states the markers are placed manually, but no patch, annotated source, microbenchmark, or independent cross-check is provided; an MPI call outside a marked region or a misplaced boundary would silently change every entry in Table IV and every trend in Sections IV–V. A validation experiment (e.g., a microbenchmark with known message counts, or a comparison against direct MPI tracing) or a released artifact is required before the reported insights can be trusted.
  2. [III-C, Tables III–IV] All experiments appear to be single measurements, with no repetitions, error bars, or confidence intervals reported. Since Section IV draws monotonic-scaling conclusions (e.g., 'Kripke shows constant communication per rank on Dane,' AMG2023's 'steadily increasing communication,' Laghos's decreasing bytes sent) and Section V compares bandwidth and message rates, the absence of variance information leaves open the possibility that reported trends are measurement noise. At least three repetitions per configuration, or an explicit statement of variability, is needed.
  3. [IV-B, Figures 2–3] The per-multigrid-level analysis in Figures 2 and 3 requires a mechanism that Section III does not describe. The only markers introduced are generic CALI_MARK_COMM_REGION_BEGIN/END around halo exchange, Sweep, and MatVecComm regions; the paper does not say whether each MG level is a separate marked region or how the profiler attributes statistics to levels. Without this information, the level-wise claims about source-rank participation and bytes sent cannot be reproduced or assessed.
minor comments (4)
  1. [II, IV, affiliations] Typographical errors: 'funcionality' in Section II, 'Lahgos' in the opening of Section IV, and 'Tennesse' in the author affiliations should be corrected.
  2. [III, Table I] The 'Min/Max' wording in Table I is ambiguous: it should state whether the extrema are taken over processes within a single region instance, over repeated iterations of the same region, or over all captured region instances.
  3. [III-D] The paper should specify the exact Caliper version or commit implementing the communication-region markers, and where the public API and profiler are documented, to make the new feature reproducible.
  4. [IV-C, Figure 4] The statement 'We observe two levels (green dots) in the plot; for the Broadcast and Reduction phases of the timestep loop' is unclear; please define 'levels' and describe what the green dots in Figure 4 represent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports directly measured MPI statistics from newly annotated communication regions, with no prediction or derivation reducing to its inputs.

full rationale

The paper is an observational tooling and measurement study. Its contributions are a new Caliper feature (communication-region markers plus a communication-pattern profiler), manually placed annotations in three benchmarks, and scaling results derived from instrumented runs. There is no fitted model, no equation that is solved to produce a claimed prediction, and no uniqueness theorem invoked to force a choice. The reported metrics (bytes sent, message counts, source/destination ranks) are direct measurements from the profiler, not outputs of a derivation whose inputs include those same metrics. The manual placement of CALI_MARK_COMM_REGION_BEGIN/END markers in AMG2023, Kripke, and Laghos (Section III-B) is an input definition rather than a circular step: the region is what the markers enclose, and the paper does not claim the region boundaries are derived from the measured statistics. The paper does cite the authors' own prior tools (Caliper [2], Benchpark [5], Thicket [6]), but these citations provide the substrate and context for the work; they are not load-bearing evidence for the central empirical claim, which stands on the runs shown in Tables IV and Figures 1-6. The skeptic concern about marker correctness and profiler validation is a correctness or artifact-quality issue, not a circularity issue under the specified criteria. Therefore no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters or invented entities. The experimental configurations (problem sizes, ranks) are inputs, not fitted quantities. The central claim assumes the correctness of MPI interception, the annotation placement, and the Benchpark build reproducibility.

assumptions (3)
  • domain assumption Caliper's PMPI/GOTCHA interception accurately captures all MPI calls executed inside an annotated communication region.
    The new profiler computes per-region aggregates from intercepted MPI calls; if interception misses calls, the metrics are incomplete. See Section III.
  • domain assumption The manual annotations in AMG2023, Kripke, and Laghos delimit exactly the intended communication phases (halo exchange, MatVecComm, sweep).
    All per-region trends in Section IV depend on markers placed by the authors; misplacement changes the results. See Section III-B.
  • domain assumption Benchpark, Spack, and Ramble reproduce the same benchmark builds and configurations on Dane and Tioga.
    The cross-system comparison assumes equivalent builds and inputs; the paper provides no reproducibility artifacts or commit hashes. See Section III-D.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Caliper and Benchpark to Analyze MPI Communication Patterns: Insights from AMG2023, Kripke, and Laghos." pith.science (2026). https://pith.science/paper/7BNLNBHR

@misc{pith2026250722372,
  author       = {Pith},
  title        = {Pith review of: Leveraging Caliper and Benchpark to Analyze MPI Communication Patterns: Insights from AMG2023, Kripke, and Laghos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BNLNBHR}},
  note         = {Machine review of arXiv:2507.22372}
}
read the original abstract

We introduce ``communication regions'' into the widely used Caliper HPC profiling tool. A communication region is an annotation enabling capture of metrics about the data being communicated (including statistics of these metrics), and metrics about the MPI processes involved in the communications, something not previously possible in Caliper. We explore the utility of communication regions with three representative modeling and simulation applications, AMG2023, Kripke, and Laghos, all part of the comprehensive Benchpark suite that includes Caliper annotations. Enhanced Caliper reveals detailed communication behaviors. Using Caliper and Thicket in tandem, we create new visualizations of MPI communication patterns, including halo exchanges. Our findings reveal communication bottlenecks and detailed behaviors, indicating significant utility of the special-regions addition to Caliper. The comparative scaling behavior of both CPU and GPU oriented systems are shown; we are able to look at different regions within a given application, and see how scalability and message-traffic metrics differ.

Figures

Figures reproduced from arXiv: 2507.22372 by the authors.

Figure 1
Figure 1. Average time per rank (seconds) for Kripke on Dane (left) and Tioga (right) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Bytes sent per level for AMG2023 on CPU-Dane(left) and GPU-Tioga(right) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Average ranks per level for AMG2023 on Dane (left) and Tioga (right) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average time per rank for Laghos on CPU-Dane system [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Bytes per second for AMG2023, Kripke and Laghos on CPU [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Bytes per second for AMG2023 and Kripke on GPU [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    Message Passing Interface Forum, MPI: A Message-Passing Interface Standard Version 4.1, Nov. 2023. [Online]. Available: https://www.mpi- forum.org/docs/mpi-4.1/mpi41-report.pdf

    work page 2023

  2. [2]

    Caliper: Performance introspection for hpc software stacks,

    D. Boehme, T. Gamblin, D. Beckingsale, P. Bremer, A. Gimenez, M. Leg- endre, O. Pearce, and M. Schulz, “Caliper: Performance introspection for hpc software stacks,” in Proceedings of SC 2016 , ser. International Conference for High Performance Computing, Networking, Storage and Analysis, SC. IEEE Computer Society, Jul. 2016, pp. 550–560, publisher Copyrig...

    work page 2016

  3. [3]

    Kripke-a massively parallel transport mini-app,

    A. J. Kunen, T. S. Bailey, and P. N. Brown, “Kripke-a massively parallel transport mini-app,” Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States), Tech. Rep., 6 2015

    work page 2015

  4. [4]

    AMG2023: Algebraic multigrid solver,

    L. L. N. Laboratory, “AMG2023: Algebraic multigrid solver,” https://github.com/LLNL/AMG2023, 2023, accessed: [12/4/2024]

    work page 2023

  5. [5]

    Towards Collaborative Continuous Benchmarking for HPC,

    O. Pearce, A. Scott, G. Becker, R. Haque, N. Hanford, S. Brink, D. Jacobsen, H. Poxon, J. Domke, and T. Gamblin, “Towards Collaborative Continuous Benchmarking for HPC,” in Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis , ser. SC-W ’23. New York, NY , USA: Association for C...

    work page arXiv 2023

  6. [6]

    Thicket: Seeing the Performance Experiment Forest for the Individual Run Trees,

    S. Brink, M. McKinsey, D. Boehme, C. Scully-Allison, I. Lumsden, D. Hawkins, T. Burgess, V . Lama, J. L ¨uttgau, K. E. Isaacs, M. Taufer, and O. Pearce, “Thicket: Seeing the Performance Experiment Forest for the Individual Run Trees,” in Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing, ser. HPDC ’23. ...

    work page arXiv 2023

  7. [7]

    Laghos: Lagrangian High-Order Hydrodynamics Miniapp – README,

    CEED, “Laghos: Lagrangian High-Order Hydrodynamics Miniapp – README,” https://github.com/CEED/Laghos/blob/master/README.md, 2024, accessed: 2025-05-06

    work page 2024

  8. [8]

    Gotcha: An function-wrapping interface for hpc tools,

    D. Poliakoff and M. LeGendre, “Gotcha: An function-wrapping interface for hpc tools,” in Programming and Performance Visualization Tools , A. Bhatele, D. Boehme, J. A. Levine, A. D. Malony, and M. Schulz, Eds. Cham: Springer International Publishing, 2019, pp. 185–197

    work page 2019

Show all 16 references
  1. [9]

    The TAU parallel performance system,

    S. S. Shende and A. D. Malony, “The TAU parallel performance system,” The International Journal of High Performance Computing Applications , vol. 20, pp. 287 – 311, 2006

    2006

  2. [10]

    Advances in the TAU performance system,

    A. D. Malony, S. S. Shende, W. Spear, C. W. Lee, and S. Biersdorff, “Advances in the TAU performance system,” in Parallel Tools Workshop, 2011

    2011

  3. [11]

    Performance optimization for large scale computing: The scalable vampir approach,

    H. Brunst, M. Winkler, W. E. Nagel, and H.-C. Hoppe, “Performance optimization for large scale computing: The scalable vampir approach,” in Computational Science-ICCS 2001: International Conference San Francisco, CA, USA, May 28—30, 2001 Proceedings, Part II 1. Springer, 2001,...

    2001

  4. [12]

    HPCTOOLKIT: tools for per- formance analysis of optimized parallel programs,

    L. Adhianto, S. Banerjee, M. Fagan, M. W. Krentel, G. Marin, J. M. Mellor-Crummey, and N. R. Tallent, “HPCTOOLKIT: tools for per- formance analysis of optimized parallel programs,” Concurrency and Computation: Practice and Experience , vol. 22, 2010

    2010

  5. [13]

    Scalable communication protocols for dynamic sparse data exchange,

    T. Hoefler, C. Siebert, and A. Lumsdaine, “Scalable communication protocols for dynamic sparse data exchange,” in Proceedings of the 15th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, ser. PPoPP ’10. New York, NY , USA: Association for Computing Mac...

  6. [14]

    Analyzing and improving mpi communication performance in overcommitted virtualized systems,

    Z. Shao, Q. Wang, X. Xie, H. Jin, and L. He, “Analyzing and improving mpi communication performance in overcommitted virtualized systems,” in 2011 IEEE 19th Annual International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems , 2011, ...

    2011

  7. [15]

    hypre: A library of high performance preconditioners,

    R. D. Falgout and U. M. Yang, “hypre: A library of high performance preconditioners,” in Computational Science — ICCS 2002 , P. M. A. Sloot, A. G. Hoekstra, C. J. K. Tan, and J. J. Dongarra, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2002, pp. 632–641

    2002

  8. [16]

    MFEM: A modular finite element methods library,

    R. Anderson, J. Andrej, A. Barker, J. Bramwell, J.-S. Camier, J. Cerveny, V . Dobrev, Y . Dudouit, A. Fisher, T. Kolev, W. Pazner, M. Stowell, V . Tomov, I. Akkerman, J. Dahm, D. Medina, and S. Zampini, “MFEM: A modular finite element methods library,” Computers & Mathematics ...

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.