pith. sign in

arxiv: 2512.11142 · v2 · submitted 2025-12-11 · 🌌 astro-ph.IM

libyt: an In Situ Interface Connecting Simulations with yt, Python, and Jupyter Workflows

Shin-Rong Tsai , Hsi-Yu Schive , Matthew J. Turk This is my paper

Pith reviewed 2026-05-16 22:26 UTC · model grok-4.3

classification 🌌 astro-ph.IM
keywords in situ analysisastrophysical simulationsytPythonJupyterAMRparallel computingdata visualization
0
0 comments X p. Extension

The pith

libyt maps simulation data directly to yt frontends so existing Python post-processing scripts become in-situ analysis with two lines of change.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

libyt is a C library that lets astrophysical simulation codes connect to yt and other Python tools for analysis while the simulation is still running. Instead of writing massive datasets to disk and analyzing them later, researchers can run the same Python routines during the computation itself. The library handles the data transfer in parallel environments and supports adaptive mesh refinement codes. Users keep their existing job scripts and analysis code with almost no modification. Demonstrations cover core-collapse supernovae, galaxy simulations, and standard test problems in codes such as GAMER and Enzo.

Core claim

libyt supplies a bidirectional interface between simulation memory and Python that automatically yields patch-based AMR data on request, maps it to yt frontends, and supports both automatic routine invocation and interactive Jupyter sessions with negligible extra code.

What carries the argument

The yt frontend mapping layer that converts in-memory simulation patches into yt-readable structures so post-processing scripts run unchanged during the simulation.

Load-bearing premise

The extra time spent moving data to Python and running the analysis stays small compared with the main simulation steps.

What would settle it

Measure wall-clock time for a full production run of GAMER or Enzo with and without libyt calls; if the version using libyt takes substantially longer, the claim does not hold.

Figures

Figures reproduced from arXiv: 2512.11142 by Hsi-Yu Schive, Matthew J. Turk, Shin-Rong Tsai.

Figure 1
Figure 1. Figure 1: The overall structure of libyt and all the related components: libyt Python Module, libyt kernel, yt libyt, and jupyter libyt), that are built around it to integrate simulations with Python, yt, and Jupyter. Although libyt can run arbitrary Python scripts, we focus on its use with yt. The library provides an API for binding simulation data and functions to Python, invoking analysis routines, and enabling e… view at source ↗
Figure 2
Figure 2. Figure 2: The procedure of performing in situ analysis with libyt and the API related to each step. These steps must be followed in every round of the analysis. Note that the simulation parameters (e.g., current time), grid distribution (e.g., with AMR), and particles associated with each grid can change during simulation runtime; therefore, they are reset at the end of each simulation iteration. Some entry points (… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of magnetic field magnitude at various post-bounce times tpb in a core-collapse supernova simulation performed with GAMER. The visualization data are extracted using the save as dataset function in yt. They are later plotted and combined into a 12-second animation that shows the evolution of the magnetic field at high tem￾poral resolution. (This figure set is available as an animation.) (Miyos… view at source ↗
Figure 4
Figure 4. Figure 4: Isolated dwarf galaxy simulation performed with GAMER. Projected gas density, annotated with young stars (white dots; age ≤ 2.5 Myr) and recent supernovae (red dots; exploded within the past 2.5 Myr). The dark-blue re￾gion near (x, y) = (0.7 kpc, −0.8 kpc) marks a recent strong explosion. The animation’s duration is 47 seconds as the simulation progresses from 0 Myr to 750 Myr. (This figure is available as… view at source ↗
Figure 6
Figure 6. Figure 6: Average density distribution in an FDM simula￾tion performed with GAMER. The yellow region marks a soli￾ton core that forms spontaneously at the center of an FDM halo. The total duration of the animation is 113 seconds, which illustrates the evolution from the initial condition to the condensation of a soliton. (This figure is available as an animation.) first creates a box data object with a width of 4.5 … view at source ↗
Figure 7
Figure 7. Figure 7: The classical Sod shock tube test results in density, velocity, pressure, and internal energy in Enzo sim￾ulation at t = 250 ms. The blue dots are results from the hydrodynamics solver in Enzo. The gray line represents the exact solution at t = 250 ms, which is plotted in the static fig￾ure for comparison purposes only. All the results in chrono￾logical time steps are combined into a 3-second animation. We… view at source ↗
Figure 9
Figure 9. Figure 9: The projected density field at t = 499.98 Myr of the AGORA galaxy simulation without star particle feed￾back using Enzo. The figures show the projections in the z-direction (top) and x-direction (bottom). We combine all the outputs into a 50-second animation each. (Both figures are available as animations.) resulting in a significant reduction in disk space usage. Because converting the post-processing scr… view at source ↗
Figure 11
Figure 11. Figure 11: The particle plot with the color bar indi￾cating the dark matter clump mass at t = 499.98 Myr of the AGORA galaxy simulation without star particle feed￾back using Enzo. The particles shown include both those formed during the simulation and those present in the initial condition. The figures show the projected particles in the z-direction (top) and x-direction (bottom). We combine all the outputs into a 5… view at source ↗
Figure 12
Figure 12. Figure 12: Strong scaling of libyt. Similar tests were done in our previous paper (Tsai et al. 2023) and (Tsai et al. 2024). Here, we re-measure performance using the same test problem but with more recent versions of libyt and yt. The test compares the performance between in situ analysis with libyt and post-processing for computing 2D profiles on a GAMER dataset with five AMR levels and a total of 9.0 × 108 cells.… view at source ↗
Figure 13
Figure 13. Figure 13: The average per-process memory usage across several libyt in situ analysis stages, including: after committing the simulation settings (After commit), while running an empty Python function (Empty function), while generating 5, 000 grids of derived field data (Derived field), while redistributing 20, 000 grids (Data redistribution), run yt operation print stats (yt print stats), and after freeing every re… view at source ↗
read the original abstract

In the exascale computing era, handling and analyzing massive datasets have become extremely challenging. In situ analysis, which processes data during simulation runtime and bypasses costly intermediate disk input and output steps, offers a promising solution. We present libyt (https://github.com/yt-project/libyt), an open-source C library that enables astrophysical simulations to analyze and visualize data in parallel computation with yt or other Python packages. libyt can invoke Python routines automatically or provide interactive entry points via a Python prompt or a Jupyter Notebook. It requires minimal intervention in researchers' workflow, allowing users to reuse job submission scripts and Python routines. We describe libyt's architecture for parallel computing in high-performance computing environments, including its bidirectional connection between simulation codes and Python, and its integration into the Jupyter ecosystem. We detail its methods for reading patch-based adaptive mesh refinement (AMR) simulations and handling in-memory data with minimal overhead, and procedures for yielding data when requested by Python. We describe how libyt maps simulation data to yt frontends, allowing post-processing scripts to be converted into in situ analysis with just two lines of change. We document libyt's API and demonstrate its integration into two astrophysical simulation codes, GAMER and Enzo, using examples including core-collapse supernovae, isolated dwarf galaxies, fuzzy dark matter, the Sod shock tube test, Kelvin-Helmholtz instability, and the AGORA galaxy simulation. Finally, we discuss libyt's performance, limitations related to data redistribution, extensibility, architecture, and comparisons with traditional post-processing approaches.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript presents libyt, an open-source C library that provides an in-situ interface between parallel astrophysical simulations (GAMER, Enzo) and yt/Python/Jupyter workflows. It claims that simulation data can be mapped to yt frontends so that existing post-processing scripts can be converted to in-situ analysis with only two lines of code change, while supporting AMR patch handling, bidirectional C-Python connections, automatic or interactive Python invocation, and minimal workflow disruption.

Significance. If the overhead remains acceptable, libyt offers a practical bridge between high-performance simulation codes and the flexible yt analysis ecosystem. The demonstrations across core-collapse supernovae, dwarf galaxies, fuzzy dark matter, standard hydro tests, and AGORA galaxy runs, together with the open-source release and Jupyter integration, provide a concrete tool that could reduce disk I/O bottlenecks in exascale astrophysics workflows.

major comments (1)
  1. [performance and limitations discussion] Performance and limitations discussion: the text notes data redistribution costs as a limitation but supplies no wall-clock timings, scaling curves, or overhead fractions measured on production-scale runs (thousands of ranks) for GAMER or Enzo. Without these numbers the claim that Python invocation and redistribution remain acceptable for exascale in-situ analysis rests on an untested assumption.
minor comments (2)
  1. [yt frontend mapping description] The exact two-line modification that converts a post-processing script to in-situ use is stated but not shown with a side-by-side code example; adding one would make the central usability claim immediately verifiable.
  2. [API documentation] The API documentation section would benefit from a concise table listing the principal C functions, their arguments, and return values rather than only prose descriptions.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive summary, recognition of libyt's practical value for exascale workflows, and recommendation of minor revision. We address the single major comment below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [performance and limitations discussion] Performance and limitations discussion: the text notes data redistribution costs as a limitation but supplies no wall-clock timings, scaling curves, or overhead fractions measured on production-scale runs (thousands of ranks) for GAMER or Enzo. Without these numbers the claim that Python invocation and redistribution remain acceptable for exascale in-situ analysis rests on an untested assumption.

    Authors: We agree that quantitative performance data strengthens the manuscript. The current text discusses data redistribution as the primary cost and notes that overhead remains acceptable for the demonstrated cases, but we acknowledge the absence of explicit wall-clock timings, scaling curves, and overhead fractions at production scales. In the revised version we will add a dedicated performance subsection that reports: (1) wall-clock timings and Python invocation overhead for the GAMER core-collapse supernova and Enzo AGORA runs on up to 512 ranks, (2) scaling curves for the in-situ data-yield and redistribution steps, and (3) fractional overhead relative to the underlying hydro step. We will also state explicitly that full exascale (thousands of ranks) benchmarks lie beyond the present test suite and will be pursued in follow-up work. These additions will be placed in the performance and limitations section and will be accompanied by the raw timing tables in the supplementary material. revision: partial

Circularity Check

0 steps flagged

No circularity; direct description of software implementation

full rationale

The paper is a software engineering description of libyt's architecture, API, AMR data handling, C-Python bidirectional connection, yt frontend mapping, and integrations with GAMER/Enzo. No mathematical derivations, fitted parameters, predictions of new quantities, or load-bearing self-citations appear. The central claim (two-line script conversion via frontend mapping) follows directly from the implemented data-yielding procedures rather than reducing to a self-referential fit or imported uniqueness theorem. Performance discussion notes redistribution costs without claiming untested predictions. The work is self-contained against external benchmarks and code examples.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is a software engineering paper describing a new library; no mathematical axioms, free parameters, or invented physical entities are involved.

pith-pipeline@v0.9.0 · 5593 in / 957 out tokens · 33145 ms · 2026-05-16T22:26:23.179571+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages

  1. [1]

    2005, Visualization Handbook, ed

    Ahrens, J., Geveci, B., & Law, C. 2005, Visualization Handbook, ed. C. D. Hansen & C. R. Johnson (Burlington, MA: Elsevier Butterworth –Heinemann), 717

    work page 2005

  2. [2]

    2015, in Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV 2015), 25–29, doi: 10.1145/2828612.2828624

    Ayachit, U., Bauer, A., Geveci, B., et al. 2015, in Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV 2015), 25–29, doi: 10.1145/2828612.2828624

    work page doi:10.1145/2828612.2828624 2015

  3. [3]

    L., et al., 2014, @doi [ ] 10.1088/0067-0049/211/2/19 , https://ui.adsabs.harvard.edu/abs/2014ApJS..211...19B 211, 19

    Bryan, G. L., Norman, M. L., O’Shea, B. W., et al. 2014, The Astrophysical Journal Supplement Series, 211, 19, doi: 10.1088/0067-0049/211/2/19

    work page doi:10.1088/0067-0049/211/2/19 2014

  4. [4]

    2012, in High Performance Visualization–Enabling Extreme-Scale Scientific Insight, 357–372, doi: 10.1201/b12985

    Childs, H., Brugger, E., Whitlock, B., et al. 2012, in High Performance Visualization–Enabling Extreme-Scale Scientific Insight, 357–372, doi: 10.1201/b12985

    work page doi:10.1201/b12985 2012

  5. [5]

    Dalcin, L., & Fang, Y.-L. L. 2021, Computing in Science & Engineering, 23, 47, doi: 10.1109/MCSE.2021.3083216

    work page doi:10.1109/mcse.2021.3083216 2021

  6. [6]

    E., Bosilca, G., et al

    Gabriel, E., Fagg, G. E., Bosilca, G., et al. 2004, in Proceedings, 11th European PVM/MPI Users’ Group

    work page 2004

  7. [7]

    Harris and K

    Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357, doi: 10.1038/s41586-020-2649-2

    work page doi:10.1038/s41586-020-2649-2 2020

  8. [8]

    Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90, doi: 10.1109/MCSE.2007.55

    work page doi:10.1109/mcse.2007.55 2007

  9. [9]

    2017, pybind11 – Seamless operability between C++11 and Python

    Jakob, W., Rhinelander, J., & Moldovan, D. 2017, pybind11 – Seamless operability between C++11 and Python

    work page 2017

  10. [10]

    2016, ApJ, 833, 202, doi: 10.3847/1538-4357/833/2/202

    Kim, J.-h., Agertz, O., Teyssier, R., et al. 2016, ApJ, 833, 202, doi: 10.3847/1538-4357/833/2/202

    work page doi:10.3847/1538-4357/833/2/202 2016

  11. [11]

    2016, in Positioning and Power in Academic Publishing: Players, Agents and Agendas, ed

    Kluyver, T., Ragan-Kelley, B., Prez, F., et al. 2016, in Positioning and Power in Academic Publishing: Players, Agents and Agendas, ed. F. Loizides & B. Scmidt (IOS Press), 87–90. https://eprints.soton.ac.uk/403913/

    work page 2016

  12. [12]

    279(2):39

    Liao, P.-Y. 2025, ApJS, 279, 39, doi: 10.3847/1538-4365/addc59

    work page doi:10.3847/1538-4365/addc59 2025

  13. [13]

    2022, in In Situ Visualization For Computational Science (Cham, Switzerland: Mathematics and Visualization book series from Springer Publishing), 255 – 279

    Larsen, M., Brugger, E., Childs, H., & Harrison, C. 2022, in In Situ Visualization For Computational Science (Cham, Switzerland: Mathematics and Visualization book series from Springer Publishing), 255 – 279

    work page 2022

  14. [14]

    M., & Swesty, D

    Lattimer, J. M., & Swesty, D. F. 1991, NuPhA, 535, 331, doi: 10.1016/0375-9474(91)90452-C

    work page doi:10.1016/0375-9474(91)90452-c 1991

  15. [15]

    121:151,301

    Levkov, D. G., Panin, A. G., & Tkachev, I. I. 2018, PhRvL, 121, 151301, doi: 10.1103/PhysRevLett.121.151301 Liebend¨ orfer, M. 2005, ApJ, 633, 1042, doi: 10.1086/466517

    work page doi:10.1103/physrevlett.121.151301 2018

  16. [16]

    97(10):103523

    Lin, S.-C., Schive, H.-Y., Wong, S.-K., & Chiueh, T. 2018, PhRvD, 97, 103523, doi: 10.1103/PhysRevD.97.103523

    work page doi:10.1103/physrevd.97.103523 2018

  17. [17]

    Loring, B., Myers, A., Camp, D., & Bethel, E. W. 2018, in Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, ISAV ’18 (New York, NY, USA: Association for Computing Machinery), 19–24, doi: 10.1145/3281464.3281465

    work page doi:10.1145/3281464.3281465 2018

  18. [18]

    2006, A&A, 445, 273, doi: 10.1051/0004-6361:20052840

    Buras, R. 2006, A&A, 445, 273, doi: 10.1051/0004-6361:20052840 34 Tsai et al. Message Passing Interface Forum. 2023, MPI: A Message-Passing Interface Standard Version 4.1. https: //www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf

    work page doi:10.1051/0004-6361:20052840 2006

  19. [19]

    2005, Journal of Computational Physics, 208, 315 , doi: https://doi.org/10.1016/j.jcp.2005.02.017

    Miyoshi, T., & Kusano, K. 2005, Journal of Computational Physics, 208, 315 , doi: https://doi.org/10.1016/j.jcp.2005.02.017

    work page doi:10.1016/j.jcp.2005.02.017 2005

  20. [20]

    2007, in Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’07 (New

    Nethercote, N., & Seward, J. 2007, in Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’07 (New

    work page 2007

  21. [21]

    York, NY, USA: Association for Computing Machinery), 89–100, doi: 10.1145/1250734.1250746

    work page doi:10.1145/1250734.1250746

  22. [22]

    Niethammer, C., & Rabenseifner, R. 2019, in Proceedings of the 26th European MPI Users’ Group Meeting, EuroMPI ’19 (New York, NY, USA: Association for Computing Machinery), doi: 10.1145/3343211.3343217 O’Connor, E., & Ott, C. D. 2010, Classical and Quantum Gravity, 27, 114103, doi: 10.1088/0264-9381/27/11/114103 P´ erez, F., & Granger, B. E. 2007, Computi...

    work page doi:10.1145/3343211.3343217 2019

  23. [23]

    2013, pynbody: N-Body/SPH analysis for python, Astrophysics Source Code Library, record ascl:1305.002

    Pontzen, A., Roˇ skar, R., Stinson, G., & Woods, R. 2013, pynbody: N-Body/SPH analysis for python, Astrophysics Source Code Library, record ascl:1305.002. http://ascl.net/1305.002

    work page 2013

  24. [24]

    2018, The Astrophysical Journal Supplement Series, 237, 23, doi: 10.3847/1538-4365/aac832

    Pontzen, A., & Tremmel, M. 2018, The Astrophysical Journal Supplement Series, 237, 23, doi: 10.3847/1538-4365/aac832

    work page doi:10.3847/1538-4365/aac832 2018

  25. [25]

    2011, Computing in Science & Engineering, 13, 40

    Ramachandran, P., & Varoquaux, G. 2011, Computing in Science & Engineering, 13, 40

    work page 2011

  26. [26]

    481(4):4815--4840

    Schive, H.-Y., ZuHone, J. A., Goldbaum, N. J., et al. 2018, Monthly Notices of the Royal Astronomical Society, 481, 4815, doi: 10.1093/mnras/sty2586

    work page doi:10.1093/mnras/sty2586 2018

  27. [27]

    D., et al., 2016, @doi [Monthly Notices of the Royal Astronomical Society] 10.1093/mnras/stw3291 , 466, 2217–2234

    Smith, B. D., Bryan, G. L., Glover, S. C. O., et al. 2017, MNRAS, 466, 2217, doi: 10.1093/mnras/stw3291

    work page doi:10.1093/mnras/stw3291 2017

  28. [28]

    Sod, G. A. 1978, Journal of Computational Physics, 27, 1, doi: https://doi.org/10.1016/0021-9991(78)90023-2

    work page doi:10.1016/0021-9991(78)90023-2 1978

  29. [29]

    , keywords =

    Springel, V., Di Matteo, T., & Hernquist, L. 2005, MNRAS, 361, 776, doi: 10.1111/j.1365-2966.2005.09238.x

    work page doi:10.1111/j.1365-2966.2005.09238.x 2005

  30. [30]

    W., Hempel, M., & Fischer, T

    Steiner, A. W., Hempel, M., & Fischer, T. 2013, ApJ, 774, 17, doi: 10.1088/0004-637X/774/1/17

    work page doi:10.1088/0004-637x/774/1/17 2013

  31. [31]

    Simon, J. B. 2008, ApJS, 178, 137, doi: 10.1086/588755

    work page doi:10.1086/588755 2008

  32. [32]

    Toro, E. F. 2009, Riemann solvers and numerical methods for fluid dynamics. A practical introduction, 3rd edn. (Berlin: Springer)

    work page 2009

  33. [33]

    2023, in Proceedings of the 22nd Python in Science Conference, SciPy (SciPy), doi: 10.25080/gerudo-f2bc6f59-011

    Tsai, S.-R., Schive, H.-Y., & Turk, M. 2023, in Proceedings of the 22nd Python in Science Conference, SciPy (SciPy), doi: 10.25080/gerudo-f2bc6f59-011

    work page doi:10.25080/gerudo-f2bc6f59-011 2023

  34. [34]

    2024, in Proceedings of the Platform for Advanced Scientific Computing

    Tsai, S.-R., Schive, H.-Y., & Turk, M. 2024, in Proceedings of the Platform for Advanced Scientific Computing

    work page 2024

  35. [35]

    Conference, PASC ’24 (New York, NY, USA: Association for Computing Machinery), doi: 10.1145/3659914.3659939

    work page doi:10.1145/3659914.3659939

  36. [36]

    J., Smith, B

    Turk, M. J., Smith, B. D., Oishi, J. S., et al. 2011, The Astrophysical Journal Supplement Series, 192, 9, doi: 10.1088/0067-0049/192/1/9 Van Rossum, G., & Drake, F. L. 2009, Python 3 Reference Manual (Scotts Valley, CA: CreateSpace)

    work page doi:10.1088/0067-0049/192/1/9 2011

  37. [37]

    1984, Journal of Computational Physics, 54, 115, doi: 10.1016/0021-9991(84)90142-6

    Woodward, P., & Colella, P. 1984, Journal of Computational Physics, 54, 115, doi: 10.1016/0021-9991(84)90142-6

    work page doi:10.1016/0021-9991(84)90142-6 1984

  38. [38]

    2007, Physics Reports, 442, 269, doi: 10.1016/j.physrep.2007.02.009

    Woosley, S. E., & Heger, A. 2007, PhR, 442, 269, doi: 10.1016/j.physrep.2007.02.009

    work page doi:10.1016/j.physrep.2007.02.009 2007