REVIEW 2 major objections 1 minor 42 references
FBench: A Flexible Benchmark for CFG-Based What-If Exploration of HPC I/O Patterns
T0 review · 2 major / 1 minor · reviewed 2026-06-30 · grok-4.3
Pith's one-line read FBench derives context-free grammars from I/O traces to enable what-if exploration of HPC performance settings.
desk verdict FBench adds a CFG replay layer on Recorder traces for HPC I/O what-if tests, but the accuracy claims for modified settings rest on unverified assumptions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Context-free grammars (CFGs) derived from Recorder traces, which capture I/O access patterns and enable parameter changes through JSON configs for what-if analysis.
What would settle it
Running the actual FLASH Sedov application with collective I/O on Lustre and finding the bandwidth difference is much smaller than the 30x reported by FBench.
Extended reading notes
Core claim
FBench leverages context-free grammars derived from Recorder traces to generate simplified global configuration files for benchmark execution or to replay I/O patterns on-the-fly. It supports POSIX and MPI-IO, and allows injection of optimization hints via JSON. Evaluation demonstrates accurate reproduction of access patterns and performance trends, including up to 30x lower write bandwidth for collective I/O on Lustre in FLASH Sedov, and up to 8x improvements with tuning in LAMMPS.
Load-bearing premise
That the context-free grammars preserve sufficient detail from the original traces to predict performance accurately when I/O parameters like collective mode or striping are altered.
Editorial extensions
If this is right
- For IOR and HACC-IO, FBench matches scaling behavior and sensitivity to Lustre striping parameters.
- For FLASH Sedov, collective I/O on Lustre yields up to 30x lower write bandwidth than independent I/O, largely independent of striping.
- Switching to a burst buffer file system increases non-collective write bandwidth by about 1.5x.
- FBench significantly reduces the time required for what-if analyses and enables improvements of up to 8x with simple tuning on LAMMPS.
Reading between the lines
- This method could allow systematic search over many I/O configurations that would be too expensive to test with full application runs.
- Similar CFG-based approaches might apply to other performance domains like network or compute patterns in HPC.
- Integration with machine learning could automate finding optimal JSON hints based on the grammar structure.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FBench, a tool that derives context-free grammars (CFGs) from Recorder traces to enable flexible what-if exploration of HPC I/O patterns. It supports POSIX and MPI-IO, allows JSON-based injection of optimization hints (e.g., collective I/O, striping, burst buffers), and can either generate benchmark configs or replay patterns on-the-fly. Evaluation claims accurate reproduction of access patterns and performance trends for IOR, HACC-IO, FLASH Sedov (revealing up to 30x lower write bandwidth for collective vs. independent I/O on Lustre), and LAMMPS (up to 8x improvement with tuning), while reducing time for what-if analyses.
Significance. If the fidelity of CFG-derived patterns holds under modified parameters, FBench provides a practical, code-transparent method for rapid I/O what-if analysis without repeated full application runs. This addresses a real need in HPC for exploring middleware and FS interactions. The approach of using CFGs for pattern capture and JSON hints is a strength for flexibility across interfaces.
major comments (2)
- [Evaluation (FLASH Sedov results)] The central claim that FBench 'accurately reproduces I/O behavior' and supports reliable what-if predictions (abstract) rests on CFG fidelity when parameters like collective I/O or striping are altered via JSON. However, the evaluation provides no explicit side-by-side comparison of FBench outputs against native application runs (e.g., FLASH Sedov) executed with identical modified settings, leaving open whether omitted context-sensitive dependencies affect the reported 30x bandwidth gap.
- [Abstract and Evaluation] The abstract reports concrete performance numbers (30x lower bandwidth, 1.5x increase, 8x improvement) and 'accurate reproduction' without describing measurement methodology, number of runs, error bars, or how baselines were established. This directly weakens support for the what-if exploration claims.
minor comments (1)
- [Methods] Clarify in the methods how timing behavior and synchronization points from traces are preserved or approximated in the CFG replay for performance predictions.
Simulated Author's Rebuttal
We thank the referee for the constructive comments, which highlight important aspects of our evaluation that require clarification and strengthening. We address each major comment below and commit to revisions that improve the manuscript's rigor without altering its core contributions.
read point-by-point responses
-
Referee: [Evaluation (FLASH Sedov results)] The central claim that FBench 'accurately reproduces I/O behavior' and supports reliable what-if predictions (abstract) rests on CFG fidelity when parameters like collective I/O or striping are altered via JSON. However, the evaluation provides no explicit side-by-side comparison of FBench outputs against native application runs (e.g., FLASH Sedov) executed with identical modified settings, leaving open whether omitted context-sensitive dependencies affect the reported 30x bandwidth gap.
Authors: We agree that validating CFG fidelity specifically under modified parameters (e.g., collective I/O or burst-buffer settings for FLASH Sedov) is essential to support the what-if claims. The current evaluation demonstrates reproduction of original traces and trends for unmodified workloads, with JSON hints then applied to explore variants; however, direct native-application runs under those exact modified configurations were not performed or reported side-by-side. This is a substantive gap. In revision we will add such comparisons for at least the key FLASH Sedov cases where feasible, or qualify the claims accordingly if full validation proves impractical due to application constraints. revision: yes
-
Referee: [Abstract and Evaluation] The abstract reports concrete performance numbers (30x lower bandwidth, 1.5x increase, 8x improvement) and 'accurate reproduction' without describing measurement methodology, number of runs, error bars, or how baselines were established. This directly weakens support for the what-if exploration claims.
Authors: The referee correctly identifies that the abstract and evaluation lack explicit details on experimental methodology. We will revise both sections to specify the number of runs, report means with error bars or standard deviations, describe how baselines (e.g., native IOR/HACC-IO/FLASH/LAMMPS executions) were established, and clarify the measurement environment and statistical practices used to obtain the reported factors (30x, 1.5x, 8x). revision: yes
Circularity Check
No circularity: tool description relies on external traces and evaluations
full rationale
The paper presents FBench as a benchmarking tool that derives CFGs from Recorder traces (external input) to support what-if analysis via JSON hints. No equations, fitted parameters, or self-referential derivations are described. Evaluation claims rest on reproduction of IOR, HACC-IO, FLASH Sedov, and LAMMPS behaviors against real workloads, not on any internal reduction or self-citation chain. The derivation chain is self-contained against external benchmarks with no load-bearing steps that reduce to the paper's own inputs by construction.
Assumptions & free parameters
Cite this review
Pith. "Pith review of FBench: A Flexible Benchmark for CFG-Based What-If Exploration of HPC I/O Patterns." pith.science (2026). https://pith.science/paper/GOUEPDHZ
@misc{pith2026260630197,
author = {Pith},
title = {Pith review of: FBench: A Flexible Benchmark for CFG-Based What-If Exploration of HPC I/O Patterns},
year = {2026},
howpublished = {\url{https://pith.science/paper/GOUEPDHZ}},
note = {Machine review of arXiv:2606.30197}
}
read the original abstract
The I/O performance of large-scale HPC applications depends on a complex interplay of access patterns, middleware optimizations, and file system configurations. To systematically explore these effects without repeatedly rerunning full applications, we introduce FBench, a flexible and code-transparent benchmarking tool for what-if analysis and I/O performance exploration. FBench leverages context-free grammars (CFGs) derived from Recorder traces to either generate simplified global configuration files for benchmark execution or replay I/O patterns on-the-fly without additional preprocessing. It supports both POSIX and MPI-IO interfaces and allows users to inject optimization hints via JSON configuration files, enabling rapid experimentation with I/O settings without code changes. Our evaluation shows that FBench accurately reproduces I/O behavior for both synthetic and real workloads, capturing access patterns and performance trends across diverse optimizations and file system settings. For IOR and HACC-IO, FBench closely matches scaling behavior and sensitivity to Lustre striping parameters. For FLASH Sedov, it reveals that collective I/O on Lustre can yield up to 30x lower write bandwidth than independent I/O, largely independent of striping, and that switching to a burst buffer file system increases non-collective write bandwidth by about 1.5x without additional tuning. The evaluation with LAMMPS shows that FBench can significantly reduce the time required for what-if analyses and, with simple tuning, enable improvements of up to 8x.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Understanding i/o behavior in scientific and data- intensive computing (dagstuhl seminar 21332).Dagstuhl reports, 11(7):16–75, 2021
Philip Carns, Julian Kunkel, Kathryn Mohror, and Martin Schulz. Understanding i/o behavior in scientific and data- intensive computing (dagstuhl seminar 21332).Dagstuhl reports, 11(7):16–75, 2021
2021
-
[2]
I/o performance chal- lenges at leadership scale
Samuel Lang, Philip Carns, Robert Latham, Robert Ross, Kevin Harms, and William Allcock. I/o performance chal- lenges at leadership scale. InProceedings of the Confer- ence on High Performance Computing Networking, Stor- age and Analysis, pages 1–12, 2009
2009
-
[3]
On the role of burst buffers in leadership-class storage systems
Ning Liu, Jason Cope, Philip Carns, Christopher Carothers, Robert Ross, Gary Grider, Adam Crume, and Carlos Maltzahn. On the role of burst buffers in leadership-class storage systems. In2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST), pages 1–11. IEEE, 2012
2012
-
[4]
Burstmem: A high- performance burst buffer system for scientific applications
Teng Wang, Sarp Oral, Yandong Wang, Brad Settlemyer, Scott Atchley, and Weikuan Yu. Burstmem: A high- performance burst buffer system for scientific applications. In2014 IEEE International Conference on Big Data (Big Data), pages 71–79. IEEE, 2014
2014
-
[5]
Automatic generation of i/o kernels for hpc applications
Babak Behzad, Hoang-Vu Dang, Farah Hariri, Weizhe Zhang, and Marc Snir. Automatic generation of i/o kernels for hpc applications. In2014 9th Parallel Data Storage Workshop, pages 31–36. IEEE, 2014
2014
-
[6]
An overview of the hdf5 technol- ogy suite and its applications
Mike Folk, Gerd Heber, Quincey Koziol, Elena Pourmal, and Dana Robinson. An overview of the hdf5 technol- ogy suite and its applications. InProceedings of the EDBT/ICDT 2011 workshop on array databases, 2011
2011
-
[7]
Netcdf: an interface for scien- tific data access.IEEE computer graphics and applications, 10(4), 1990
Russ Rew and Glenn Davis. Netcdf: an interface for scien- tific data access.IEEE computer graphics and applications, 10(4), 1990
1990
-
[8]
Parallel netcdf: A high-performance scientific i/o interface
Jianwei Li, Wei-keng Liao, Alok Choudhary, Robert Ross, Rajeev Thakur, William Gropp, Rob Latham, Andrew Siegel, Brad Gallagher, and Michael Zingale. Parallel netcdf: A high-performance scientific i/o interface. In Proceedings of the 2003 ACM/IEEE conference on Super- computing, page 39, 2003
2003
Show all 42 references
-
[9]
A user-level infiniband-based file system and checkpoint strategy for burst buffers
Kento Sato, Kathryn Mohror, Adam Moody, Todd Gam- blin, Bronis R De Supinski, Naoya Maruyama, and Satoshi Matsuoka. A user-level infiniband-based file system and checkpoint strategy for burst buffers. In2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Comp...
2014
-
[10]
Trio: Burst buffer based i/o orchestration
Teng Wang, Sarp Oral, Michael Pritchard, Bin Wang, and Weikuan Yu. Trio: Burst buffer based i/o orchestration. In 2015 IEEE International Conference on Cluster Comput- ing, pages 194–203. IEEE, 2015
2015
-
[11]
Unifyfs: A user-level shared file system for unified access to distributed local storage
Michael J Brim, Adam T Moody, Seung-Hwan Lim, Ross Miller, Swen Boehm, Cameron Stanavige, Kathryn M Mohror, and Sarp Oral. Unifyfs: A user-level shared file system for unified access to distributed local storage. In 2023 IEEE International Parallel and Distributed Process- ing...
2023
-
[12]
PhD thesis, University of Illinois at Urbana-Champaign, 2022
Chen Wang.Parallel File System with Tunable Consistency. PhD thesis, University of Illinois at Urbana-Champaign, 2022
2022
-
[13]
Plfs: a checkpoint filesystem for parallel appli- cations
John Bent, Garth Gibson, Gary Grider, Ben McClelland, Paul Nowoczynski, James Nunez, Milo Polte, and Meghan Wingate. Plfs: a checkpoint filesystem for parallel appli- cations. InProceedings of the Conference on High Per- formance Computing Networking, Storage and Analysis, pag...
2009
-
[14]
I/o access patterns in hpc applications: A 360-degree survey.ACM Computing Surveys, 56(2):1–41, 2023
Jean Luca Bez, Suren Byna, and Shadi Ibrahim. I/o access patterns in hpc applications: A 360-degree survey.ACM Computing Surveys, 56(2):1–41, 2023
2023
-
[15]
I/o bottleneck detection and tuning: Connecting the dots using interactive log analysis
Jean Luca Bez, Houjun Tang, Bing Xie, David Williams- Young, Rob Latham, Rob Ross, Sarp Oral, and Suren Byna. I/o bottleneck detection and tuning: Connecting the dots using interactive log analysis. In2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW), 2021
2021
-
[16]
Drishti: Guiding end-users in the i/o optimization journey
Jean Luca Bez, Hammad Ather, and Suren Byna. Drishti: Guiding end-users in the i/o optimization journey. In2022 IEEE/ACM International Parallel Data Systems Workshop (PDSW), pages 1–6. IEEE, 2022
2022
-
[17]
Evaluating similarity-based trace reduction techniques for scalable performance analysis
Kathryn Mohror and Karen L Karavanic. Evaluating similarity-based trace reduction techniques for scalable performance analysis. InProceedings of the conference on high performance computing networking, storage and analysis, pages 1–12, 2009. Preprint– FBench: A FlexibleBenchma...
2009
-
[18]
Recorder 2.0: Efficient parallel i/o tracing and analysis
Chen Wang, Jinghan Sun, Marc Snir, Kathryn Mohror, and Elsa Gonsiorowski. Recorder 2.0: Efficient parallel i/o tracing and analysis. In2020 IEEE International Par- allel and Distributed Processing Symposium Workshops (IPDPSW), pages 1–8. IEEE, 2020
2020
-
[19]
Parallel i/o evaluation techniques and emerging hpc workloads: A perspective
Sarah Neuwirth and Arnab K Paul. Parallel i/o evaluation techniques and emerging hpc workloads: A perspective. In2021 IEEE International Conference on Cluster Com- puting (CLUSTER), pages 671–679, 2021
2021
-
[20]
Boosting application-specific par- allel i/o optimization using iosig
Yanlong Yin, Surendra Byna, Huaiming Song, Xian-He Sun, and Rajeev Thakur. Boosting application-specific par- allel i/o optimization using iosig. In2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Com- puting (ccgrid 2012), pages 196–203. IEEE, 2012
2012
-
[21]
Repli- cating hpc i/o workloads with proxy applications
James Dickson, Steven Wright, Satheesh Maheswaran, Andy Herdman, Mark C Miller, and Stephen Jarvis. Repli- cating hpc i/o workloads with proxy applications. In2016 1st Joint International Workshop on Parallel Data Storage and data Intensive Scalable Computing Systems (PDSW- DI...
2016
-
[22]
Skel: generative software for producing skeletal i/o applications
Jeremy Logan, Scott Klasky, Jay Lofstead, Hasan Abbasi, Stephane Ethier, Ray Grout, Seung-Hoe Ku, Qing Liu, Xiaosong Ma, Manish Parashar, et al. Skel: generative software for producing skeletal i/o applications. In2011 IEEE Seventh International Conference on e-Science Work- s...
2011
-
[23]
Introduction to automata theory, languages, and computa- tion.Acm Sigact News, 32(1):60–65, 2001
John E Hopcroft, Rajeev Motwani, and Jeffrey D Ullman. Introduction to automata theory, languages, and computa- tion.Acm Sigact News, 32(1):60–65, 2001
2001
-
[24]
Siesta: Synthesizing proxy applications for mpi programs
Jiyu Luo, Tao Yan, Qingguo Xu, Jingwei Sun, and Guangzhong Sun. Siesta: Synthesizing proxy applications for mpi programs. In2024 IEEE International Conference on Cluster Computing (CLUSTER), pages 14–26. IEEE, 2024
2024
-
[25]
Omnisc’io: A grammar-based approach to spatial and temporal i/o patterns prediction
Matthieu Dorier, Shadi Ibrahim, Gabriel Antoniu, and Rob Ross. Omnisc’io: A grammar-based approach to spatial and temporal i/o patterns prediction. InSC’14: Proceed- ings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2014
2014
-
[26]
Recorder: Comprehensive parallel i/o tracing and analysis.arXiv preprint arXiv:2501.04654, 2025
Chen Wang, Izzet Yildirim, Hariharan Devarajan, Kathryn Mohror, and Marc Snir. Recorder: Comprehensive parallel i/o tracing and analysis.arXiv preprint arXiv:2501.04654, 2025
2025
-
[27]
Pilgrim: scalable and (near) lossless mpi tracing
Chen Wang, Pavan Balaji, and Marc Snir. Pilgrim: scalable and (near) lossless mpi tracing. InProceedings of the In- ternational Conference for High Performance Computing, Networking, Storage and Analysis, 2021
2021
-
[28]
FLASH I/O Benchmark Rou- tine
Michael Zingale. FLASH I/O Benchmark Rou- tine. https://www.ucolick.org/~zingale/flash_ benchmark_io/. Accessed: 2025-11-26
2025
-
[29]
Lockwood
Glenn K. Lockwood. HACC-IO Repository. https: //github.com/glennklockwood/hacc-io, 2012. Ac- cessed: 2025-11-17
2012
-
[30]
Ior and mdtest parallel i/o benchmark repository
hpc/ior. Ior and mdtest parallel i/o benchmark repository. https://github.com/hpc/ior, 2024. Accessed: 2025- 11-17
2024
-
[31]
Darshan — scalable hpc i/o characterization framework
Darshan Development Team. Darshan — scalable hpc i/o characterization framework. https://github.com/ darshan-hpc/darshan. Accessed: 2025-11-30
2025
-
[32]
Design & implementation of macsio
Mark C Miller. Design & implementation of macsio. Tech- nical report, Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States), 2015
2015
-
[33]
Oak Ridge Leadership Computing Facility. Adios. https: //www.olcf.ornl.gov/center-projects/adios/,
-
[34]
Accessed: 2024-12-04
2024
-
[35]
Techniques for modeling large-scale hpc i/o workloads
Shane Snyder, Philip Carns, Robert Latham, Misbah Mubarak, Robert Ross, Christopher Carothers, Babak Be- hzad, Huong Vu Thanh Luu, Surendra Byna, and Prabhat. Techniques for modeling large-scale hpc i/o workloads. In Proceedings of the 6th International Workshop on Perfor- man...
2015
-
[36]
Scalaioextrap: Elastic i/o tracing and extrapolation
Xiaoqing Luo, Frank Mueller, Philip Carns, Jonathan Jenkins, Robert Latham, Robert Ross, and Shane Sny- der. Scalaioextrap: Elastic i/o tracing and extrapolation. In 2017 IEEE International Parallel and Distributed Process- ing Symposium (IPDPS), pages 585–594. IEEE, 2017
2017
-
[37]
Hpc i/o trace extrapolation
Xiaoqing Luo, Frank Mueller, Philip Carns, John Jenkins, Robert Latham, Robert Ross, and Shane Snyder. Hpc i/o trace extrapolation. InProceedings of the 4th Workshop on Extreme Scale Programming Tools, pages 1–6, 2015
2015
-
[38]
Corona Compute Platform
LLNL. Corona Compute Platform. https://hpc.llnl. gov/hardware/compute-platforms/corona, 2025. Accessed: 2025-07-03
2025
-
[39]
Flash-x: A multiphysics scientific software system
flash x.org. Flash-x: A multiphysics scientific software system. https://flash-x.org, 2024. Accessed: 2025- 11-17
2024
-
[40]
Austin Harris, Tom Klosterman, Rajeev Jain, Johann Rudi, Bronson Messer, Michael Pajkos, Jared Carlson, Ran Chu, Mohamed Wahib, Saurabh Chawd- hary, Paul M
Anshu Dubey, Klaus Weide, Jared O’Neal, Akash Dhruv, Sean Couch, J. Austin Harris, Tom Klosterman, Rajeev Jain, Johann Rudi, Bronson Messer, Michael Pajkos, Jared Carlson, Ran Chu, Mohamed Wahib, Saurabh Chawd- hary, Paul M. Ricker, Dongwook Lee, Katie Antypas, Katherine M. Ri...
2022 doi
-
[41]
CRC press, 2018
Leonid Ivanovich Sedov.Similarity and dimensional meth- ods in mechanics. CRC press, 2018
2018
-
[42]
Lammps: large-scale atomic/molecular massively parallel simulator.Zenodo, 2007
Steven J Plimpton, Axel Kohlmeyer, Aidan P Thompson, Stan G Moore, and Richard Berger. Lammps: large-scale atomic/molecular massively parallel simulator.Zenodo, 2007
2007
Reviewed June 30, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.