REVIEW 4 major objections 5 minor 40 references
Data Readiness for Scientific AI at Scale
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Scientific AI data readiness is best seen as a two-dimensional maturity matrix crossing five readiness levels with five processing stages.
desk verdict A useful position paper on data readiness for scientific AI at HPC scale, but the central matrix is internally inconsistent with its own pipeline definitions and needs revision before it can serve as a benchmark. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the two-dimensional conceptual maturity matrix built from Data Readiness Levels and Data Processing Stages, together with the abstracted pipeline ingest to preprocess to transform to structure to shard that organizes it. The five readiness levels (raw, cleaned, labeled, feature-engineered, and fully AI-ready) define how prepared a dataset is, while the five stages define what transformations a domain applies; the matrix's cells map which level-specific operations occur at each stage, for example sharding into binary formats appears only at level five. This grid does the argument's work because it turns scattered, bespoke domain pipelines into a single vocabulary: a dataset's position in the matrix states both its operational maturity and which domain-specific conversions remain.
What would settle it
A concrete disconfirmation would be finding a substantial scientific AI workflow that cannot be placed on the ingest to preprocess to transform to structure to shard pipeline without stretching, for example a production system that trains directly on raw streaming sensor data with no sharded storage, and showing that this workflow is common rather than rare across several domains.
Extended reading notes
Core claim
The central claim is that scientific data readiness is a two-dimensional, context-dependent process rather than a binary threshold. The first dimension, Data Readiness Levels, ranks a dataset's technical preparation from raw input to fully AI-trainable format; the second, Data Processing Stages, describes the structural workflow (ingest, preprocess, transform, structure, shard) that a domain must apply. By abstracting shared patterns from climate, fusion, bio/health, and materials workflows, the paper shows that the outer stages are common across domains while the transform and structure stages carry domain-specific work such as regridding, anonymization, or graph encoding. The resulting conceptual maturity matrix is offered as a tool for evaluating, comparing, and designing AI-ready data pipelines at leadership-scale computing facilities, with particular attention to transformer-based generative models.
Load-bearing premise
The framework's cross-domain claims rest on the assumption that the four surveyed domains (climate, fusion, bio/health, and materials) are representative of scientific AI at leadership scale, and that their workflows reveal the general pipeline all scientific data follows.
Editorial extensions
If this is right
- Datasets across disciplines can be assigned a readiness level and stage, making it possible to compare climate, fusion, bio/health, and materials pipelines on a common scale.
- Facilities can identify the shared outer stages (ingest, preprocess, shard) and invest in reusable high-throughput tooling, while leaving transform and structure to domain specialists.
- A dataset is fully AI-ready only when all five stages meet level-five criteria, so sharded, normalized, labeled binary formats become the operational definition of readiness for transformer-based training.
- The framework implies that preprocessing pipelines should be designed with parallel I/O and sharded storage in mind from the start, rather than as an afterthought, to support petascale and exascale training.
- Because the transform and structure stages are recognized as domain-specific, the matrix can guide future work on standardized domain templates without forcing one universal pipeline.
Reading between the lines
- One extension the authors leave implicit is that the matrix could be turned into a scoring instrument: a checklist or automated audit that inspects a dataset's metadata and storage layout to assign readiness levels without expert interviews.
- The paper's own scope warning invites a testable extension: applying the same abstraction to additional fields such as cosmology or particle physics would either confirm the five-stage pipeline as a true universal or reveal a missing stage, such as continuous online streaming.
- Because the frame is oriented to transformer-based generative models, an inference is that readiness criteria may need revision as other architectures (state-space models, non-transformer GNNs at scale) change what 'AI-ready' storage must support.
- Another tacit implication is that the boundary between transform and structure could be used as a modular interface in pipeline libraries, letting domain-specific converters plug into generic ingest-and-shard infrastructure.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-dimensional data-readiness framework for training scientific foundation models at leadership-scale HPC facilities. It reviews preprocessing patterns in four domains—climate, fusion, bio/health, and materials—distills them into a five-stage pipeline (ingest → preprocess → transform → structure → shard), defines five Data Readiness Levels (raw, cleaned, labeled, feature-engineered, fully AI-ready), and combines these into a conceptual maturity matrix. The authors also identify cross-cutting challenges such as data scarcity, preprocessing scalability, provenance, fragmentation, privacy, and bias, and they close with future directions. The paper explicitly frames itself as an initial conceptual step that requires further iteration and validation.
Significance. If made internally consistent, the framework addresses a genuine gap: existing work such as Hiniduma et al. is broad and conceptual, while domain-specific maturity models are fragmented, and the paper offers a unified, HPC-oriented vocabulary for comparing dataset readiness. Strengths of the paper include its grounding in four concrete workflows (ClimaX/ORBIT, DIII-D/IMAS, TwoFold/AlphaFold, OMat24/HydraGNN), its honest acknowledgment of limitations in Sections 1 and 6, and the absence of any fitted-parameter circularity, since the paper makes no quantitative predictions. The main technical weakness is that the central artifact, Table 2, conflicts with the stage definitions in Section 3.5, so the framework is not yet usable as the benchmarking tool it claims to be.
major comments (4)
- [Section 3.5 and Table 2] Table 2 assigns "Initial spatial/temporal alignment or regridding" to the Preprocess column, "Normalization or anonymization ... basic labels added" to the Transform column, and "Domain-specific feature extraction" to the Structure column. However, Section 3.5 explicitly defines transform as capturing domain-specific conversions, with regridding in climate and anonymization in health as examples, and defines structure as organizing data into standardized formats. These assignments are contradictory: regridding appears in both Preprocess (Level 2) and Transform (by the Section 3.5 definition), and feature extraction is placed in Structure rather than Transform. Because Table 2 is the paper's central contribution and is invoked in the abstract and conclusions as a practical benchmarking tool, this stage-boundary ambiguity must be resolved before the central claim can be accepted.
- [Section 3] The paper states that the analysis "draws on interviews with domain scientists at ORNL," but it never reports the number of interviews, the selection criteria for participants, the question protocol, or how the interview content was analyzed and coded into the five levels and stages. Since the cross-domain generalization is the paper's main contribution, the evidentiary basis for the four "archetypes" is not verifiable. Please add a methods paragraph, or, if the interviews were informal, say so explicitly and describe their role as anecdotal rather than systematic evidence.
- [Section 4 and Table 2] Section 4 calls the matrix "a pragmatic tool for evaluating technical readiness," and Section 6 says it provides "a practical foundation for benchmarking datasets," but the paper gives no operational criteria for assigning a dataset to a particular level or stage. Descriptions such as "optimized high-throughput ingestion" (Level 4) and "ingestion pipelines fully automated and performance-optimized" (Level 5) overlap, and Table 2 leaves several cells undefined. Without worked examples that map the Table 1 datasets onto the matrix, or a scoring rubric, a facility cannot currently apply the matrix in a reproducible way. This is fixable, but the absence of operational criteria is load-bearing for the paper's stated practical utility.
- [Section 3.5] The abstracted pipeline ingest → preprocess → transform → structure → shard does not align cleanly with the representative workflows summarized in Sections 3.1–3.4: climate is described as download → regrid → normalize → shard, fusion as extract → align → normalize → shard, and materials as parse → normalize → encode → shard. If these four workflows are the empirical basis for the framework, the mapping from each archetype to the five stages needs to be shown explicitly, for example in a table. Otherwise the stage axis appears imposed rather than abstracted from the domain evidence, which weakens the paper's central generalization claim.
minor comments (5)
- [Section 3.1] Pangu-Weather is mentioned without a citation; please add the original reference so readers can locate the described preprocessing pipeline.
- [References] Reference [4] contains a typo in the arXiv identifier: "24010.12771" should be "2410.12771."
- [Section 2.1] The prose in Section 2.1 lists cleaning, normalization, labeling, feature engineering, and format conversion, but it does not map these terms to the five Data Processing Stages introduced in Section 3.5; a concise mapping would prevent terminological drift.
- [Figure 1] Figure 1 is referenced but never described in detail; please add a caption that defines each arrow and the meaning of the terminal "AI-ready" state.
- [Section 3.4] The claim that OMat24 contains "over 100 million" DFT-derived structures should be verified against the source, since the cited paper's exact count may differ and the statement is presented without qualification.
Circularity Check
No circularity: the paper is a conceptual taxonomy with illustrative self-citations, but no derivation chain that reduces to its own inputs.
full rationale
This paper does not present a mathematical derivation, a fitted model, or a quantitative prediction. It proposes a two-dimensional readiness framework—Data Readiness Levels and Data Processing Stages—as an explicitly conceptual maturity matrix grounded in surveyed domain workflows. There are no equations, fitted parameters, or predicted quantities whose values could be forced by construction. The self-citations to ORNL projects such as ORBIT, TwoFold, C-HER, HydraGNN, SCWorks, and ProvEn are used as illustrative examples of domain workflows, not as load-bearing evidence for the framework's validity. The cross-domain patterns are also independently supported by external references such as ClimaX, Pangu-Weather, AlphaFold, Enformer, OMat24, AFLOW, DIII-D, and IMAS. The framework is an abstraction over these workflows rather than a result derived from them by definition. The inconsistency noted between Table 2 and the stage definitions in Section 3.5 is an internal consistency concern, not a circularity concern, because the framework makes no predictive or empirical claim that could reduce to its inputs. Accordingly, no circular step can be exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption The four selected domains, climate, fusion, bio/health, and materials, are representative of leadership-scale scientific AI data readiness.
- ad hoc to paper Data readiness can be decomposed into five discrete Data Readiness Levels and five Data Processing Stages.
- domain assumption Expert interviews with ORNL domain scientists accurately reflect community-wide workflow patterns.
Cite this review
Pith. "Pith review of Data Readiness for Scientific AI at Scale." pith.science (2026). https://pith.science/paper/RWBAPBSV
@misc{pith2026250723018,
author = {Pith},
title = {Pith review of: Data Readiness for Scientific AI at Scale},
year = {2026},
howpublished = {\url{https://pith.science/paper/RWBAPBSV}},
note = {Machine review of arXiv:2507.23018}
}
read the original abstract
This paper examines how Data Readiness for AI (DRAI) principles apply to leadership-scale scientific datasets used to train foundation models. We analyze archetypal workflows across four representative domains - climate, nuclear fusion, bio/health, and materials - to identify common preprocessing patterns and domain-specific constraints. We introduce a two-dimensional readiness framework composed of Data Readiness Levels (raw to AI-ready) and Data Processing Stages (ingest to shard), both tailored to high performance computing (HPC) environments. This framework outlines key challenges in transforming scientific data for scalable AI training, emphasizing transformer-based generative models. Together, these dimensions form a conceptual maturity matrix that characterizes scientific data readiness and guides infrastructure development toward standardized, cross-domain support for scalable and reproducible AI for science.
Figures
Reference graph
Works this paper leans on
-
[1]
Report of the Workshop on Advanced Fusion with Machine Learning, April 30-May 2, 2019
2019. Report of the Workshop on Advanced Fusion with Machine Learning, April 30-May 2, 2019. Technical Report. USDOE Office of Science (SC)(United States)
work page 2019
-
[2]
Subil Abraham, Paul Abston, Ryan Adamson, Valentine Anantharaj, Ashley Barker, Aaron Barlow, Tom Beck, Katie Bethea, Josh Cunningham, Rafael Ferreira Da Silva, et al. 2024. 2023 Operational Assessment Oak Ridge Leadership Computing Facility . Technical Report. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
work page 2024
-
[3]
Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwińska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. 2021. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods 18, 10 (2021), 1196–1203. https://doi.org/10.1038/s41592-021-01252-x
-
[4]
Wood, Misko Dzamba, Meng Gao, Ammar Rizvi, C
Luis Barroso-Luque, Muhammed Shuaibi, Xiang Fu, Brandon M. Wood, Misko Dzamba, Meng Gao, Ammar Rizvi, C. Lawrence Zitnick, and Zachary W. Ulissi. 2024. Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models. arXiv:24010.12771 (2024). arXiv:2410.12771 [cond-mat.mtrl-sci]
-
[5]
John J. Bates and Jeffrey L. Privette. 2012. A maturity model for assessing the completeness of climate data records. Eos, Transactions American Geophysical Union 93, 44 (2012), 441–441. https://doi.org/10.1029/2012EO440006 arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2012EO440006
-
[6]
Cristian Bodnar, Wessel P Bruinsma, Ana Lucic, Megan Stanley, Anna Allen, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan A Weyn, Haiyu Dong, et al. 2025. A foundation model for the Earth system. Nature (2025), 1–8
work page 2025
-
[7]
Wesley Brewer, Aditya Kashi, Sajal Dash, Aristeidis Tsaris, Junqi Yin, Mallikarjun Shankar, and Feiyi Wang. 2024. Scalable Artificial Intelligence for Science: Perspectives, Methods and Exemplars. arXiv preprint arXiv:2406.17812 (2024)
arXiv 2024
-
[8]
Mark Cianciosa, Richard Archibald, Wael Elwasif, Ana Gainaru, Jin Myung Park, and Ross Whitfield. 2022. Adaptive Generation of Training Data for ML Reduced Model Creation. In 2022 IEEE International Conference on Big Data (Big Data) . IEEE, 3408–3416
work page 2022
Show all 40 references
-
[9]
Cami Collins, Jin Myung Park, Rhea Barnett, Katarzyna Borowiec, Ehab Hassan, Paul Humrickhouse, Jeremy Lore, Kyungjin Kim, Vittorio Badalassi, and Phil Snyder. 2023. Integrated plasma and engineering design and assessment for Tokamak reactor components . Technical Report. Oak ...
2023
-
[10]
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. 2020. Lagrangian neural networks. arXiv:2003.04630 (2020)
2020 arXiv
-
[11]
Stefano Curtarolo, Wahyu Setyawan, Gus LW Hart, Michal Jahnatek, Roman V Chepulskii, Richard H Taylor, Shidong Wang, Junkai Xue, Kesong Yang, Ohad Levy, et al. 2012. AFLOW: An automatic framework for high-throughput materials discovery. Computational Materials Science 58 (2012...
2012
-
[12]
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. 2022. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature 602, 7897 (2022), 414–419
2022
-
[13]
DIII-D Team. 2024. DIII-D research to provide solutions for ITER and fusion energy.Nuclear Fusion 64, 11 (2024), 112003. https://doi.org/10.1088/1741- 4326/ad2fe9
2024 doi
-
[14]
Christo El Morr, Manar Jammal, Hossam Ali-Hassan, and Walid El-Hallak. 2022. Data Preprocessing. Springer International Publishing, Cham, 117–163. https://doi.org/10.1007/978-3-031-16990-8_4
2022 doi
-
[15]
National Artificial Intelligence Research Resource Task Force. 2023. Strengthening and Democratizing the U.S. Artificial Intelligence Innovation Ecosystem: An Implementation Plan for a National Artificial Intelligence Research Resource . Technical Report. National Artificial I...
2023
-
[16]
T Fredian, J Stillerman, G Manduchi, A Rigoni, K Erickson, and T Schröder. 2018. MDSplus yesterday, today and tomorrow. Fusion Engineering and Design 127 (2018), 106–110
2018
-
[17]
Yashika Ghai, Donald Spong, Jacobo Varela, Luis Garcia, Juan Ortiz, and Wisdom Dayok. 2024. Surrogate model of energetic particle transport in reactor-relevant fusion devices. Bulletin of the American Physical Society (2024)
2024
-
[18]
Kaveen Hiniduma, Suren Byna, and Jean Luca Bez. 2025. Data readiness for AI: A 360-degree survey. Comput. Surveys 57, 9 (2025), 1–39
2025
-
[19]
Darren J Hsu, Hao Lu, Aditya Kashi, Michael Matheson, John Gounley, Feiyi Wang, Wayne Joubert, and Jens Glaser. 2023. TwoFold: Highly accurate structure and affinity prediction for protein-ligand complexes from sequences. The International Journal of High Performance Computing...
2023
-
[20]
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al . 2021. Highly accurate protein structure prediction with AlphaFold. Nature 596, 7873 (2021), 583–589. https...
2021 doi
-
[21]
Patrick Kage, Jay C Rothenberger, Pavlos Andreadis, and Dimitrios I Diochnos. 2024. A review of pseudo-labeling for computer vision.arXiv:2408.07221 (2024). 10 Brewer et al
2024
-
[22]
Aditya Kashi, Hao Lu, Wesley Brewer, David Rogers, Michael Matheson, Mallikarjun Shankar, and Feiyi Wang. 2024. Mixed-precision numerics in scientific applications: survey and perspectives. arXiv preprint arXiv:2412.19322 (2024)
2024 arXiv
-
[23]
Hilda Klasky, Josh Grant, Midgie MacFarland, Heidi Hanson, Jodie Trafton, and Anuj Kapadia. 2024. V A EDH Advanced Software Pipeline Framework Report: Enhancing Automation and Scalability . Technical Report ORNL/TM-2024/3669. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN...
2024
-
[24]
Kurihana, Takuya and Skluzacek, Tyler and da Silva, Rafael Ferreira da Silva and Anantharaj, Valentine. 2024. Scalable Multi-Facility Workflows for Artificial Intelligence Applications in Climate Research. https://sc24.supercomputing.org/proceedings/workshops/workshop_pages/ws...
2024
-
[25]
Jay F Lofstead, Scott Klasky, Karsten Schwan, Norbert Podhorszki, and Chen Jin. 2008. Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). In Proceedings of the 6th international workshop on Challenges of large applications in distributed e...
2008
-
[26]
Chuizheng Meng, Sam Griesemer, Defu Cao, Sungyong Seo, and Yan Liu. 2025. When physics meets machine learning: A survey of physics-informed machine learning. Machine Learning for Computational Science and Engineering 1, 1 (2025), 1–23
2025
-
[27]
National Artificial Intelligence Research Resource Task Force. 2023. Strengthening and Democratizing the U.S. Artificial Intelligence Innovation Ecosystem: An Implementation Plan for a National Artificial Intelligence Research Resource. U.S. Office of Science and Technology Po...
2023
-
[28]
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. 2023. ClimaX: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343 (2023)
2023 arXiv
-
[29]
Massimiliano Lupo Pasini, Jong Youl Choi, Kshitij Mehta, Pei Zhang, David Rogers, Jonghyun Bae, Khaled Z Ibrahim, Ashwin M Aji, Karl W Schulz, Jorda Polo, et al. 2024. Scalable Training of Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN. ar...
2024 arXiv
-
[30]
Edward O Pyzer-Knapp, Matteo Manica, Peter Staar, Lucas Morin, Patrick Ruch, Teodoro Laino, John R Smith, and Alessandro Curioni. 2025. Foundation models for materials discovery–current state and future directions. npj Computational Materials 11, 1 (2025), 61
2025
-
[31]
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. 2019. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics 378 (2019), 686–707
2019
-
[32]
M Romanelli, R Coelho, D Coster, J Ferreira, L Fleury, S Henderson, J Hollocombe, F Imbeaux, Tomas Jonsson, L Kogan, et al. 2020. Code integration, data verification, and models validation using the ITER integrated modeling and analysis system (IMAS) in EUROfusion. Fusion Scie...
2020
-
[33]
Daniel Schwabe, Katinka Becker, Martin Seyferth, Andreas Klaß, and Tobias Schaeffter. 2024. The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review. npj Digital Medicine 7, 1 (03 Aug 2024), 203. https://doi.org/10.1038/s41746-024-01196-4
2024 doi
-
[34]
Skluzacek, Sean R
Renan Souza, Tyler J. Skluzacek, Sean R. Wilkinson, Maxim Ziatdinov, and Rafael Ferreira da Silva. 2023. Towards Lightweight Data Integration Using Multi-Workflow Provenance and Data Observability. In 2023 IEEE 19th International Conference on e-Science (e-Science) . 1–10. htt...
2023
-
[35]
TensorFlow Team. 2024. TFRecord and tf.train.Example. https://www.tensorflow.org/tutorials/load_data/tfrecord. Accessed: 2024-06-09
2024
-
[36]
Department of Energy
U.S. Department of Energy. 2022. Artificial Intelligence for Earth System Predictability: Workshop Report Executive Summary . Technical Report. U.S. Department of Energy, Office of Biological and Environmental Research (BER). https://www.ai4esp.org/files/EVS_AI4ESP_Executive_S...
2022
-
[37]
Government Accountability Office
U.S. Government Accountability Office. 2022.Artificial Intelligence in Health Care: Benefits and Challenges of Machine Learning Technologies for Medical Diagnostics. Technical Report GAO-22-104629. U.S. Government Accountability Office (GAO). https://www.gao.gov/assets/gao-22-...
2022
-
[38]
Xiao Wang, Aristeidis Tsaris, Siyan Liu, Jong-Youl Choi, Ming Fan, Wei Zhang, Junqi Yin, Moetasim Ashfaq, Dan Lu, and Prasanna Balaprakash
-
[39]
Huimin Zhao, Nathan Hillson, Kerstin Kleese van Dam, and Deepti Tanjore. 2022. Artificial Intelligence and Machine Learning for Bioenergy Research: Opportunities and Challenges. (2022)
2022
-
[2024]
arXiv preprint arXiv:2404.14712 (2024)
ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability. arXiv preprint arXiv:2404.14712 (2024)
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.