REVIEW 3 major objections 5 minor 3 cited by
SubMIT: A Physics Analysis Facility at MIT
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that a compact cluster of servers with 100 Gbps networking and fast local storage lets physicists analyse terabyte-scale datasets interactively, with turnaround times of minutes, and argues this is a working prototype for…
desk verdict A credible, useful description of a working analysis facility prototype; the headline 'minutes on terabyte-scale' claim outruns the evidence, but the facility itself is real and the paper deserves refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pairing of high-bandwidth local I/O with columnar selective reads. Physically, each login and storage server has at least 100 Gbps Ethernet to a managed NVMe scratch filesystem and a large CephFS spinning-disk store; logically, the analysis stack (RDataFrame over NanoAOD, with Dask Gateway for distributed execution and Slurm for resource management) reads only the columns and events the analysis needs, so the fast network and storage are not wasted on unwanted bytes. A local XCache service fronting remote experiment storage accelerates repeat reads. Together this abstracts the infrastructure so the same code runs interactively on a login node, on a local Slurm batch partition, or on external HTCondor pools.
What would settle it
Rerun the Analysis Grand Challenge on the same hardware while reading 50% and 100% of the stored bytes, and plot end-to-end turnaround versus bytes read; if turnaround scales roughly with bytes read instead of staying in minutes, the paper's general claim is falsified.
Extended reading notes
Core claim
The central claim is that SubMIT, as built, delivers interactive analysis at terabyte scale. In the Analysis Grand Challenge benchmark, a 1.7 TB input dataset is analysed with an end-to-end turnaround of minutes, with the analysis reading only about 5% of the stored bytes thanks to columnar NanoAOD data and RDataFrame; distributed execution through a Dask Gateway and Slurm integration adds no measurable scheduling overhead over a single-node multithreaded run. The same setup processed a 10 TB FCC simulation sample in about 30 minutes by saturating the 100 Gbps network read from Ceph. From this the authors conclude that the facility is an ideal prototype for the HL-LHC analysis facility model.
Load-bearing premise
The minute-scale turnaround claim rests on a benchmark in which the analysis reads only about 5% of the 1.7 TB dataset; if realistic High-Luminosity LHC analyses read a larger fraction of their datasets or have different I/O patterns, the observed performance may not generalise.
Editorial extensions
If this is right
- Analysis workflows that today spend days in grid queues can become interactive exploration, shrinking the time from a physics question to an answer.
- I/O-bound medium-scale tasks that are inefficient on remote grid sites can run on local batch nodes with fast access to NVMe and Ceph storage.
- The recipe—high-bandwidth networking per server, NVMe scratch, CephFS, CVMFS, containers, and Slurm-backed Dask—can be reproduced at other institutions building HL-LHC-era analysis facilities.
- A single facility can serve both research and teaching, with CVMFS-distributed software and Slurm reservations handling hundreds of concurrent classroom users.
- Compact columnar data formats like NanoAOD reduce the need for analysts to maintain intermediate datasets, cutting storage needs and manual bookkeeping.
Reading between the lines
- An extension the paper does not run: rerun the same benchmark reading 50% and 100% of the stored bytes; the observed scaling would separate the benefit of fast storage from the benefit of columnar projection.
- The recipe generalises beyond particle physics: any workload that selects columns from large tabular files could use the same login-pool-plus-NVMe-plus-batch layout, so SubMIT is also a test case for general interactive data science.
- The paper measures throughput on its own platform but not side by side with other analysis-facility prototypes; running the Analysis Grand Challenge across facilities would quantify how much of the gain comes from this particular design.
- The FCC example, reading 10 TB from Ceph in about 30 minutes, implies the network saturates; scaling to the exabyte datasets of the HL-LHC era will require per-server bandwidth to grow with data size, a scaling law that could be tested by measuring throughput as dataset size increases.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes SubMIT, a physics analysis facility prototype at MIT, detailing its hardware (login pool, NVMe and Ceph storage, CPU/GPU resources), software environment (containers, CVMFS, JupyterHub), batch integration (Slurm, HTCondor, OSG), and user-support tools (chatbot, monitoring). It presents operational evidence of the facility's usefulness: an Analysis Grand Challenge benchmark, four published CMS analyses that used SubMIT, FCC feasibility studies, and educational use. The central quantitative claim is that SubMIT enables interactive analyses with turnaround times on the order of minutes even for terabyte-scale datasets, supported primarily by the AGC benchmark and the FCC Higgs-mass example.
Significance. If the central claim is properly supported, the paper is a valuable contribution to the HL-LHC analysis-facility discussion: it provides a concrete, operating prototype with real user adoption, published physics results, and an honest description of system architecture and operational lessons. The paper's strongest assets are its grounding in actual analyses (Section 6.2, Table 1) and its transparent discussion of limitations, such as the selective-read AGC workload and the CephFS read-ahead behavior. The quantitative benchmark evidence, however, is currently too thin to support the strong turnaround-time claim in the Abstract and Conclusion, so the paper's significance is contingent on either adding experimental detail or softening the claim.
major comments (3)
- [Abstract and Section 7] The claim that SubMIT achieves turnaround times 'on the order of minutes even when working with terabyte-scale datasets' is not supported by the evidence presented. The only direct measurement supporting this claim is the Analysis Grand Challenge benchmark in Section 6.1.1, which reads only ~5% of the 1.7 TB dataset (roughly 85 GB), runs on a dedicated 192-core/1.5 TB node with local NVMe storage, and reports times of approximately 100-160 seconds. In contrast, the FCC example in Section 6.3 reports 10 TB read from Ceph in about 30 minutes, which is 'tens of minutes' for 10 TB, not 'order of minutes' for terabyte-scale datasets. The Hrare workflow in Figure 7 explicitly includes a batch-processing step of O(days) before an interactive O(min) step. I recommend either softening the claim to refer specifically to post-preparation iteration on selected data or to I/O-bound reads, or providing additional measurements that cover fuller dataset scans with realistic I/O patterns.
- [Section 6.1.1 and Figure 6] The benchmark methodology is insufficient to establish the stability or representativeness of the reported times. Figure 6 shows no uncertainties, no indication of the number of repeated runs, and no description of the variance across runs. No benchmark scripts or raw data are provided. Given that the AGC benchmark is the basis for the paper's headline quantitative claim, the authors should report run-to-run variability, specify the number of trials, and either release the benchmark scripts or describe the measurement procedure in enough detail for reproduction. Without this, the reader cannot assess whether the measured times are stable or just single favorable runs.
- [Section 6.1.1] The statement that reading only about 5% of the dataset 'is typical for such studies' is an unsupported assumption that is load-bearing for the generalization from the AGC benchmark to terabyte-scale interactive analyses. The paper provides no citation or systematic evidence for this typicality. The claim matters because the turnaround-time argument depends on the read fraction: if realistic HL-LHC analyses read a larger fraction of stored data (for example, in full-scan measurements like the W-mass analysis described in Section 6.2.2), the observed performance would be optimistic. I ask the authors to either cite relevant studies documenting typical I/O read fractions for NanoAOD-based analyses or to explicitly restrict the 'minutes' claim to selective-read workloads.
minor comments (5)
- [Section 4.3] The sentence 'All SubMIT disks are accessible via a network of 100 Gbps Ethernet to guarantee fast I/O' could be clarified: the network connects the servers at 100 Gbps, but end-user access via SSH or Jupyter may not saturate this link. Suggest rephrasing to avoid implying a per-user guaranteed bandwidth.
- [Figure 2] The left panel shows a sharp drop in users around January 2025, which the text attributes to a user-removal process, but the y-axis label and legend do not distinguish 'registered' from 'active' users. Adding a clear definition in the caption would improve readability.
- [Section 6.2.2] The phrase 'A fast turnover in the event loop' appears to mean 'a quick turnaround' in the analysis-development loop. The wording 'turnover' is unusual in this context; consider using 'turnaround' consistently.
- [Section 6.3] The sentence 'Typically the reading speeds of Ceph processing saturates the available network bandwidth of 100 Gbps' contains a grammatical agreement error ('speeds ... saturates'). It would also benefit from stating whether the saturation is measured on the node's network interface or on the storage side.
- [References] Reference [49] is listed as 'to appear (2025)'; if this is a private note or preprint, please provide a DOI or arXiv identifier, or cite the public version of CombineTF if available, so readers can verify the claim about automatic differentiation acceleration.
Circularity Check
No circularity found: SubMIT is validated by external AGC benchmarks and published analyses; the minutes-for-terabytes claim is an empirical generalization with a selective-read caveat, not a construction.
full rationale
The paper contains no derivation chain whose output is assumed in its inputs. The central performance claim is supported by the externally defined Analysis Grand Challenge benchmark (1.7 TB CMS open data, Section 6.1) and by published CMS/FCC physics analyses (Table 1, Sections 6.2-6.3), with the AGC RDataFrame implementation cited to an external paper [39]. The CombineTF reference [49] is a self-citation by two of the authors, but it is peripheral: it supports a statistical-fitting feature used in one analysis and is not the load-bearing evidence for the facility's turnaround-time or scale-out claims, which stand on independent benchmarks. The fact that the AGC task reads only about 5% of the 1.7 TB dataset weakens the generalization in the Conclusion to 'terabyte-scale datasets,' but that is an evidentiary limitation, not circularity; no fitted parameter is relabeled as a prediction and no uniqueness theorem is imported from the authors' prior work.
Assumptions & free parameters
assumptions (3)
- domain assumption Physics events are statistically independent, so scientific workflows can be parallelized over disjoint data chunks.
- domain assumption The current WLCG computing model will not meet HL-LHC data processing requirements.
- ad hoc to paper Reading only about 5% of the dataset is typical for analysis tasks.
Cite this review
Pith. "Pith review of SubMIT: A Physics Analysis Facility at MIT." pith.science (2026). https://pith.science/paper/QPPJ6UFZ
@misc{pith2026250601958,
author = {Pith},
title = {Pith review of: SubMIT: A Physics Analysis Facility at MIT},
year = {2026},
howpublished = {\url{https://pith.science/paper/QPPJ6UFZ}},
note = {Machine review of arXiv:2506.01958}
}
read the original abstract
The recently completed SubMIT platform is a small set of servers that provide interactive access to substantial data samples at high speeds, enabling sophisticated data analyses with very fast turnaround times. Additionally, it seamlessly integrates massive processing resources for large-scale tasks by connecting to a set of powerful batch processing systems. It serves as an ideal prototype for an Analysis Facility tailored to meet the demanding data and computational requirements anticipated during the High-Luminosity phase of the Large Hadron Collider. The key features that make this facility so powerful include highly optimized data access with a minimum of 100Gbps networking per server, a large managed NVMe storage system, and a substantial spinning-disk Ceph file system. The platform integrates a diverse set of high multicore CPU machines for tasks benefiting from the multithreading and GPU resources for example for neural network training. SubMIT also provides and supports a flexible environment for users to manage their own software needs for example by using containers. This article describes the facility, its users, and a few complementary, generic and real-life analyses that are used to benchmark its various capabilities.
Forward citations
Cited by 3 Pith papers
-
Lattice evidence that scalar glueballs are small
First lattice extraction of scalar glueball gravitational form factors gives a mass radius of 0.263(31) fm, smaller than typical hadrons.
-
Simplex Demixing: Disentangling Multiple Light-Flavor Jets at Colliders
Simplex demixing recovers T mutually irreducible jet-flavor topics from M mixed samples via the (T−1)-simplex geometry of a multi-category classifier, demonstrated on Pythia dijets.
-
D$e^+e^-$ffusion: Capturing the Beam-Beam Physics of $e^+e^-$ Collisions with Diffusion Models
A diffusion model trained on GuineaPig++ reproduces FCC-ee beam-induced pair-production distributions at particle and detector level, about 10^4 times faster.
Reference graph
Works this paper leans on
-
[1]
Bird, I., et al.: Update of the computing models of the WLCG and the LHC experiments. Technical report (2014). https://cds.cern.ch/record/1695401
arXiv 2014
-
[2]
: Distributing LHC application software and conditions databases using the CernVM file system
Blomer, J., Aguado-S´ anchez, C., et al. : Distributing LHC application software and conditions databases using the CernVM file system. Journal of Physics: Con- ference Series 331(4), 042003 (2011) https://doi.org/10.1088/1742-6596/331/4/ 042003
-
[3]
Brun, R., Rademakers, F.: ROOT — An object oriented data analysis framework. Nucl. Instrum. Methods Phys. Res., Sect. A 389(1), 81–86 (1997) https://doi. org/10.1016/S0168-9002(97)00048-X
-
[4]
Rizzi, A., Petrucciani, G., Peruzzi, M.: A further reduction in CMS event data for analysis: the NANOAOD format. EPJ Web Conf. 214, 06021 (2019) https: //doi.org/10.1051/epjconf/201921406021
arXiv 2019
-
[8]
: Coffea: Columnar object framework for effective analysis
Smith, N., Gray, L., et al. : Coffea: Columnar object framework for effective analysis. EPJ Web Conf. 245, 06012 (2020) https://doi.org/10.1051/epjconf/ 202024506012
doi:10.1051/epjconf/ 2020
-
[9]
https://servicex.readthedocs.io/en/latest/
IRIS-HEP: ServiceX. https://servicex.readthedocs.io/en/latest/
- [10]
- [11]
Show all 51 references
-
[12]
https://spark.apache.org
Apache Software Foundation: Spark. https://spark.apache.org
-
[13]
Ciangottini, D., et al.: Analysis facilities white paper (2024) arXiv:2404.02100
2024 arXiv
-
[14]
Future Generation Computer 24 Systems 78, 1071–1078 (2018) https://doi.org/10.1016/j.future.2016.11.035
Piparo, D., Tejedor, E., Mato, P., Mascetti, L., Moscicki, J., Lamanna, M.: Swan: A service for interactive analysis in the cloud. Future Generation Computer 24 Systems 78, 1071–1078 (2018) https://doi.org/10.1016/j.future.2016.11.035
2018 doi
-
[15]
Kluyver, T., Ragan-Kelley, B., et al.: Jupyter Notebooks—a publishing format for reproducible computational workflows, 87–90 (2016) https://doi.org/10.3233/ 978-1-61499-649-1-87
2016
-
[16]
https://kubernetes.io
The Kubernetes Authors: Kubernetes. https://kubernetes.io
-
[17]
Concurrency and Computation - Practice and Experience 17(2-4), 323–356 (2005) https://doi.org/10.1002/cpe.938
Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. Concurrency and Computation - Practice and Experience 17(2-4), 323–356 (2005) https://doi.org/10.1002/cpe.938
2005 doi
-
[18]
https://coffea
University of Nebraska–Lincoln: Coffea-Casa Analysis Facility. https://coffea. casa/
-
[19]
https://eafjupyter.readthedocs.io/en/latest/
Fermilab: Elastic Analysis Facility. https://eafjupyter.readthedocs.io/en/latest/
-
[20]
https://analysis-facility.physics
Purdue university: Purdue analysis facility. https://analysis-facility.physics. purdue.edu/en/latest/
-
[21]
https://usatlas.readthedocs.io/ projects/af-docs/
US ATLAS: US ATLAS analysis facilities. https://usatlas.readthedocs.io/ projects/af-docs/
-
[22]
Journal of Physics: Conference Series 219(5), 052007 (2010) https://doi.org/10.1088/ 1742-6596/219/5/052007
Haupt, A., Kemp, Y.: The NAF: National analysis facility at desy. Journal of Physics: Conference Series 219(5), 052007 (2010) https://doi.org/10.1088/ 1742-6596/219/5/052007
2010
-
[23]
https://infn-cms-analysisfacility.readthedocs
INFN: INFN CMS analysis facility. https://infn-cms-analysisfacility.readthedocs. io/en/latest/
-
[24]
https://jupyterhub.readthedocs.io/en/stable/
Project Jupyter: JupyterHub. https://jupyterhub.readthedocs.io/en/stable/
-
[25]
In: Job Scheduling Strategies for Parallel Processing, pp
Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: Simple linux utility for resource management. In: Job Scheduling Strategies for Parallel Processing, pp. 44–60. Springer, Berlin, Heidelberg (2003)
2003
-
[26]
In: Proc
Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: a scal- able, high-performance distributed file system. In: Proc. 7th Symp. Oper. Syst. Design Implement. OSDI ’06, pp. 307–320. USENIX Association, USA (2006)
2006
-
[27]
WSEAS Transactions on Computers 4(4), 348–353 (2005)
Dorigo, A., Elmer, P., Furano, F., Hanushevsky, A.: XROOTD - a highly scalable architecture for data access. WSEAS Transactions on Computers 4(4), 348–353 (2005)
2005
-
[28]
https://mitqcd.mit.edu/lattice-qcd/
LQCD. https://mitqcd.mit.edu/lattice-qcd/
-
[29]
https://gateway.dask.org 25
Dask core developers: Dask Gateway. https://gateway.dask.org 25
-
[30]
In: Proceedings, 6th Annual International Conference on Parallel Processing and Applied Mathematics, Poznan, Poland (2005)
Graham, R.L., Woodall, T.S., Squyres, J.M.: Open MPI: A flexible high perfor- mance MPI. In: Proceedings, 6th Annual International Conference on Parallel Processing and Applied Mathematics, Poznan, Poland (2005)
2005
-
[31]
https://docs.conda.io/en/latest/
Anaconda, I.: Conda. https://docs.conda.io/en/latest/
-
[32]
https://submit.mit.edu/ submit-users-guide/
Josh, B., et al.: SubMIT Users’ guide. https://submit.mit.edu/ submit-users-guide/
-
[33]
https://arxiv.org/abs/2312.10997
Gao, Y., et al.: Retrieval-Augmented Generation for Large Language Models: A Survey (2024). https://arxiv.org/abs/2312.10997
2024 arXiv
-
[34]
https://oss.oetiker.ch/rrdtool/index.en.html
Oetiker, T.: RRDtool. https://oss.oetiker.ch/rrdtool/index.en.html
-
[35]
: Monitoring with Ganglia, First edition
Massie, M., et al. : Monitoring with Ganglia, First edition. edn. O’Reilly, Beijing (2012 - 2013)
2012
-
[36]
https://iris-hep.org/projects/agc.html
IRIS-HEP: Analysis Grand Challenge. https://iris-hep.org/projects/agc.html
-
[37]
PoS ICHEP2022, 235 (2022) https://doi.org/10.22323/1.414.0235
Held, A., Shadura, O.: The IRIS-HEP Analysis Grand Challenge. PoS ICHEP2022, 235 (2022) https://doi.org/10.22323/1.414.0235
2022 doi
-
[38]
https://opendata.cern.ch/docs/cms-getting-started-2015
CMS Data preservation and open access group: Getting Started with CMS 2015 Open Data. https://opendata.cern.ch/docs/cms-getting-started-2015
2015
-
[39]
EPJ Web Conf
Padulano, V.E., Guiraud, E., Falko, A., Gazzarrini, E., Garcia Garcia, E., Gosein, D.: First implementation and results of the Analysis Grand Challenge with a fully pythonic RDataFrame. EPJ Web Conf. 295, 06011 (2024) https://doi.org/ 10.1051/epjconf/202429506011
2024
-
[40]
IOP Publishing 2438(1), 012118 (2023) https://doi.org/10.1088/1742-6596/2438/1/ 012118
Lopez-Gomez, J., Blomer, J.: RNtuple performance: Status and outlook. IOP Publishing 2438(1), 012118 (2023) https://doi.org/10.1088/1742-6596/2438/1/ 012118
2023 doi
-
[41]
CMS Collaboration: Measurement of inclusive and differential cross sections for W +W− production in proton-proton collisions at √s = 13.6 TeV. Phys. Lett. B 861, 139231 (2025) https://doi.org/10.1016/j.physletb.2024.139231 arXiv:2406.05101
2025
-
[42]
CMS Collaboration: High-precision measurement of the W boson mass with the CMS experiment at the LHC (2024) arXiv:2412.13872
2024 arXiv
-
[43]
CMS Collaboration: Search for the Higgs boson decays to a ρ0, ϕ, or K ∗0 meson and a photon in proton-proton collisions at √s = 13 TeV. Phys. Lett. B 862, 139296 (2025) https://doi.org/10.1016/j.physletb.2025.13929 arXiv:2410.18289
2025
-
[44]
CMS Collaboration: Search for soft unclustered energy patterns in proton-proton collisions at 13 TeV. Phys. Rev. Lett. 133(19), 191902 (2024) https://doi.org/10. 26 1103/PhysRevLett.133.191902 arXiv:2403.05311
2024 arXiv
-
[45]
JINST 12, 10003 (2017) https://doi.org/10.1088/ 1748-0221/12/10/P10003 arXiv:1706.04965
CMS Collaboration: Particle-flow reconstruction and global event description with the CMS detector. JINST 12, 10003 (2017) https://doi.org/10.1088/ 1748-0221/12/10/P10003 arXiv:1706.04965
2017 arXiv
-
[46]
https://cms-nanoaod.github.io/correctionlib/
Correction Lib. https://cms-nanoaod.github.io/correctionlib/
- [47]
-
[48]
CMS Collaboration: The CMS Statistical Analysis and Combination Tool: Combine. Comput. Softw. Big Sci. (2024) https://doi.org/10.1007/ s41781-024-00121-4 arXiv:2404.06614
2024 arXiv
-
[49]
to appear (2025)
Bendavid, J., Walter, D.: Efficient bin by bin likelihood minimization with profiling. to appear (2025)
2025
-
[50]
https://arxiv.org/ abs/2111.09874
Key4hep collaboration: Key4hep: Status and Plans (2021). https://arxiv.org/ abs/2111.09874
2021 arXiv
-
[51]
: DELPHES 3, A modular framework for fast simulation of a generic collider experiment
Favereau, J., et al. : DELPHES 3, A modular framework for fast simulation of a generic collider experiment. JHEP 02, 057 (2014) https://doi.org/10.1007/ JHEP02(2014)057 arXiv:1307.6346
2014 arXiv
-
[52]
(eds.) The TEAL Physics Project at MIT, pp
Dourmashkin, P., Tomasik, M., Rayyan, S.: In: Mintzes, J.J., Walter, E.M. (eds.) The TEAL Physics Project at MIT, pp. 499–520. Springer, Cham (2020). https: //doi.org/10.1007/978-3-030-33600-4 31
2020 doi
-
[53]
ApJ 935(2), 167 (2022) https://doi.org/10.3847/1538-4357/ ac7c74 arXiv:2206.14220 [astro-ph.IM]
Astropy Collaboration: The Astropy Project: Sustaining and Growing a Community-oriented Open-source Project and the Latest Major Release (v5.0) of the Core Package. ApJ 935(2), 167 (2022) https://doi.org/10.3847/1538-4357/ ac7c74 arXiv:2206.14220 [astro-ph.IM]
2022 arXiv
-
[54]
MNRAS 364(4), 1105–1134 (2005) https://doi.org/10.1111/j.1365-2966.2005.09655.x arXiv:astro- ph/0505010 [astro-ph] 27
Springel, V.: The cosmological simulation code GADGET-2. MNRAS 364(4), 1105–1134 (2005) https://doi.org/10.1111/j.1365-2966.2005.09655.x arXiv:astro- ph/0505010 [astro-ph] 27
2005
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.