{"id":"d620970e-49c4-4db9-b151-4478f4f9de96","arxiv_id":"2607.16059","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new portal and toolkit give researchers managed, query-style access to flagship HACC cosmological simulations, scaling from laptop-sized subsets to leadership-class HPC jobs.","lead":"OpenCosmo is a web portal and Python toolkit for pulling custom subsets out of huge cosmological simulations, including the two-trillion-particle Frontier-E gravity-only run. It aims to let scientists query and analyze petabyte-scale data from a laptop, without needing to become HPC experts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Production-ready, minutes-latency claim rests on one anecdote; no benchmark, failure-rate data, or evidence of the OLCF endpoint is provided.","rationale":"The reader's weakest assumption is well-targeted: reliability of the multi-facility orchestration layer is the make-or-break premise for the paper's main claim. My pass through the manuscript found no benchmark or independent verification of this layer, and one additional gap the reader implied but did not emphasize: Figure 6 and Table 1 include OLCF/Frontier-E, but §5.4.4 only claims container compatibility at NERSC and ALCF, and no OLCF endpoint deployment is described. The paper deserves credit for the public toolkit URL, public container images, and a plausibly complete architecture description; the latency claim is not impossible. However, 'production-ready' is an operational claim, and the only quantitative evidence is a single five-minute example. Since this gap is addressable rather than fatal, the CONDITIONAL verdict is appropriate; I would not move it. I also noted §2.3's inference that 1.7e7 returned halos 'indicates the sample is complete' is under-justified (being below a 1e8 cap does not prove completeness), but that is a data-product caveat, not the central infrastructure claim.","tokens_in":19315,"tokens_out":5376,"duration_ms":49889,"concrete_test":"Publish a benchmark run through the production portal: N=20 repetitions each of the §2.3 Frontier-E halo query, a Last Journey particle query, a SciDAC galaxy query, a Diffsky catalog query, and a Frontier-E map query, with per-stage timestamps (Flows submission, endpoint resolution, scheduler queue, compute, transfer) taken from Globus Flows logs and scheduler accounting. Also record success rate and include an OLCF-hosted query. If median end-to-end latency is not in the claimed minutes range, or any facility endpoint is missing/fails, the 'production-ready' claim should be downgraded to pilot deployment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — a production-ready, multi-facility platform that 'brings analysis to data' and returns query results in minutes — depends on the Globus Flows/Compute orchestration chain working at every facility hosting data. The paper's support for this is one worked example (§2.3, ~5 min for one Frontier-E query) and a qualitative scheduler remark (§5.1). No end-to-end latency distribution, queue-time breakdown, success/failure rate, endpoint availability, or load test is reported. The container-compatibility statement (§5.4.4) names only NERSC and ALCF, yet Figure 6 and the dataset list include OLCF as a hosting facility; there is no stated deployment or successful query at OLCF. The conclusion's 'production-ready / 1.0 releases' assertion (§6) is therefore not backed by operational evidence, and the paper even concedes that HPC deployment tooling is immature and that persistent endpoints may be replaced. This does not mean the system fails; it means the load-bearing operational premise is currently asserted rather than demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OpenCosmo is presented as a production-ready, multi-facility data-access and analysis platform for large HACC/CRK-HACC cosmological simulations. The paper describes a web portal (query tasks for halo catalogs, profiles, particles, galaxies, and lightcone maps), a lazy-evaluation Python toolkit with transparent MPI support, and a Globus Flows/Compute orchestration layer that routes queries to the facility hosting each dataset. Datasets include the Frontier-E gravity-only run, Last Journey, Diffsky, two Discovery simulations, and a 64-member SciDAC 128 SG5 hydrodynamic suite. The central value proposition is that users can request and analyze subsets of petabyte-scale data without HPC expertise, with results typically returned in minutes, and that the same toolkit runs unchanged from laptops to leadership-class systems. The paper is primarily a system description; it includes one worked end-to-end example but no operational validation data.","tokens_in":19547,"tokens_out":9050,"duration_ms":72938,"significance":"If the platform behaves as claimed, this is a valuable community resource: it lowers the barrier to working with extreme-scale simulation data, builds on mature Globus services, and the registry/task-definition design cleanly separates domain-specific from domain-generic components, giving the infrastructure genuine transfer potential. The manuscript also ships concrete artifacts — open-source toolkit code, publicly available container images, example notebooks, and a reproducible halo-mass-function example — which are strengths. The main risk is that the headline operational claims (minutes-latency, multi-facility production readiness, transparent laptop-to-supercomputer scaling) are supported by a single anecdote and qualitative statements rather than measurements; the architecture is coherent, but its validation is incomplete.","major_comments":[{"comment":"The central operational claim — that results are 'typically returned within minutes' and that the platform is 'production-ready' — is supported only by the single worked example in §2.3 and the qualitative scheduler remark in §5.1. No end-to-end latency distribution, queue-time breakdown, success/failure rate, endpoint availability, or load test is reported for the Globus Flows/Compute chain. Because this chain is the load-bearing premise for 'bring analysis to data,' the manuscript currently asserts rather than demonstrates the claimed latency and reliability. Please add a representative measurement campaign (e.g., a latency CDF over a set of queries at each facility, including queue and transfer components) or temper the production-readiness claim accordingly.","section":"§4, §5.4, §6"},{"comment":"OLCF is listed in Figure 6 and Frontier-E GO is described as hosted at OLCF (§3.1), and the endpoint-resolution design routes each query to the facility housing the requested dataset (§5.4.3). However, §5.4.4 states that the containerized query environment is 'known to work' only at NERSC and ALCF, and §4.5's access model names only NERSC/ALCF accounts. No successful OLCF query or OLCF compute-endpoint deployment is described. The multi-facility claim is therefore not yet substantiated for OLCF. If Frontier-E GO data are actually replicated to NERSC/ALCF and current queries run there, state that explicitly; otherwise document the OLCF deployment and at least one successful end-to-end query.","section":"§5.4.4, Figure 6, §4.5"},{"comment":"The claim that 'the same tooling runs unchanged from laptops to leadership-class systems' and that a serial script can be run across many nodes without modifications is central to the toolkit's value proposition, but no benchmark or scaling test is provided. Figure 5 demonstrates a scale-transition workflow but gives no runtimes, memory usage, or parallel efficiency. In fact, §5.3 notes that the portal's largest queries currently run on a single compute node, so the 'leadership-class' mode is an assertion about untested behavior. Please include at least one strong- or weak-scaling measurement of the toolkit, or clearly distinguish 'designed to support' from 'validated at' leadership scale.","section":"§2.1, §5.3"}],"minor_comments":[{"comment":"The sentence 'well below the 10^8-halo limit... indicating that the sample is complete' overreaches: a non-binding cap does not establish catalog completeness. It shows only that the cap was not reached; if completeness is intended, a separate validation should be cited or described.","section":"§2.3"},{"comment":"The wall-clock accounting is inconsistent: the text first says the query completes in ~5 minutes and later says ~10 minutes are added for query completion and download. Please clarify whether the 5 minutes excludes queue/scheduler and transfer time.","section":"§2.3"},{"comment":"The paper calls OpenCosmo a 'community portal' ready for general use, but access is currently limited to NERSC/ALCF account holders and DOE-laboratory affiliates. This is a legitimate restriction, but it should be stated earlier (abstract or introduction) so that readers are not misled about public availability.","section":"§4.5, §6"}],"recommendation":"major_revision","confidential_remarks":"This is a software/platform description rather than a scientific analysis paper. The architecture is credible and the open-source artifacts are a strength, but the 'production-ready / minutes-latency' headline is not yet backed by operational evidence. The requested additions are within scope and should be obtainable from the authors' existing deployment; I view this as a major revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid infrastructure paper with one genuinely valuable new data release, and one overreach in the prose. The SciDAC 128 SG5 suite — 64 hydrodynamic runs with shared initial conditions and varied subgrid parameters — is exactly the kind of ensemble the community needs, and the OpenCosmo toolkit that ships with it is more than a portal wrapper: lazy queries, unit-aware expressions, automatic linking of halos to profiles to particles, and MPI transparency so a laptop script runs unchanged on a supercomputer. That design is the paper's real contribution. The MCP agent interface is a nice, timely add. Citations to HACC, Globus, HDF5, HEALPix are all appropriate; there's no circular-science problem here, the system is demonstrated on the authors' own simulations but that's what the platform is for.\n\nThe soft spot is the operational claim, not the architecture. 'Production-ready' and 'results typically returned within minutes' rest on a single worked example in §2.3: no latency distribution, no success/failure counts, no queue-time breakdown, no load test. The completeness inference in that same section is too quick: the query returned 1.7×10^7 halos against a 10^8 cap, and the paper says that 'indicates that the sample is complete' — but the cap is an output limit, not a statement about catalog completeness. Those two things need to be separated. Minor fix, but easy to catch.\n\nThe multi-facility story has a real hole. §5.4.4 says containers are known to work at NERSC and ALCF, but Figure 6 and the dataset list include OLCF's Frontier, where the flagship Frontier-E GO data presumably lives. There is no stated deployment or successful query at OLCF. The paper itself concedes that HPC deployment tooling is immature and persistent endpoints may be replaced. None of this means the system doesn't work; it means the central claim of reliable multi-facility operation is asserted rather than demonstrated.\n\nThis paper deserves a serious referee. The data release alone is worth reviewing, and the toolkit is well enough described that someone could test it on the public parts (the GitHub repo and container images are linked). My recommendation: send it to review, and ask the authors to add operational benchmarks, clarify the OLCF deployment status, and fix the completeness wording. For the reading group, I'd bring it up when we talk about data-access infrastructure; it's a useful concrete example of the bring-analysis-to-data model.","headline":"Solid infrastructure paper with a genuinely useful new data release; the 'production-ready / minutes-latency' claim is thinner than the prose suggests, but all issues are addressable.","tokens_in":20093,"tokens_out":3165,"would_cite":true,"duration_ms":28273,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenCosmo is a production-ready portal and toolkit for querying and analyzing petabyte-scale cosmological simulations, bringing analysis to the data and returning custom subsets in minutes.","keywords":["OpenCosmo","cosmological simulations","data portal","high-performance computing","HACC","halo catalogs","lightcone maps","Python toolkit"],"falsifier":"Submit a standard halo query (e.g., mass greater than 10^13 solar masses) against the two-trillion-particle simulation from a fresh account and measure time-to-delivery; if delivery is not 'within minutes' or the endpoint fails, the platform's central promise fails. A concurrent load test with dozens of simultaneous users would also reveal whether per-user rate limits and scheduler allocations actually protect the service.","tokens_in":19234,"feed_emoji":"🔭","tokens_out":3892,"duration_ms":38282,"temperature":0.7,"pith_summary":"OpenCosmo claims to make petabyte-scale cosmological simulation datasets tractable by bringing analysis to the data rather than moving the data to the analyst. A web portal serves custom subsets—halo catalogs, particle data, galaxy catalogs, and lightcone maps—from several flagship simulation runs, with results returned in minutes. A companion Python toolkit reads the returned files and scales without modification from a laptop to a full supercomputer, so the same script works at every data size. The paper argues that this pattern, built on standard orchestration and container infrastructure, is production-ready and can be extended to other data-intensive science domains.","feed_headline":"Portal serves trillion-particle simulations to laptops in minutes","feed_subtitle":"Query halo catalogs, particle data, galaxy catalogs, and lightcones from flagship simulations without HPC expertise.","key_machinery":"The central mechanism is a multi-facility orchestration layer built on Globus Flows and Globus Compute. A submitted query is routed to the facility hosting the requested dataset, where a pre-registered, containerized function is submitted to the local scheduler; results are returned as HDF5 in the OpenCosmo Data Format and read by the OpenCosmo toolkit, a lazy-query engine that keeps most data on disk and runs serially or under MPI. The dataset registry and swappable JSON task definitions keep domain-specific knowledge out of the infrastructure layer, which is what makes the platform adaptable to other sciences.","core_discovery":"The paper's central claim is that OpenCosmo is a fully managed platform for analyzing petabyte-scale cosmological datasets on leadership-class computing facilities, and that the same tooling runs unchanged from laptops to those systems. It backs this with a working portal that executes predefined query tasks—halo finding, profile and particle retrieval, galaxy catalogs, and HEALPix maps—against flagship simulations including a two-trillion-particle gravity-only run, and returns HDF5 files typically within minutes. The data products are drawn from completed simulations with existing validation, and the toolkit is a lazy-query engine with transparent MPI support that automatically links relate","pith_inferences":["If the promised latency and multi-facility reliability hold under real community load, the architecture points toward a general 'Science as a Service' pattern—domain-agnostic orchestration over domain-specific query tooling—that could be adopted by other data-intensive fields such as climate or genomics.","The stated next steps—user-submitted code and remote Python access to full datasets—are qualitatively harder than the current predefined queries, because they require secure code execution and environment construction on HPC resources; their feasibility is not established by this paper.","The MCP-based agent interface suggests a testable extension: an AI agent could be benchmarked against a human researcher on a standard analysis task to measure whether agent-driven workflows actually reduce time-to-insight."],"forward_implications":["Researchers can compare observational data to synthetic galaxy catalogs without downloading or managing large simulation files.","The same Python script runs on a laptop subset and across an entire flagship simulation on a supercomputer, lowering the barrier for students and newcomers.","The 64-member hydrodynamic suite with shared initial conditions enables systematic studies of how subgrid parameters affect astrophysical predictions.","Lightcone catalogs and HEALPix maps from the largest gravity-only run support weak-lensing and survey-comparison analyses directly in the observer frame."],"fun_headline_variants":["Trillion-particle cosmology, on-demand via web portal","OpenCosmo: supercomputing data, laptop-friendly access","Petabyte simulations, analyzed with unmodified laptop code","Halo catalogs and lightcones, delivered in minutes","Cosmology flagship runs, now a web query away"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim rests on the multi-facility orchestration layer—queries routed through orchestration services to compute endpoints at each facility, which submit containerized jobs to local schedulers and return results in minutes—reliably working as described; the paper asserts this but provides no benchmark, load test, or independent verification.","fun_headline_variants_meta":{"raw":{"variants":["Trillion-particle cosmology, on-demand via web portal","OpenCosmo: supercomputing data, laptop-friendly access","Petabyte simulations, analyzed with unmodified laptop code","Halo catalogs and lightcones, delivered in minutes","Cosmology flagship runs, now a web query away"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1528,"prompt_tokens":729,"completion_tokens":799,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":719}},"tokens_in":473,"tokens_out":799,"duration_ms":8784,"temperature":1.0,"reasoning_tokens":719,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T21:27:44.613615+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Submit a standard halo query (e.g., mass greater than 10^13 solar masses) against the two-trillion-particle simulation from a fresh account and measure time-to-delivery; if delivery is not 'within minutes' or the endpoint fails, the platform's central promise fails. A concurrent load test with dozens of simultaneous users would also reveal whether per-user rate limits and scheduler allocations actually protect the service.","supporting_citations":[],"review_version":1}