Pith. sign in

REVIEW 3 major objections 6 minor 17 references

Data Movement Model for the Vera C. Rubin Observatory

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that Rubin's replicated data can be made self-registering at every facility by chaining Rucio and FTS transfers into Butler ingestion via three custom daemons, guaranteeing that pipelines see transferred data immediately.

desk verdict A clear, honest systems description of Rubin's data movement stack; the load-bearing path-preservation claim needs one end-to-end check before publication. read the letter →

arxiv 2507.12955 v1 pith:3IJWLZGU submitted 2025-07-17 astro-ph.IM

Fabio Hernandez (1) , Mark G. Beckett (2) , Andrew Hanushevsky (3) , Tim Jenness (4) , Kian-Tat Lim (3) , Peter Love (5) , Timothy Noble (6) , Stephen R. Pietrowicz (7)
show 11 more authors
Wei Yang (3) ((1) CC-IN2P3/CNRS (2) Institute for Astronomy University of Edinburgh (3) SLAC National Accelerator Laboratory (4) Vera C. Rubin Observatory Project Office (5) Lancaster University (6) Rutherford Appleton Laboratory Science Technology Facilities Council (7) National Center for Supercomputing Applications University of Illinois at Urbana-Champaign)
This is my paper · ORCID
classification astro-ph.IM
keywords VeraC.RubinObservatoryLegacySurveyofSpaceandTimedatamovementRucioFileTransferServiceButlerreplicationtime-domainalerts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Every night the Vera C. Rubin Observatory's 3.2-gigapixel camera will record roughly five petabytes of new images per year, and those images must flow from the Chilean summit to processing centers on three continents, with alerts about transients due within sixty seconds of readout. This paper presents the operational design that keeps that multi-site dataset coherent: the generic transfer pair Rucio and FTS moves files between storage elements, while three custom tools—rucio_register, HermesK, and ingestd—ensure that what arrives at a facility is immediately known to that facility's Butler repository, the storage-access layer the science pipelines actually query. The reason to care is that the entire LSST science program, from nightly alert generation to annual data releases, depends on replicated data becoming available to pipelines the moment a transfer completes, not merely sitting on a disk somewhere.

What carries the argument

The load-bearing mechanism is the registration-to-ingestion chain: rucio_register → Rucio Datasets and subscriptions → FTS transfer → HermesK Kafka topic → ingestd → Butler ingestion, with identity filename mapping preserving the paths the Butler expects at the destination. The named objects are the three custom components: rucio_register, which registers selected Butler files into Rucio with Rubin metadata; HermesK, which reroutes Rucio transfer-completion notices through Kafka topics named per RSE; and ingestd, a stateless daemon that performs idempotent batch ingestion of newly arrived files into the local Butler. The chain also solves a small-file problem: JSON and YAML metadata files from a processing run are zipped into single containers so Rucio and FTS move one large object instead of tens of thousands of tiny ones.

What would settle it

Measure the wall-clock time from a completed exposure readout at the summit to its ingestion into the US archive Butler repository via the production transfer path: a consistent per-exposure time above about seven seconds, or an alert issued more than sixty seconds after readout, would falsify the prompt-processing model; likewise, any file Rucio marks as transferred but that is missing from the destination Butler registry would show the synchronization chain is broken.

Watch

Extended reading notes

Core claim

The central claim is that a closed integration loop makes inter-facility replication self-registering. rucio_register selects already-written files from a source Butler repository, attaches a compact JSON record of Rubin-specific metadata, and groups them into Rucio Datasets; preconfigured subscriptions translate dataset registrations into transfer rules that Rucio hands to FTS. Each facility exposes Rucio Storage Elements configured with identity logical-to-physical filename mapping, so pathnames relative to the Butler datastore are preserved verbatim. When FTS reports a completed transfer, HermesK, a modification of Rucio's Hermes daemon, publishes a message to a Kafka topic named after the destination storage element, and the receiving facility's ingestd daemon consumes that message and ingests the new file into the local Butler repository. If the model behaves as designed, every replicated file is automatically registered at its destination and immediately visible to the processing pipelines.

Load-bearing premise

The design assumes the end-to-end latency target of seven seconds for four gigabytes of compressed data per exposure is actually achievable over the international network links, but the paper states this as a target without presenting measurements or network validation.

Editorial extensions

If this is right

  • A successfully replicated file is automatically ingested into the destination Butler repository, so science pipelines see newly arrived data without any manual registration step.
  • Rucio tracks only files that are replicated between facilities; intermediates, which can be ten times more numerous than final products, stay in local Butler registries and keep the global catalog small.
  • Naming Kafka topics after the destination storage element means each facility receives only notifications for its own endpoints, letting ingestd scale to multiple consumers as the file count grows.
  • Grouping files into Rucio Datasets with spatially meaningful names lets subscriptions associate sky regions with storage elements and provides a deterministic dataset-closure signal.
  • Maintaining separate input and products storage elements lets the project prioritize raw-data transfers over regenerable data products and protects raw data against modification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the register-transfer-notify-ingest pattern is generic; any multi-site experiment whose local catalog must reflect replicated files could reuse this design, because the daemons are stateless and ingestion is batched and idempotent.
  • Editorial inference: the seven-second transfer target is a design budget, not a measured result, so an instrumented end-to-end test of exposure-to-ingestion latency on the production links would be the natural validation of the 60-second alert commitment.
  • Editorial inference: the paper does not describe monitoring for divergence between Rucio's record of completed transfers and the destination Butler registry; a lag alert on the HermesK-to-ingestd path would close the one loop where silent inconsistency could enter.
  • Editorial inference: once the annual Data Release is registered as Rucio Datasets, the same subscription machinery could drive the tiered distribution to independent data access centers, reducing each release campaign to dataset-to-RSE rules.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper describes the data movement architecture for the Vera C. Rubin Observatory, covering three use cases: raw data transfer from the summit in Chile to the US archive at SLAC, replication from the archive to the France and UK data facilities for processing, and distribution of annual data releases to data access centers. The proposed model combines Rucio and FTS for transfer management with three custom tools: rucio_register (selection and registration of Butler files into Rucio datasets), HermesK (Kafka-based notification routing of successful transfers), and ingestd (automatic ingestion of replicated files into destination Butler repositories). The paper argues that this design keeps each facility's Butler repository synchronized without manual intervention, relying on Rucio's identity logical-to-physical filename mapping to preserve paths relative to the Butler datastore.

Significance. If the described architecture works as claimed, it is a valuable operational blueprint for a multi-petabyte, multi-continental scientific data system, and it offers a concrete integration pattern between Rucio and a domain-specific storage abstraction (Data Butler). The paper's strengths are its clear use-case decomposition, explicit tool choices with public code repositories, and the identification of design conventions (e.g., separate input and product RSEs, cost-motivated avoidance of registering intermediate files in Rucio). However, the central automatic-ingestion claim rests on a path-preservation invariant and a minimal metadata schema that are asserted but not demonstrated; no end-to-end transfer, schema, or test is reported. The paper is also entirely free of performance measurements, which is acceptable for an architecture description only if the targets are explicitly labeled as targets.

major comments (3)
  1. [Section 3, 'All RSEs are configured...'] The automatic ingestion claim depends on the invariant that a replicated file's physical path at the destination is exactly the path expected by the destination Butler datastore. The identity logical-to-physical filename mapping preserves relative pathnames only if every RSE's root prefix coincides with the corresponding Butler datastore root, or if rucio_register emits LFNs that are already Butler-relative. The paper does not state this configuration invariant, nor does it provide evidence from a live transfer. Because Section 3.2 says ingestd registers files into the local Butler repository, a path mismatch would place files outside the repository and break pipeline access. Please state the invariant explicitly and provide at least one end-to-end transfer test, or clearly label this as an unvalidated design assumption.
  2. [Section 3.1, 'rucio_register' paragraph] The JSON metadata attached by rucio_register is described as a 'minimal set of information' sufficient for Butler ingestion, but no schema, example, or coverage test is given. Without this, a reader cannot assess whether all file types (FITS, Parquet, JSON/YAML, and the Zip aggregates mentioned in Section 3.1) are handled correctly. Please include the metadata schema (or a link to it) and describe how it is validated for the distinct file categories.
  3. [Section 2.1, latency target] The seven-second end-to-end latency target for transferring four gigabytes of compressed exposure data is load-bearing for the 60-second prompt-alert requirement, but it is presented as a target with no measurements, network characterization, or reference. If this is a design target, say so explicitly and cite any related network validation or requirements document; if it has been tested, report the test conditions. This is a performance claim rather than a correctness claim, but it should be labeled to avoid overstatement.
minor comments (6)
  1. [Section 3, first paragraph] There is a typo: 'webDA V protocol' should read 'WebDAV protocol'.
  2. [Section 3.2, last paragraph] The phrase 'triggering o ff known synchronization points' appears to contain a spacing error ('off').
  3. [Section 2, first paragraph] The sentence contains a duplicated article: 'part of the the published release'.
  4. [Section 2.2, paragraph on replication] The 'approximately ten times' ratio for intermediate data relative to final products is given without a citation or derivation; please add a reference or label it as an operational estimate.
  5. [Section 3.2, paragraph on ingestd] The paper mentions idempotent batch ingestion and stateless daemons, but does not discuss failure handling (e.g., retries for failed transfers, handling of partial or duplicate notifications). A brief statement on how consistency is maintained under failures would strengthen the operational description.
  6. [Section 2.2, back-flow paragraph] The flow from processing facilities back to the archive is described at a high level, but it is not stated whether the same rucio_register/subscription mechanism is used symmetrically for final data products; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper is a systems description with no fitted predictions or derivation that reduces to its inputs.

full rationale

This paper is a descriptive architecture overview, not a derivation with predictions. Its claims—that Rucio/FTS handle transfers, that rucio_register, HermesK, and ingestd integrate Rucio with the Data Butler, and that the identity LFN-to-PFN mapping plus JSON metadata allow automatic ingestion—are engineering design statements, not results derived from fitted parameters or from prior theorems. Citations to Rubin technical notes and author papers are used as references for project conventions and background, but none is invoked as an external theorem that forces the architecture. The only potentially load-bearing assertion, that identity mapping preserves pathnames relative to the Butler datastore, is an unvalidated configuration assumption (a correctness risk) rather than a circular step: it does not define the conclusion in terms of itself. No equation reduces to another, and no fitted quantity is renamed as a prediction. Accordingly, no circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 3 invented entities

The paper introduces no physical entities, fitted parameters, or mathematical postulates. The free-parameter list is empty because numbers like 5 PB/yr, 4 GB, and 7 seconds are stated requirements or targets, not parameters fitted to data. The listed axioms are operational assumptions about network, storage configuration, and notification reliability. The three custom software tools are included as invented entities because they are new system components, each with public code as independent evidence.

assumptions (4)
  • domain assumption The dedicated network links among the summit, SLAC, and European facilities have sufficient capacity and reliability to move 5 PB per year and to transfer 4 GB per exposure within 7 seconds.
    Invoked in Sections 2.1 and 2.2 as the basis for the transfer latency and throughput targets; no network measurements are provided.
  • domain assumption All Rucio Storage Elements use the identity logical-to-physical filename mapping, preserving paths relative to the Butler repository's datastore.
    Stated in Section 3 as critical for proper file replication; if path mapping differs at any site, ingestion into the destination Butler will fail.
  • domain assumption FTS transfer completion notifications are reliable, and Kafka delivers messages without loss to the correct facility, while ingestion remains idempotent.
    Described in Section 3.2 as the synchronization mechanism; the paper acknowledges ordering guarantees are not strictly needed but assumes reliable delivery.
  • domain assumption Each facility's Butler registry can serve as the authoritative local catalog and can be kept consistent through the described replicate-and-ingest process.
    Implied in Sections 2.2 and 3.2; the paper provides no consistency check or reconciliation mechanism beyond idempotent ingestion.
invented entities (3)
  • rucio_register independent evidence
    purpose: Selects existing files from a Butler repository, attaches Rubin-specific metadata, and registers them into Rucio datasets for replication.
    Public GitHub repository at github.com/lsst/rucio_register, though no commit hash is given in the paper.
  • HermesK independent evidence
    purpose: Modified Rucio Hermes daemon that filters transfer completion messages and publishes notifications to Kafka topics named after destination RSEs.
    Public GitHub repository at github.com/lsst-dm/ctrl_rucio_ingest, though no commit hash is given in the paper.
  • ingestd independent evidence
    purpose: Daemon running at each facility that consumes Kafka notifications and ingests newly replicated files into the local Butler repository.
    Public GitHub repository at github.com/lsst-dm/ctrl_ingestd, though no commit hash is given in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data Movement Model for the Vera C. Rubin Observatory." pith.science (2026). https://pith.science/paper/3IJWLZGU

@misc{pith2026250712955,
  author       = {Pith},
  title        = {Pith review of: Data Movement Model for the Vera C. Rubin Observatory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3IJWLZGU}},
  note         = {Machine review of arXiv:2507.12955}
}
read the original abstract

The sky images captured nightly by the camera on the Vera C. Rubin Observatory's telescope will be processed across facilities on three continents. Data acquisition will occur at the observatory's location on Cerro Pach\'{o}n in the Andes mountains of Chile. A first copy of the raw image data set is stored at the summit and immediately transmitted via dedicated network links to the archive center within the US Data Facility at SLAC National Accelerator Laboratory in California, USA and from there to two European facilities for processing and archiving purposes. Data products resulting from periodic processing campaigns of the entire set of images collected since the beginning of the survey are made available to the scientific community in the form of data releases. In this paper we present an overall view of how we leverage the tools selected for managing the movement of data among the Rubin processing and serving facilities, including Rucio and FTS. We also present the tools we developed to integrate Rucio's data model and Rubin's Data Butler, the software abstraction layer that mediates all access to storage by pipeline tasks that implement science algorithms.

Figures

Figures reproduced from arXiv: 2507.12955 by the authors.

Figure 1
Figure 1. Raw images flow from the Summit Site, where the telescope is located in Chile, to the Base Site and then to the Archive Center within the US Data Facility through long haul network links specifically deployed for the needs of the Observatory. Data is transferred from the Archive Center to the European Data Facilities for processing and archiving. The US, UK and France Data Facilities collectively provide the computa… view at source ↗
Figure 2
Figure 2. Number of files and total file sizes from a data preview processing run. facility’s ingestd is configured to only monitor Kafka messages specifically targeted to the facility’s RSEs (see [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. HermesK emits notifications about successful file transfers via Kafka topics named after the destination RSE. At the receiving facility ingestd monitors those notifications and ingests the newly received file into the local Butler repository. The JSON-encoded, Rubin-specific metadata associated to the file when it was first registered into Rucio contains the details needed for ingestion [PITH_FULL_IMAGE:figures/ful… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

  1. [1]

    Ivezi ´c et al., LSST: From Science Drivers to Reference Design and Anticipated Data Products, ApJ 873, 111 (2019), arXiv:0805.2366

    Ž. Ivezi ´c et al., LSST: From Science Drivers to Reference Design and Anticipated Data Products, ApJ 873, 111 (2019), arXiv:0805.2366. 10.3847/1538-4357/ab042c

    arXiv 2019

  2. [2]

    Juri ´c, T

    M. Juri ´c, T. Axelrod, A. Becker, J. Becla, E. Bellm, J. Bosch, D. Ciardi, A. Connolly, G. Dubois-Felsmann, F. Economou et al., Systems Engineering Controlled Document LSE-163, NSF-DOE Vera C. Rubin Observatory (2023), https://lse-163.lsst. io/

    work page 2023

  3. [3]

    Graham, The Rubin Data Products, Abridged (2022), https://doi.org/10

    M. Graham, The Rubin Data Products, Abridged (2022), https://doi.org/10. 5281/zenodo.7011229

    work page 2022

  4. [4]

    Amazon S3 REST API introduction, https://docs.aws.amazon.com/AmazonS3/ latest/API/Welcome.html

    work page

  5. [6]

    Rubin Observatory (2025), https://doi.org/10

    Rubin Observatory Science Pipelines Developers, Project Science Technical Note PSTN-019, NSF-DOE Vera C. Rubin Observatory (2025), https://doi.org/10. 71929/rubin/2570545

    arXiv 2025

  6. [7]

    The Vera C. Rubin Observatory Data Butler and Pipeline Execution System

    T. Jenness, J.F. Bosch, A. Salnikov, N.B. Lust, N.M. Pease, M. Gower, M. Kowa- lik, G.P. Dubois-Felsmann, F. Mueller, P. Schellart, The Vera C. Rubin Observatory Data Butler and pipeline execution system, in "Software and Cyberinfrastructure for Astronomy VII" (2022), V ol. 12189 of Proc. SPIE, p. 1218911, arXiv:2206.14941. 10.1117/12.2629569

    work page Pith review arXiv 2022

  7. [8]

    Bolton et al., Technical Note RTN-086, NSF-DOE Vera C

    A. Bolton et al., Technical Note RTN-086, NSF-DOE Vera C. Rubin Observatory (2025), https://rtn-086.lsst.io/

    work page 2025

  8. [9]

    Barisits, T

    M. Barisits, T. Beermann, F. Berghaus, B. Bockelman, J. Bogado, D. Cameron, D. Christidis, D. Ciangottini, G. Dimitrov, M. Elsing et al., Rucio: Scientific Data Man- agement, Computing and Software for Big Science 3, 11 (2019). 10.1007/s41781-019- 0026-3

    work page doi:10.1007/s41781-019- 2019

Show all 17 references
  1. [10]

    File Transfer Service, https://fts.web.cern.ch/fts

  2. [11]

    XRootD, https://xrootd.github.io

  3. [12]

    Mkrtchyan, K

    T. Mkrtchyan, K. Chitrapu, V . Garonne, D. Litvintsev, S. Meyer, P. Millar, L. Morschel, A. Rossi, M. Sahakyan, dCache: Inter-disciplinary storage system, EPJ Web Conf.251, 02010 (2021). 10.1051/epjconf/202125102010

    2021

  4. [13]

    Dusseault, HTTP Extensions for Web Distributed Authoring and Versioning (Web- DA V), RFC-4918,https://www.ietf.org/rfc/rfc4918.txt

    L. Dusseault, HTTP Extensions for Web Distributed Authoring and Versioning (Web- DA V), RFC-4918,https://www.ietf.org/rfc/rfc4918.txt

  5. [14]

    Fielding, J

    R. Fielding, J. Mogule, H. Frystyk, L. Masinter, P. Leach, T. Berners-Lee, Hypertext Transfer Protocol – HTTP/1.1, RFC-2616, https://datatracker.ietf.org/doc/ html/rfc2616

  6. [15]

    Lim, Data Management Technical Note DMTN-213, NSF-DOE Vera C

    K.T. Lim, Data Management Technical Note DMTN-213, NSF-DOE Vera C. Rubin Observatory (2023), https://dmtn-213.lsst.io/

    2023

  7. [16]

    Le Boulc’h, F

    Q. Le Boulc’h, F. Hernandez, G. Mainetti, The Rubin Observatory’s Legacy Survey of Space and Time DP0.2 processing campaign at CC-IN2P3, EPJ Web of Conf. 295, 04049 (2024). 10.1051/epjconf/202429504049

    2024

  8. [17]

    Fausti Neto, F

    A. Fausti Neto, F. Economou, M.A. Reuter, J. Sick, R. Allbery, A.J. Thornton, Sasquatch: Rubin Observatory metrics and telemetry service, in Software and Cyber- infrastructure for Astronomy VIII, edited by J. Ibsen, G. Chiozzi (2024), V ol. 13101 of Proc. SPIE, p. 131011M. 10....

    2024 doi

  9. [18]

    Ribeiro, R.E

    T. Ribeiro, R.E. Owen, D.J. Mills, M.A. Reuter, A.W. Clements, W. O’Mullane, Re- placing DDS with Apache Kafka as middleware technology for the Rubin Observatory control system, in Software and Cyberinfrastructure for Astronomy VIII, edited by J. Ib- sen, G. Chiozzi (2024), V ...

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.