Rubin Observatory's data movement relies on Rucio and FTS for transfers, plus three custom tools that tie Rucio to the Data Butler registry.
The Vera C. Rubin Observatory Data Butler and Pipeline Execution System
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations.
fields
astro-ph.IM 1years
2025 1verdicts
ACCEPT 1representative citing papers
citing papers explorer
-
Data Movement Model for the Vera C. Rubin Observatory
Rubin Observatory's data movement relies on Rucio and FTS for transfers, plus three custom tools that tie Rucio to the Data Butler registry.