BOOM and Babamul: a real-time, multi-survey, optical alert broker system operating at scale
Pith reviewed 2026-05-18 02:13 UTC · model grok-4.3
The pith
BOOM processes ZTF alerts at seven times the prior throughput while preserving full feature parity.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
BOOM is an analysis framework for real-time, joint brokering of alert streams from multiple surveys. Built on a Rust-based stack that uses MongoDB for storage, Valkey for in-memory processing, and Kafka for message sharing, the system achieves feature parity with the prior ZTF alert broker while delivering approximately seven times higher throughput. The workflow supports custom filters for targeted transient detection, and the architecture is positioned to scale to the alert volumes expected from LSST, with Babamul serving as its public-facing interface.
What carries the argument
The Rust-MongoDB-Valkey-Kafka architecture, which supplies real-time storage, queuing, and message distribution for high-volume astronomical alert streams.
If this is right
- The broker can ingest and filter the full nightly LSST alert volume in real time.
- Custom filters enable immediate identification of specific transient classes across surveys.
- Babamul will expose the processed LSST alerts to the broader astronomical community.
- The same pipeline supports simultaneous handling of alerts from multiple optical surveys plus multi-messenger triggers.
Where Pith is reading between the lines
- The same stack could be reused for other high-rate data streams such as gravitational-wave or neutrino alerts once interfaces are added.
- Embedding lightweight machine-learning classifiers inside the Valkey layer might reduce the time from alert arrival to classification without extra hardware.
- Public release of Babamul would let external teams test their own filters against real LSST data at scale.
Load-bearing premise
The architecture will retain feature parity and the stated throughput when alert input rates rise from current ZTF levels to the LSST expectation of roughly twenty million alerts per night.
What would settle it
A throughput and completeness test that feeds the system with simulated alert streams at two times ten to the seventh alerts per night and measures both processing latency and retention of all ZTF-equivalent filters.
Figures
read the original abstract
With the arrival of ever higher throughput wide-field surveys and a multitude of multi-messenger and multi-wavelength instruments to complement them, software capable of harnessing these associated data streams is urgently required. To meet these needs, a number of community supported alert brokers have been built, currently focused on processing of Zwicky Transient Facility (ZTF; $\sim 10^5$-$10^6$ alerts per night) with an eye towards Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST; $\sim 2 \times 10^7$ alerts per night). Building upon the system that successfully ran in production for ZTF's first seven years of operation, we introduce BOOM (Burst & Outburst Observations Monitor), an analysis framework focused on real-time, joint brokering of these alert streams. BOOM harnesses the performance of a Rust-based software stack relying on a non-relational MongoDB database combined with a Valkey in-memory processing queue and a Kafka cluster for message sharing. With this system, we demonstrate feature parity with the existing ZTF system with a throughput $\sim 7 \times$ higher. We describe the workflow that enables the real-time processing as well as the results with custom filters we have built to demonstrate the system's capabilities. In conclusion, we present the development roadmap for both BOOM and Babamul - the public-facing LSST alert broker built atop BOOM - as we begin the Rubin era.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces BOOM, a Rust-based real-time analysis framework for joint brokering of optical alert streams that employs a MongoDB non-relational database, Valkey in-memory queue, and Kafka cluster. Building on the prior ZTF production system, it claims feature parity at ∼7× higher throughput and describes custom filters, the workflow for real-time processing, and the development roadmap for Babamul, the public LSST-facing broker built atop BOOM.
Significance. If the performance claims hold under quantified conditions, the work would offer a concrete, modern-technology path toward handling LSST-scale alert volumes (∼2×10^7 per night) while preserving the functionality already demonstrated on ZTF. The architecture choices and emphasis on multi-survey capability address a recognized community need, though the absence of detailed benchmarks currently limits the strength of the readiness argument.
major comments (2)
- [Abstract] Abstract: the claim that the system demonstrates “feature parity with the existing ZTF system with a throughput ∼7× higher” provides no information on the alert rate or latency distribution at which the factor was measured, the hardware baseline used for the comparison, or the quantitative criteria employed to establish feature parity. These omissions are load-bearing for the central assertion of suitability for LSST rates.
- [Results / workflow description] Performance and scaling discussion: the architecture description does not report direct throughput or latency measurements at input rates approaching or exceeding current ZTF levels, nor does it contain scaling tests, queue-depth analysis, or credible extrapolation to the ∼2×10^7 alerts/night LSST expectation. Without such data the claim that the MongoDB–Valkey–Kafka stack will remain stable under a further 20–200× volume increase cannot be evaluated.
minor comments (2)
- [Abstract] The abstract would benefit from a single sentence stating the peak sustained alert rate and median latency achieved in the reported tests.
- Figure captions and table headings should explicitly define all throughput and latency units and the exact hardware configuration used for each measurement.
Simulated Author's Rebuttal
We thank the referee for their careful and constructive review of our manuscript on the BOOM alert broker. We address each major comment below and will revise the manuscript to incorporate additional performance details and clarifications.
read point-by-point responses
-
Referee: [Abstract] Abstract: the claim that the system demonstrates “feature parity with the existing ZTF system with a throughput ∼7× higher” provides no information on the alert rate or latency distribution at which the factor was measured, the hardware baseline used for the comparison, or the quantitative criteria employed to establish feature parity. These omissions are load-bearing for the central assertion of suitability for LSST rates.
Authors: We agree that the abstract would be strengthened by specifying the conditions of the measurement. In the revised manuscript we will state that the ∼7× throughput improvement was measured at ZTF-comparable input rates (∼10^5–10^6 alerts per night) on a standard multi-core server with the MongoDB–Valkey–Kafka stack, using end-to-end processing latency and filter execution time as the primary metrics. Feature parity is defined by replication of the prior ZTF system’s core alert ingestion, filtering, and classification functions. These details will be added to the abstract and elaborated in a new performance subsection. revision: yes
-
Referee: [Results / workflow description] Performance and scaling discussion: the architecture description does not report direct throughput or latency measurements at input rates approaching or exceeding current ZTF levels, nor does it contain scaling tests, queue-depth analysis, or credible extrapolation to the ∼2×10^7 alerts/night LSST expectation. Without such data the claim that the MongoDB–Valkey–Kafka stack will remain stable under a further 20–200× volume increase cannot be evaluated.
Authors: The current text emphasizes architectural design and initial ZTF-scale results. We acknowledge that explicit scaling data and extrapolation are not yet presented. In revision we will add a dedicated performance section reporting measured throughput and latency at ZTF rates, preliminary load tests at elevated rates, queue-depth statistics under sustained operation, and a resource-based extrapolation to LSST volumes that accounts for the horizontal scaling properties of the Rust/Kafka components. This will allow readers to evaluate stability under the projected volume increase. revision: yes
Circularity Check
No circularity: empirical demonstration of system performance
full rationale
The paper reports construction and operation of a software broker (BOOM/Babamul) and states an empirical result: feature parity with the prior ZTF broker plus ~7x throughput. This is presented as a measured outcome from running the Rust-MongoDB-Valkey-Kafka stack on real alert streams, not as a mathematical derivation, fitted parameter renamed as prediction, or self-referential definition. No equations, ansatzes, or uniqueness theorems appear. The architecture description stands independently of the performance numbers, and the throughput claim is benchmarked against an external prior system rather than reducing to the paper's own inputs by construction. Self-citations, if any, are not load-bearing for the central claim.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Alert streams from ZTF and future surveys can be processed in real time using the described database and queue technologies without loss of essential features.
invented entities (2)
-
BOOM framework
no independent evidence
-
Babamul broker
no independent evidence
Lean theorems connected to this paper
-
IndisputableMonolith/Foundation/RealityFromDistinction.leanreality_from_one_distinction unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
BOOM harnesses the performance of a Rust-based software stack relying on a non-relational MongoDB database combined with a Valkey in-memory processing queue and a Kafka cluster for message sharing. With this system, we demonstrate feature parity with the existing ZTF system with a throughput ∼7× higher.
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
Figure 3 shows throughput testing results for both BOOM and Kowalski in terms of alerts processed per second versus the number of worker processes.
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Forward citations
Cited by 2 Pith papers
-
NOMAI : A real-time photometric classifier for superluminous supernovae identification. A science module for the Fink broker
NOMAI applies XGBoost to SALT2 and Rainbow-derived features on ZTF alerts to reach 66% completeness and 58% purity for SLSNe, recovering 22 of 24 known active SLSNe in a two-month real-time test.
-
The ZTF-ULTRASAT experiment: Characterizing the non-transients in ULTRASAT's high cadence survey
ZTF high-cadence data shows RR Lyrae stars and flaring sources can mimic UV transients, with pre-existing ML catalogs offering a concrete mitigation approach.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter doi edition editor eprint howpublished institution journal key month number organization pages publisher school series title misctitle type volume year version url label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts ...
-
[2]
" write newline "" before.all 'output.state := FUNCTION format.url url empty "" new.block "" url * "" * if FUNCTION format.eprint eprint empty "" archivePrefix empty "" archivePrefix "arXiv" = new.block " " eprint * " " * new.block " " eprint * " " * if if if FUNCTION format.doi doi empty "" " " doi * " " * if FUNCTION format.pid doi empty eprint empty ur...
-
[3]
- [1] #1 = = ^ ^ ^ .\!\!^ d .\!\!^ h .\!\!^ m .\!\!^ s .\!\!^ @mss
thebibliography [1] 20pt to REFERENCES 6pt =0pt -12pt 10pt plus 3pt =0pt =0pt =1pt plus 1pt =0pt =0pt -12pt =13pt plus 1pt =20pt =13pt plus 1pt \@M =10000 =-1.0em =0pt =0pt 0pt =0pt =1.0em @enumiv\@empty 10000 10000 `\.\@m \@noitemerr \@latex@warning Empty `thebibliography' environment \@ifnextchar \@reference \@latexerr Missing key on reference command E...
-
[4]
G., Ackermann, M., Adams, J., et al
Aartsen, M., Ackermann, M., Adams, J., et al. 2017, Journal of Instrumentation, 12, P03012, 10.1088/1748-0221/12/03/p03012
-
[5]
2015, Classical and Quantum Gravity, 32, 074001
Aasi et al . 2015, Classical and Quantum Gravity, 32, 074001
work page 2015
-
[6]
Abbott et al. 2017 a , Phys. Rev. Lett., 119, 161101, 10.1103/PhysRevLett.119.161101
-
[7]
2017 b , The Astrophysical Journal Letters, 848, L12
---. 2017 b , The Astrophysical Journal Letters, 848, L12. http://stacks.iop.org/2041-8205/848/i=2/a=L12
work page 2017
-
[8]
2015, Classical and Quantum Gravity, 32, 024001
Acernese et al . 2015, Classical and Quantum Gravity, 32, 024001
work page 2015
-
[9]
Andreoni, I., Coughlin, M. W., Perley, D. A., et al. 2022, Nature, 612, 430–434, 10.1038/s41586-022-05465-8
-
[10]
Bellm , E. C., Kulkarni , S. R., Barlow , T., et al. 2019, , 131, 068003
work page 2019
-
[11]
Bloom , J. S., Giannios , D., Metzger , B. D., et al. 2011, Science, 333, 203, 10.1126/science.1207150
-
[12]
Cabrera , T., Palmese , A., Hu , L., et al. 2024, Phys. Rev. D, 110, 123029, 10.1103/PhysRevD.110.123029
-
[13]
Cao, Y., Kulkarni, S. R., Howell, D. A., et al. 2015, Nature, 521, 328, 10.1038/nature14440
-
[14]
Chambers , K. C., Magnier , E. A., Metcalfe , N., et al. 2016, arXiv e-prints, arXiv:1612.05560. 1612.05560
work page internal anchor Pith review Pith/arXiv arXiv 2016
-
[15]
Cook, D. O., Mazzarella, J. M., Helou, G., et al. 2023, The Astrophysical Journal Supplement Series, 268, 14, 10.3847/1538-4365/acdd06
-
[16]
Coughlin, M. W., Bloom, J. S., Nir, G., et al. 2023, The Astrophysical Journal Supplement Series, 267, 31, 10.3847/1538-4365/acdee1
-
[17]
Coulter , D. A., Jones , D. O., McGill , P., et al. 2023, arXiv e-prints, arXiv:2303.02154. 2303.02154
-
[18]
A., et al., 2017, @doi [Science] 10.1126/science.aap9811 , 358, 1556
Coulter et al. 2017, Science, 358, 1556, 10.1126/science.aap9811
-
[19]
Dekany, R., Smith, R. M., Riddle, R., et al. 2020, Publications of the Astronomical Society of the Pacific, 132, 038001, 10.1088/1538-3873/ab4ca2
-
[20]
Dey, A., Schlegel, D. J., Lang, D., et al. 2019, The Astronomical Journal, 157, 168, 10.3847/1538-3881/ab089d
-
[21]
Duev, D. A., & van der Walt, S. J. 2021, Phenomenological classification of the Zwicky Transient Facility astronomical event alerts. 2111.12142
-
[22]
Duev , D. A., & van der Walt , S. J. 2021, arXiv e-prints, arXiv:2111.12142, 10.48550/arXiv.2111.12142
-
[23]
Duev, D. A., Mahabal, A., Masci, F. J., et al. 2019, Monthly Notices of the Royal Astronomical Society, 489, 3582–3590, 10.1093/mnras/stz2357
-
[24]
Flesch, E. W. 2023, The Open Journal of Astrophysics, 6, 10.21105/astro.2308.01505
-
[25]
2021, The Astronomical Journal, 161, 242, 10.3847/1538-3881/abe9bc
Förster, F., Cabrera-Vives, G., Castillo-Navarrete, E., et al. 2021, The Astronomical Journal, 161, 242, 10.3847/1538-3881/abe9bc
-
[26]
2012, Science, 337, 932, doi: 10.1126/science.1216793
Gehrels, N., & M \'e sz \'a ros, P. 2012, Science, 337, 932, 10.1126/science.1216793
- [27]
-
[28]
Graham, M. J., Kulkarni, S. R., Bellm, E. C., et al. 2019, Publications of the Astronomical Society of the Pacific, 131, 078001, 10.1088/1538-3873/ab006c
-
[29]
J., Hu, L., Palmese, A., et al
Hall, X. J., Hu, L., Palmese, A., et al. 2025, Transient Name Server AstroNote, 84, 1. https://ui.adsabs.harvard.edu/abs/2025TNSAN..84....1H
work page 2025
-
[30]
Ho, A. Y. Q., Goldstein, D. A., Schulze, S., et al. 2019, The Astrophysical Journal, 887, 169, 10.3847/1538-4357/ab55ec
-
[31]
Ho, A. Y. Q., Perley, D. A., Kulkarni, S. R., et al. 2020, The Astrophysical Journal, 895, 49, 10.3847/1538-4357/ab8bcf
-
[32]
Hosseinzadeh, G., Sand, D. J., Valenti, S., et al. 2017, The Astrophysical Journal Letters, 845, L11, 10.3847/2041-8213/aa8402
-
[33]
2022, ApJ, 936, 157, doi: 10.3847/1538-4357/ac7394
Hu, L., Wang, L., Chen, X., & Yang, J. 2022, The Astrophysical Journal, 936, 157, 10.3847/1538-4357/ac7394
-
[34]
2025, arXiv e-prints, arXiv:2506.22626, 10.48550/arXiv.2506.22626
Hu , L., Cabrera , T., Palmese , A., et al. 2025, arXiv e-prints, arXiv:2506.22626, 10.48550/arXiv.2506.22626
-
[35]
LSST: From Science Drivers to Reference Design and Anticipated Data Products , journal =
Ivezi \'c , Z ., Kahn , S. M., Tyson , J. A., et al. 2019, , 873, 111, 10.3847/1538-4357/ab042c
-
[36]
Jegou du Laz, T., Coughlin, M. W., Bachant, P., et al. 2025, 10.22002/640bx-nbn45
-
[37]
Kasliwal , M. M., Nakar , E., Singer , L. P., et al. 2017, Science, 358, 1559, 10.1126/science.aap9455
-
[38]
M., Cannella, C., Bagdasaryan, A., et al
Kasliwal , M. M., Cannella , C., Bagdasaryan , A., et al. 2019, , 131, 038003, 10.1088/1538-3873/aafbc2
-
[39]
Liu, C., Miller, A. A., Bloom, J. S., Knop, R. A., & Nugent, P. E. 2025, A Morphological Model to Separate Resolved --unresolved Sources in the DESI Legacy Surveys : Application in the LS4 Alert Stream , arXiv, 10.48550/arXiv.2505.17174
-
[40]
The Zwicky Transient Facility: Data Processing, Products, and Archive
Masci , F. J., Laher , R. R., Rusholme , B., et al. 2019, , 131, 018003, 10.1088/1538-3873/aae8ac
work page internal anchor Pith review doi:10.1088/1538-3873/aae8ac 2019
-
[41]
2021, The Astronomical Journal, 161, 107, 10.3847/1538-3881/abd703
Matheson, T., Stubens, C., Wolf, N., et al. 2021, The Astronomical Journal, 161, 107, 10.3847/1538-3881/abd703
-
[42]
2009, The Astrophysical Journal, 702, 791
Meegan et al. 2009, The Astrophysical Journal, 702, 791. http://stacks.iop.org/0004-637X/702/i=1/a=791
work page 2009
-
[43]
Miller, A. A., Magee, M. R., Polin, A., et al. 2020, The Astrophysical Journal, 898, 56, 10.3847/1538-4357/ab9e05
-
[44]
Miller, A. A., Abrams, N. S., Aldering, G., et al. 2025, The La Silla Schmidt Southern Survey. 2503.14579
-
[45]
Möller, A., Peloton, J., Ishida, E. E. O., et al. 2020, Monthly Notices of the Royal Astronomical Society, 501, 3272, 10.1093/mnras/staa3602
-
[46]
2019, , 631, A147, 10.1051/0004-6361/201935634
Nordin , J., Brinnel , V., van Santen , J., et al. 2019, , 631, A147, 10.1051/0004-6361/201935634
-
[47]
2009, The Astrophysical Journal, 701, 824
Nysewander, M., Fruchter, A., & Pe'Er, A. 2009, The Astrophysical Journal, 701, 824
work page 2009
-
[48]
2022, Transient Name Server AstroNote, 107, 1
Palmese, A., Wang, L., Chen, X., et al. 2022, Transient Name Server AstroNote, 107, 1. https://ui.adsabs.harvard.edu/abs/2022TNSAN.107....1P
work page 2022
-
[49]
Perley , D. A., Mazzali , P. A., Yan , L., et al. 2019, , 484, 1031, 10.1093/mnras/sty3420
-
[50]
Perley, D. A., Ho, A. Y. Q., Yao, Y., et al. 2021, Monthly Notices of the Royal Astronomical Society, 508, 5138–5147, 10.1093/mnras/stab2785
-
[51]
Prentice, S. J., Maguire, K., Smartt, S. J., et al. 2018, The Astrophysical Journal, 865, L3, 10.3847/2041-8213/aadd90
-
[52]
Rehemtulla , N., Miller , A. A., Jegou Du Laz , T., et al. 2024, , 972, 7, 10.3847/1538-4357/ad5666
- [53]
-
[54]
Singer, L., & Racusin, J. 2023, Bulletin of the AAS, 55
work page 2023
-
[55]
J., et al., 2017, @doi [Nature] 10.1038/nature24303 , 551, 75
Smartt et al. 2017, Nature, 551, 75 EP . http://dx.doi.org/10.1038/nature24303
-
[56]
Smith, K. W., Williams, R. D., Young, D. R., et al. 2019, Research Notes of the AAS, 3, 26, 10.3847/2515-5172/ab020f
-
[57]
Street, R. A., Bowman, M., Saunders, E. S., & Boroson, T. 2018, in Software and Cyberinfrastructure for Astronomy V, ed. J. C. Guzman & J. Ibsen, Vol. 10707, International Society for Optics and Photonics (SPIE), 274 -- 284, 10.1117/12.2312293
-
[59]
Summary of the content and survey properties
---. 2023 b , Astronomy & Astrophysics, 674, A1, 10.1051/0004-6361/202243940
-
[60]
van der Walt and Arien Crellin-Quick and Joshua S
van der Walt, S. J., Crellin-Quick, A., & Bloom, J. S. 2019, Journal of Open Source Software, 4, 10.21105/joss.01247
-
[61]
Handbook of X-ray and Gamma-ray Astrophysics , year = 2022, editor =
Yuan, W., Zhang, C., Chen, Y., & Ling, Z. 2022, The Einstein Probe Mission (Springer Nature Singapore), 1–30, 10.1007/978-981-16-4544-0_151-1
-
[62]
T., Mishra-Sharma, S., & Villar, V
Zhang, G., Helfer, T., Gagliano, A. T., Mishra-Sharma, S., & Villar, V. A. 2024, Maven: A Multimodal Foundation Model for Supernova Science. 2408.16829
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.