REVIEW 3 major objections 5 minor 40 references
Using anomaly detection to search for technosignatures in Breakthrough Listen observations
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper claims that a simulation-calibrated anomaly-detection pipeline cuts a 10^11-spectrogram SETI search to a few thousand candidates and that none of them survive human scrutiny.
desk verdict A solid, honest null-result SETI paper with a genuinely new triage pipeline; the improvement claim is plausible but lacks statistical rigor, and the null is conditional on the simulated signal family. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the simulated on-target-only region in a UMAP plane. For each cadence of six 80-by-16-pixel spectrograms, the pipeline computes pairwise cross-correlations, extracts 15 summary statistics, and projects them to two UMAP coordinates; kernel density estimation trained on 640,000 Setigen simulations defines a probability that a cadence belongs to the 'only on-target' category, and a threshold of 0.0618 selects candidates. Two additional scores then rank survivors: a frequency score from bagged Gaussian mixture models fitted to candidate density per band, and a similarity score comparing intra-on-target to on-off distances in a second UMAP embedding. The mechanism's role is to convert the expensive search into a small, rank-ordered list without assuming a specific drift rate or signal morphology.
What would settle it
Insert narrow-band synthetic signals with time-frequency morphologies absent from Setigen's profile library into real off-source cadences and run the pipeline; if the top-percentile 'top-candidate' fraction drops to the random baseline, the simulation-defined accepted region is too narrow and the null result is not informative for those signal classes.
Extended reading notes
Core claim
The central claim is that the two-stage filter pipeline materially improves candidate selection for human vetting, and that on the Breakthrough Listen sample it yields no surviving technosignature. Each six-pointing observing cadence is compressed to 15 cross-correlation statistics, mapped with UMAP into a two-dimensional plane, and accepted only if it falls in the region where simulated on-target-only signals cluster, which cuts the candidate pool by a factor of about $10^5$. The survivors are then ranked twice: a frequency score prefers candidates in radio-quiet parts of the band, and a similarity score prefers cadences whose on-target observations resemble each other in UMAP space while staying distant from off-target observations. The paper measures top-candidate fractions of about 7, 14, and 22 percent for the frequency, similarity, and combined rankings versus below 3 percent for random selection, and reports that roughly 20,000 inspected candidates, including about 2,000 second-stage inspections, were all ruled out as radio-frequency interference or instrumental noise.
Load-bearing premise
The pipeline's accepted region is learned from simulated signals generated with the Setigen package against one S-band background, so the whole search assumes that real extraterrestrial narrow-band signals resemble that simulation family closely enough to land in the same UMAP cluster.
Editorial extensions
If this is right
- A cadence-only statistic set, not a signal-shape template, can cut SETI candidate lists by five orders of magnitude before human vetting.
- Ranking by both quietness in frequency and temporal self-similarity outperforms either score alone, so future searches should combine the two axes.
- Known promising candidates from a comparable deep-learning search mostly pass through the new filter and receive high scores, implying the two approaches select a consistent population.
- The two candidates lost by the cross-correlation filter had the highest drift rates and low signal-to-noise ratios, marking a boundary in drift sensitivity.
- No candidate survived second-stage inspection, so if the simulation family is representative, the Breakthrough Listen sample contains no narrow-band on-target-only signal of the modeled kinds.
Reading between the lines
- Because the frequency score discards the noisiest 5 percent of star-band combinations, a real strong signal in an RFI-dense band would be deliberately sacrificed; a pipeline variant that only down-ranks rather than discards those combinations could be tested against this sample.
- The similarity score's definition of 'persistent' is scale-bound at 220 Hz by 100 s, so a multi-scale version that zooms out automatically would likely catch drifting signals the current first filter misses, as the paper's two lost high-drift candidates hint.
- The validation strategy is relative (versus random selection), so the 22-percent top-candidate rate is not an absolute detection probability; comparing against a conventional TurboSETI-ranked sample would show whether the gain is unique to this pipeline.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript presents LE2.0, an anomaly-detection pipeline for technosignature searches in Breakthrough Listen radio observations. The pipeline reduces roughly 10^11 spectrograms to a few thousand human-vetted candidates through three stages: a cross-correlation/UMAP/KDE filter that selects cadences with signals appearing only in on-target pointings, with the accepted region calibrated by Setigen simulations; a frequency score based on GMM density estimation that up-weights candidates in RFI-quiet spectral regions; and a similarity score based on UMAP distances that rewards on-target spectrograms that cluster together and away from off-target pointings. The authors report that the frequency score, similarity score, and combined score produce top-candidate fractions of about 7%, 14%, and 22% during human vetting, versus less than 3% for random selection, and that no candidate survived detailed scrutiny. They also compare their candidate list with the deep-learning search of Ma et al. (2023).
Significance. If the validation claims survive scrutiny, the paper offers a useful triage method for SETI data and a substantial null result from a large Breakthrough Listen sample. Its strengths are the scale of the analysis (roughly 10^11 spectrograms), the explicit description of the human vetting stages, the use of simulations to calibrate the UMAP accepted region, and the direct comparison with an independent machine-learning search. The main caveats are that the quantitative validation is carried out inside the same simulation family used to define the accepted region, and that the headline improvement over random selection is reported without uncertainty estimates or significance tests. These issues are fixable with additional analysis rather than fundamental flaws.
major comments (3)
- [Section 4] The central claim that the filters "significantly improve" candidate quality is not quantified. State the exact number of candidates in each of the four sub-samples and provide binomial confidence intervals or a two-proportion test comparing 7%, 14%, and 22% against the control value of less than 3%. Without these, the reader cannot tell whether the differences are within sampling noise, and the word "significant" is not supported. Also specify whether the same candidates can appear in more than one sub-sample, since the three scored samples may not be independent.
- [Sections 3.1 and 4] The accepted "only on-target" region is defined and validated with Setigen simulations that use amplitudes uniform in [0, 4 times the noise maximum], all available Setigen time/frequency profiles, and signals present in every A panel of the cadence. Because the validation in Section 4 uses the same simulation family, the reported top-candidate fractions are in-family recovery rates. Add out-of-family injections into real data—for example signals in only one or two A panels, amplitudes below the simulated range, higher drift rates, and morphologies not available in Setigen—and measure recovery rates after each filter. This is needed to support both the triage-quality claim and the null conclusion. The self-identified caveat in Section 4 that "any simulation will be obviously biased by the populations we choose to simulate" is appropriate but does not replace such tests.
- [Section 3.1] The probability threshold of 0.0618 for the "only on-target" category is described as somewhat arbitrary and highly dependent on the KDE bandwidth, and the statements that the UMAP hyper-parameter choices do not affect the results are not demonstrated. Provide a sensitivity analysis showing how the threshold and hyper-parameters change the number of candidates surviving the first filter, the 7%/14%/22% validation fractions, and the final null result. Alternatively, define the threshold from a pre-specified acceptable contamination rate rather than by eye from Figure 4.
minor comments (5)
- [Section 2] The scikit-learn implementation is cited as "SCIKIT-LEARN (de Boor 2001)"; the correct citation for the software is Pedregosa et al. (2011), with de Boor (2001) cited only for the underlying B-spline mathematics.
- [Section 2] The relation between "about 10^7 of these per star-band combination" and "~10^11 spectrograms in total" is not derived. State the number of star-band combinations and clarify whether the counts refer to 80x16 spectrogram images or to full six-panel cadences.
- [Section 4] "These number were selected" should read "These numbers were selected," and the term "top-candidate" should be defined consistently as a candidate that passes the first vetting stage; the current text uses it before that definition is fully established.
- [Section 3.3] Equation (1) assumes a complete six-panel cadence. State what is done if a pointing is missing or flagged as unusable, since the formulas and the UMAP embedding may otherwise receive incomplete inputs.
- [Figure 4] The histogram on the y-axis is labeled "Counts" but the sample size, bin width, and the number of real targets used to build it are not given; these details would make the threshold selection more reproducible.
Circularity Check
Similarity-score validation is self-definitional: the score encodes the same on/off pattern used to define 'top candidates', so part of the claimed improvement reduces to the construct itself; the frequency-score result and the null result remain independent.
-
self definitional
[Section 3.3, Equation (1), and Section 4 (human vetting definition)]
"We expect a real ET signal to be similar to itself in all the 'A' observations, and be dissimilar from the off-target observations. ... A good candidate will have all the 'on-target' observations similar to each other, and therefore they will be close to each other in the UMAP space, relative to the distances to the 'off-target'. This ratio is the similarity score for a candidate"
The validation label 'top-candidate' is defined in Section 4 by exactly the same on/off criterion that the similarity score is built to measure: cadences are discarded when they show 'signals in more than just the on-target panels', and surviving cadences are 'top-candidates'. Equation (1) is a smooth surrogate for that same only-on-target pattern. Therefore the reported 14% top-candidate fraction for the similarity-only sample is an internal consistency check — how well the score recovers its own defining pattern — not an independent validation that the score improves candidate quality. The frequency score (7%) is an independent axis based on RFI density, so the circularity is partial; the combined 22% score inherits the self-definitional component.
full rationale
The paper's central validation claim is that its filters improve the quality of candidates sent to human vetting relative to random selection, with quality measured by the fraction of 'top-candidates'. The similarity score is explicitly constructed from the same on-target/off-target separation that defines a top-candidate, so the similarity-score component of the validation is partly circular: it demonstrates that the score agrees with the human application of the same heuristic, not that the heuristic has external predictive value. The frequency-score result is independent, since it ranks by quietness of the frequency window rather than by the on/off pattern, and the paper's null result is an honest non-detection within the pipeline's accepted region. I weighed the paper's own limitation note — 'Any simulation will be obviously biased by the populations we choose to simulate' — as a coverage caveat about the Setigen-based first filter, not as a circular step, because no equation or fitted parameter reduces the null result to the simulation input. There is no load-bearing self-citation chain: citations to prior work by authors are contextual examples, and the Setigen package is public tooling, not an imported uniqueness theorem. Overall, one prediction metric reduces by construction to its target definition, giving partial circularity rather than complete equivalence, hence a score of 6.
Assumptions & free parameters
free parameters (6)
- Only on-target probability threshold =
0.0618
- UMAP hyperparameters =
n_neighbors=25, min_dist=0.01; Canberra metric in first UMAP, Euclidean in second
- KDE bandwidth =
0.3 (exponential kernel)
- GMM components per band =
200 (L), 200 (S), 50 (C)
- Star-band background cutoff =
Candidates from the 5% of star-band combinations with the most candidates are discarded (65% of candidates)
- Simulation amplitude range =
0 to 4 times the maximum noise in the frequency window
assumptions (5)
- domain assumption A real technosignature is narrow-band (about Hz) and appears in on-target A observations but not in off-target BCD observations (Sections 1 and 3.1).
- ad hoc to paper Setigen simulations with the chosen amplitude range and the HIP 17147 S-band background are representative of possible ET signals (Section 3.1).
- standard math UMAP embedding preserves enough local structure that clusters in the 2D plane correspond to meaningful signal categories (Sections 3.1 and 3.3).
- domain assumption Human visual inspection is a reliable gold standard for rejecting RFI and identifying top candidates (Section 4).
- domain assumption RFI is the main contaminant and quieter frequency regions are less likely to contain it (Section 3.2).
Cite this review
Pith. "Pith review of Using anomaly detection to search for technosignatures in Breakthrough Listen observations." pith.science (2026). https://pith.science/paper/X23V34O2
@misc{pith2026250503927,
author = {Pith},
title = {Pith review of: Using anomaly detection to search for technosignatures in Breakthrough Listen observations},
year = {2026},
howpublished = {\url{https://pith.science/paper/X23V34O2}},
note = {Machine review of arXiv:2505.03927}
}
read the original abstract
We implement a machine learning algorithm to search for extra-terrestrial technosignatures in radio observations of several hundred nearby stars, obtained with the Parkes and Green Bank Telescopes by the Breakthrough Listen collaboration. Advances in detection technology have led to an exponential growth in data, necessitating innovative and efficient analysis methods. This problem is exacerbated by the large variety of possible forms an extraterrestrial signal might take, and the size of the multidimensional parameter space that must be searched. It is then made markedly worse by the fact that our best guess at the properties of such a signal is that it might resemble the signals emitted by human technology and communications, the main (yet diverse) contaminant in radio observations. We address this challenge by using a combination of simulations and machine learning methods for anomaly detection. We rank candidates by how unusual they are in frequency, and how persistent they are in time, by measuring the similarity between consecutive spectrograms of the same star. We validate that our filters significantly improve the quality of the candidates that are selected for human vetting when compared to a random selection. Of the ~ 10^11 spectrograms that we analyzed, we visually inspected thousands of the most promising spectrograms, and thousands more for validation, about 20,000 in total, and report that no candidate survived basic scrutiny.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter doi edition editor eprint howpublished institution journal key month number organization pages publisher school series title misctitle type volume year version url label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts ...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION format.url url empty "" new.block "" url * "" * if FUNCTION format.eprint eprint empty "" archivePrefix empty "" archivePrefix "arXiv" = new.block " " eprint * " " * new.block " " eprint * " " * if if if FUNCTION format.doi doi empty "" " " doi * " " * if FUNCTION format.pid doi empty eprint empty ur...
-
[3]
,1B 7Jܰ| Au Myy9g?c1zo v& * 'Y 1ǣ _ @ @ r Vh 1 A L( P . P ח T o ]= a|zh @ q\3㇄ 8
thebibliography [1] 20pt to REFERENCES 6pt =0pt -12pt 10pt plus 3pt =0pt =0pt =1pt plus 1pt =0pt =0pt -12pt =13pt plus 1pt =20pt =13pt plus 1pt \@M =10000 =-1.0em =0pt =0pt 0pt =0pt =1.0em @enumiv\@empty 10000 10000 `\.\@m \@noitemerr \@latex@warning Empty `thebibliography' environment \@ifnextchar \@reference \@latexerr Missing key on reference command E...
-
[4]
2017, Monthly Notices, 465, 4530, 10.1093/mnras/stw3021
Baron, D., & Poznanski, D. 2017, Monthly Notices, 465, 4530, 10.1093/mnras/stw3021
-
[5]
Brzycki, B., Siemion, A. P., Croft, S., et al. 2020, Publications of the Astronomical Society of the Pacific, 132, 114501, 10.1088/1538-3873/ABAAF7
-
[6]
Brzycki, B., Siemion, A. P. V., Pater, I. D., et al. 2022, The Astronomical Journal, 164, 33, 10.3847/1538-3881/ac5e3d
-
[7]
2024, , 167, 10, 10.3847/1538-3881/acf576
Choza , C., Bautista , D., Croft , S., et al. 2024, , 167, 10, 10.3847/1538-3881/acf576
-
[8]
Cox, G. A., Egly, S., Harp, G. R., et al. 2018, Proceedings of the IEEE Aerospace Conference, 1, 10.48550/arxiv.1803.08624
Show all 40 references
-
[9]
2021, , 133, 064502, 10.1088/1538-3873/abf329
Czech , D., Isaacson , H., Pearce , L., et al. 2021, , 133, 064502, 10.1088/1538-3873/abf329
2021 doi
-
[10]
2001, A Practical Guide to Splines - Revised Edition, Vol
de Boor, C. 2001, A Practical Guide to Splines - Revised Edition, Vol. 115 (Springer), 366. https://link.springer.com/book/9780387953663
2001
-
[11]
Drake, F. D. 1961, Citation: Physics Today, 14, 48, 10.1063/1.3057500
1961 doi
-
[12]
2019, turboSETI: Python-based SETI search algorithm , Astrophysics Source Code Library, record ascl:1906.006
Enriquez , E., & Price , D. 2019, turboSETI: Python-based SETI search algorithm , Astrophysics Source Code Library, record ascl:1906.006
2019
-
[13]
E., Siemion, A., Foster, G., et al
Enriquez, J. E., Siemion, A., Foster, G., et al. 2017, The Astrophysical Journal, 849, 104, 10.3847/1538-4357/aa8d1b
2017 doi
-
[14]
Franz, N., Croft, S., Siemion, A. P. V., et al. 2022, The Astronomical Journal, 163, 104, 10.3847/1538-3881/ac46c9
2022 doi
-
[15]
I., Siemion, A
Gajjar, V., Perez, K. I., Siemion, A. P. V., et al. 2021, The Astronomical Journal, 162, 33, 10.3847/1538-3881/abfd36
2021 doi
- [16]
-
[17]
A., & Yawney, J
Hilal, W., Gadsden, S. A., & Yawney, J. 2022, Expert Systems with Applications, 193, 116429, https://doi.org/10.1016/j.eswa.2021.116429
2022
- [18]
-
[19]
Lebofsky, M., Croft, S., Siemion, A. P. V., et al. 2019, Publications of the Astronomical Society of the Pacific, 131, 10.1088/1538-3873/ab3e82
2019 doi
-
[20]
X., Croft, S., Lintott, C., & Siemion, A
Ma, P. X., Croft, S., Lintott, C., & Siemion, A. P. V. 2024, RAS Techniques and Instruments, 3, 33, 10.1093/rasti/rzad056
2024 doi
-
[21]
X., Ng, C., Rizk, L., et al
Ma, P. X., Ng, C., Rizk, L., et al. 2023, Nature Astronomy, 10.1038/s41550-022-01872-z
2023 doi
-
[22]
H., Price, D
Macmahon, D. H., Price, D. C., Lebofsky, M., et al. 2018, Publications of the Astronomical Society of the Pacific, 130, 044502, 10.1088/1538-3873/AA80D2
2018 doi
-
[23]
2020, The Astronomical Journal, 161, 55, 10.3847/1538-3881/abcc77
Margot, J.-L., Pinchuk, P., Geil, R., et al. 2020, The Astronomical Journal, 161, 55, 10.3847/1538-3881/abcc77
2020 doi
-
[24]
G., Pinchuk , P., et al
Margot , J.-L., Li , M. G., Pinchuk , P., et al. 2023, , 166, 206, 10.3847/1538-3881/acfda4
2023 doi
- [25]
-
[26]
A., Pota, M., & Esposito, M
Panza, M. A., Pota, M., & Esposito, M. 2023, Electronics, 12, 10.3390/electronics12183971
2023 doi
-
[27]
2011, Journal of Machine Learning Research, 12, 2825, 10.5555/1953048
Pedregosa, F., Michel, V., Grisel, O., et al. 2011, Journal of Machine Learning Research, 12, 2825, 10.5555/1953048
2011 doi
-
[28]
2022, The Astronomical Journal, 163, 76, 10.3847/1538-3881/AC426F
Pinchuk, P., & Margot, J.-L. 2022, The Astronomical Journal, 163, 76, 10.3847/1538-3881/AC426F
2022 doi
-
[29]
H., et al
Pinchuk, P., Margot, J.-L., Greenberg, A. H., et al. 2019, The Astronomical Journal, 157, 122, 10.3847/1538-3881/AB0105
2019 doi
-
[30]
C., Enriquez, J
Price, D. C., Enriquez, J. E., Brzycki, B., et al. 2019, The Astronomical Journal, 159, 86, 10.3847/1538-3881/ab65f1
2019 doi
-
[31]
2018, , 476, 2117, 10.1093/mnras/sty348
Reis , I., Poznanski , D., Baron , D., Zasowski , G., & Shahaf , S. 2018, , 476, 2117, 10.1093/mnras/sty348
2018 doi
-
[32]
X., & Wolf , L
Reis , I., Rotman , M., Poznanski , D., Prochaska , J. X., & Wolf , L. 2021, Astronomy and Computing, 34, 100437, 10.1016/j.ascom.2020.100437
2021
-
[33]
R., Vandermeulen, R
Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., et al. 2021, Proceedings of the IEEE, 109, 756–795, 10.1109/jproc.2021.3052449
2021
-
[34]
J., & Vanderburg , A
Shallue , C. J., & Vanderburg , A. 2018, , 155, 94, 10.3847/1538-3881/aa9e09
2018 doi
-
[35]
P., Demorest, P., Korpela, E., et al
Siemion, A. P., Demorest, P., Korpela, E., et al. 2013, Astrophysical Journal, 767, 94, 10.1088/0004-637X/767/1/94
2013 doi
-
[36]
Siemion, A. P. V., Bower, G. C., Foster, G., et al. 2011, Astrophysical Journal, 744, 10.1088/0004-637X/744/2/109
2011 doi
-
[37]
E., & Gadermayr, M
Tschuchnig, M. E., & Gadermayr, M. 2022, Anomaly Detection in Medical Imaging - A Mini Review (Springer Fachmedien Wiesbaden), 33–38, 10.1007/978-3-658-36295-9_5
2022 doi
-
[38]
2008, Journal of Machine Learning Research, 9, 2579
van der Maaten, L., & Hinton, G. 2008, Journal of Machine Learning Research, 9, 2579. https://www.jmlr.org/papers/v9/vandermaaten08a.html
2008
-
[39]
A., Cranmer, M., Berger, E., et al
Villar, V. A., Cranmer, M., Berger, E., et al. 2021, The Astrophysical Journal Supplement Series, 255, 24, 10.3847/1538-4365/ac0893
2021 doi
-
[40]
G., Won, K
Zhang, Y. G., Won, K. H., Son, S. W., Siemion, A., & Croft, S. 2019, 2018 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2018 - Proceedings, 1114, 10.1109/GLOBALSIP.2018.8646437
2019
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.