Pith. sign in

REVIEW 4 major objections 6 minor 19 references

Anomaly detection in radio galaxy data with trainable COSFIRE filters

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Trainable COSFIRE filters combined with an unsupervised local outlier detector can spot unusual radio galaxy morphologies in the FRGADB benchmark, reaching a geometric-mean score of about 0.79 that edges out a deep autoencoder's 0.77…

desk verdict Lightweight COSFIRE+LOF anomaly detection on radio galaxies is a real but modest contribution; the headline superiority claim collapses under the authors' own leak-free protocol. read the letter →

arxiv 2505.18643 v1 pith:MCHWLONC submitted 2025-05-24 astro-ph.IM cs.AI

Steven Ndung'u , Trienko Grobler , Stefan J. Wijnholds , George Azzopardi This is my paper · ORCID
classification astro-ph.IMcs.AI
keywords radiogalaxiesanomalydetectionCOSFIREfilterslocaloutlierfactormorphologicaldescriptorsautoencoderFRGADBbenchmarkrotationinvariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that trainable COSFIRE filters—automatically configured descriptors of the spatial arrangement of radio emission blobs—can replace heavy deep autoencoders for anomaly detection in radio galaxy surveys. On the FRGADB benchmark, the COSFIRE plus Local Outlier Factor (LOF) pipeline achieves an average geometric-mean score of about 0.79, above the 0.77 reported for convolutional autoencoders and the 0.60 for PCA. The method is semi-supervised: it learns what normal FRI and FRII galaxies look like and flags anything far from that in descriptor space, without ever seeing an anomalous example during training. The motivation is the coming flood of data from telescopes like SKA, where fast, explainable screening could surface rare or unknown morphologies.

What carries the argument

The machinery is the trainable COSFIRE filter, short for Combination of Shifted Filter Responses. For a training image, keypoints are detected as local maxima of Difference-of-Gaussians responses along concentric circles around the source center; each keypoint is encoded by its distance, angle, scale, and polarity, and a filter is built from these tuples. The filter response at a new image is the geometric mean of blurred, shifted DoG responses, and rotation invariance comes from taking the maximum over twelve rotated filter copies. The final descriptor is the L2-normalized vector of maximum responses from many such filters (90 per galaxy class in the main experiment), and anomalies are scored by the Local Outlier Factor, which measures how much a point's local density deviates from its neighbors in this descriptor space.

What would settle it

Re-run the COSFIRE+LOF pipeline on the FRGADB test set using hyperparameters chosen exclusively with the synthetic validation set (no real anomalies seen during tuning), repeated over many random training/validation splits, and compare the G-Mean distribution to the autoencoder's 0.77; if the distributions overlap substantially or the autoencoder wins, the claimed superiority is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a COSFIRE-based feature representation, followed by unsupervised Local Outlier Factor scoring, outperforms the published deep-learning and PCA baselines on the FIRST Radio Galaxy Anomaly Detection Benchmark (FRGADB). On the test set the authors report G-Mean of $0.79 \pm 0.01$ with recall $0.89$ and specificity $0.70$, versus $0.77 \pm 0.02$ for the best autoencoder variants and $0.60 \pm 0.05$ for PCA+LOF. The paper also runs a leakage-aware variant in which hyperparameters are tuned on a synthetic validation set built from rotated and superimposed typical sources, yielding $0.78 \pm 0.03$, which still sits near the autoencoder level. The authors argue the approach is efficient, rotation-invariant, and requires no anomalous training examples, making it suitable for discovering genuinely new radio source types.

Load-bearing premise

The headline 0.79 result assumes that tuning hyperparameters on a validation set that already contains real examples of the anomalous classes (XRG and RRG) does not inflate the test score; the paper's own leakage-free variant, tuned on synthetic anomalies, scores 0.78, statistically overlapping with the autoencoder baseline.

Editorial extensions

If this is right

  • Radio survey pipelines could flag unusual galaxy morphologies without training deep neural networks, using simple convolutions and a nearest-neighbor density estimate.
  • Because COSFIRE descriptors are rotation-invariant, galaxies with identical intrinsic shapes will not be flagged as anomalies just because they appear at different orientations on the sky.
  • The method can in principle detect morphologies never seen during training, which is what matters for serendipitous discovery of new radio source classes.
  • The reported 0.78 G-Mean under the leakage-free protocol suggests the approach stays competitive with autoencoders even when real anomalies are kept out of validation.
  • Separating feature extraction (COSFIRE) from anomaly scoring (LOF) allows either stage to be swapped out, e.g., replacing LOF with another density estimator or adding new filter classes as surveys grow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The small gap between the headline 0.79 and the leakage-free 0.78, both within error of the autoencoder's 0.77, suggests the practical advantage over deep autoencoders may be modest; an independent, larger benchmark is needed before concluding that COSFIRE is the better detector.
  • The synthetic anomaly recipe—rotating and superimposing typical sources—could be standardized as a way to build leakage-free validation sets for other anomaly detection tasks in astronomy, where rare classes are hard to label.
  • Because the descriptors are cheap and explainable, they could be combined with more powerful downstream models, such as one-class SVMs or deep density estimators, potentially improving robustness on noisy or multi-scale images.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript proposes an anomaly-detection pipeline for radio galaxy images that combines trainable COSFIRE filters, which produce rotation-invariant morphological descriptors, with the Local Outlier Factor (LOF) algorithm for unsupervised anomaly scoring. The method is evaluated on the FRGADB benchmark, and the authors report a G-Mean of about 0.79, which they claim surpasses deep-learning autoencoder baselines (about 0.77) from prior work. They also describe an alternative validation protocol in which real anomalous classes are removed from the validation set and replaced by synthetic anomalies to avoid leakage; that protocol yields 0.78 ± 0.03. The manuscript includes links to code and data and positions the approach as a lightweight, interpretable alternative to deep autoencoders.

Significance. If the headline result were robust, COSFIRE + LOF would be a useful, interpretable, and computationally light baseline for anomaly detection in radio surveys. The paper has clear strengths: the code and data are made publicly available, the method is compared against several baselines, and the authors explicitly attempt to address validation leakage by constructing a synthetic validation set. However, the claimed advantage over deep autoencoders is not statistically established. The leak-free protocol gives a G-Mean of 0.78 ± 0.03, which overlaps with the reported 0.77 values for SCAE and MemSCAE, and the paper provides no significance test separating these numbers. As written, the contribution is therefore better characterized as 'competitive with deep autoencoders at lower computational cost' than as 'surpassing' them.

major comments (4)
  1. [Sections 2.1, 3.1, 3.2; Table 3] The central superiority claim rests on a validation procedure that the authors themselves identify as leaky. Section 2.1 states that the original validation set contains real XRG and RRG anomalies and that this 'introduces potential data leakage ... can lead to an overestimation of its true generalization performance.' Yet the headline result in Table 3 (COSFIRE = 0.79 ± 0.01) is obtained with hyperparameters selected on that original validation set. The leak-free alternative, COSFIRE* tuned on the synthetic validation set, gives 0.78 ± 0.03, which overlaps within uncertainty with SCAE (0.77 ± 0.01) and MemSCAE (0.77 ± 0.02). Because no significance test is reported to separate these values, the Abstract's claim of surpasssing the 77% deep autoencoder, and the similar claims in Sections 3.2 and 4, are not supported by the evidence. The leak-free protocol should be the primary result, and the claim should be scaled accordingly.
  2. [Section 3.2, Table 3] No statistical comparison is made between the COSFIRE results and the baseline results. The baseline G-Mean values are taken from a separate study (reference [3]) with their own standard deviations, but the paper does not report a test (e.g., a paired comparison on the same test folds, or an appropriate re-sampling test) that would establish that 0.78–0.79 is meaningfully better than 0.77. Given the small test set (110 images, with only 22 XRG and 13 RRG anomalies), the reported differences are within the noise of the evaluation. The authors should either provide such a test or explicitly state that the method is statistically indistinguishable from the autoencoder baselines under the leak-free protocol.
  3. [Section 2.1] The synthetic validation set is a load-bearing element of the leak-free evaluation, but the manuscript gives no evidence that rotating and superimposing normal FRI/FRII images produces anomalies whose difficulty is representative of real XRG/RRG anomalies. The paper cites reference [11] for 'natural synthetic anomalies,' but the construction described in Section 2.1 is substantially simpler and is not validated against the real test anomalies. At minimum, the authors should report both the leaky and leak-free protocols transparently, as they already do in Table 3, and discuss the proxy assumption as an explicit limitation; ideally they would provide a sensitivity analysis showing that ranking configurations on synthetic anomalies correlates with ranking on held-out real anomalies.
  4. [Section 3.1] The model-selection procedure is ambiguous about how many times the test set is used. The text says that for each of the two selected COSFIRE descriptors, 'the top 10 performing LOF hyperparameter sets were selected and evaluated on the test set,' and Table 3 reports a single G-Mean with a small standard deviation. If the reported number is the mean over those 10 configurations, the variance is over LOF hyperparameter sets, not independent data splits, and the standard deviation is not a measure of generalization error. If instead the best configuration on the test set was selected, the reported performance is optimistic. The authors should clarify the exact selection and reporting protocol and, if possible, use nested cross-validation or repeated splits to provide a meaningful performance estimate.
minor comments (6)
  1. [Abstract and Section 4] The claims 'achieving a G-Mean score of 79%, surpassing the 77% achieved by a deep learning autoencoder' should be revised to reflect the leak-free result and the absence of statistical significance; as written, the abstract overstates the finding.
  2. [Table 3] The footnote for COSFIRE* is only defined at the table level; consider also defining it in Section 2.1 or Section 3.2, and explicitly stating that COSFIRE (without asterisk) uses the original validation set containing real anomalies.
  3. [Section 2.3] There is a typo in 'non-anamolous'; it should be 'non-anomalous.'
  4. [Section 3.2] The phrase 'surpassed these benchmarks' is used before the caveat about the synthetic validation set appears; reorder or qualify the sentence so the reader is not left with the impression that the leaky result is the final word.
  5. [Section 2.1 and Reference [11]] The connection between the proposed synthetic generation procedure and reference [11] is not clear; the paper should either explain how the cited method informs the current procedure, or cite a more directly relevant source for rotation/superimposition anomalies.
  6. [Figure 2] The figure caption could state more explicitly which steps are performed once during training and which are applied to each test image; the current description is adequate but the inference path would be clearer with an explicit 'test image' arrow.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the COSFIRE+LOF benchmark comparison is empirical and self-contained; the validation-set leakage concern is a correctness risk, not a circular derivation.

full rationale

The paper's central claim is an empirical benchmark result: COSFIRE descriptors plus LOF achieve G-Mean 0.79 on the FRGADB test set versus 0.77 for deep autoencoders. No derivation is offered that would make the output equivalent to the input by construction. The COSFIRE descriptors are configured from normal FRI/FRII training images, and LOF is an unsupervised outlier method; the test evaluation is a separate measurement on held-out data. The main methodological concern is that hyperparameters were tuned on the original validation set that contains real XRG/RRG anomalies, and the paper itself cautions that this 'introduces potential data leakage... can lead to an overestimation of its true generalization performance.' When tuning is instead done on synthetic anomalies, the COSFIRE* G-Mean is 0.78±0.03, which overlaps with the 0.77 baselines. This is a legitimate weakness in the strength of the superiority claim, but it is not circularity: tuning hyperparameters on a validation set and then reporting test performance is standard practice, and no fitted parameter is renamed as a prediction. The self-citations to prior COSFIRE work [5,6] justify the descriptor framework and hyperparameter-space reduction, but they are not the load-bearing evidence for the reported test score; the baseline numbers from [3] are external published measurements, even though one author overlaps. Thus the paper contains no circular step that reduces the claimed result to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities, particles, forces, or conserved quantities are introduced; COSFIRE filters are existing constructs from prior work. The free parameters are all hyperparameters tuned on validation data, and the performance claim depends on those choices.

free parameters (4)
  • COSFIRE hyperparameters (sigma, P, t1, sigma0', alpha) = sigma=6, P={0,5,...,30}, sigma0'=0.75, alpha=0.15, t1 in {0.10,0.05}
    Tuned via grid search on the validation set; the two retained descriptors share these values.
  • Number of COSFIRE filters per class = 90 (67 for synthetic-validation variant)
    Optimal G-Mean on the validation set; descriptor dimensionality scales as 2n.
  • LOF contamination = from {0.37,0.38,0.39}
    Tuned on validation; sets the expected outlier fraction, directly controlling the anomaly threshold.
  • LOF n_neighbors, leaf_size, metric = n_neighbors from {8,9,10}, leaf size from {40,60,80}, metric from {Cosine, Manhattan, Minkowski}
    Grid-searched; top 10 validation configurations averaged on test.
assumptions (4)
  • domain assumption FRGADB class labels and train/validation/test splits are correct and representative.
    The benchmark [9,10] is taken as ground truth; no independent labeling check.
  • domain assumption COSFIRE descriptor responses capture the morphological information needed to separate normal from anomalous radio galaxies.
    Inherited from prior COSFIRE classification/retrieval work [5,6]; not re-derived here.
  • ad hoc to paper Synthetic anomalies created by rotating/superimposing FRI and FRII images are a valid proxy for real XRG/RRG anomalies during validation.
    Introduced in Section 2.1 to avoid leakage; no evidence they match the distribution of real anomalies.
  • domain assumption Local Outlier Factor density model is appropriate for the COSFIRE descriptor space.
    Standard algorithm choice; no justification specific to this feature distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Anomaly detection in radio galaxy data with trainable COSFIRE filters." pith.science (2026). https://pith.science/paper/MCHWLONC

@misc{pith2026250518643,
  author       = {Pith},
  title        = {Pith review of: Anomaly detection in radio galaxy data with trainable COSFIRE filters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MCHWLONC}},
  note         = {Machine review of arXiv:2505.18643}
}
read the original abstract

Detecting anomalies in radio astronomy is challenging due to the vast amounts of data and the rarity of labeled anomalous examples. Addressing this challenge requires efficient methods capable of identifying unusual radio galaxy morphologies without relying on extensive supervision. This work introduces an innovative approach to anomaly detection based on morphological characteristics of the radio sources using trainable COSFIRE (Combination of Shifted Filter Responses) filters as an efficient alternative to complex deep learning methods. The framework integrates COSFIRE descriptors with an unsupervised Local Outlier Factor (LOF) algorithm to identify unusual radio galaxy morphologies. Evaluations on a radio galaxy benchmark data set demonstrate strong performance, with the COSFIRE-based approach achieving a geometric mean (G-Mean) score of 79%, surpassing the 77% achieved by a computationally intensive deep learning autoencoder. By characterizing normal patterns and detecting deviations, this semi-supervised methodology overcomes the need for anomalous examples in the training set, a major limitation of traditional supervised methods. This approach shows promise for next-generation radio telescopes, where fast processing and the ability to discover unknown phenomena are crucial.

Figures

Figures reproduced from arXiv: 2505.18643 by the authors.

Figure 1
Figure 1. Examples of typical radio galaxies and both [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A schematic illustration of the proposed COSFIRE framework for detecting anomalous radio galaxies, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

  1. [3]

    ApplyCOSFIREfilters

    Experimental Results 3.1. Performance To evaluate the performance of our COSFIRE fil- ter approach in anomaly detection, we adopted an ex- perimental design closely aligned with the methodolo- gies presented in [5] and [6]. Training data underwent a stratified hold-out split to ensure representative sampling across classes. This means the dataset was rand...

    work page

  2. [11]

    The morphology of extra- galacticradiosourcesofhighandlowluminosity,

    B. L. Fanaroff and J. M. Riley, “The morphology of extra- galacticradiosourcesofhighandlowluminosity,” MNRAS, 167, 1, 1974, pp. 31P-36P

    work page 1974

  3. [1]

    While processing such vast data sets poses significant challenges, it also presents unique opportuni- ties for serendipitous discoveries of novel astrophysical phenomena

    Introduction The advent of next-generation radio telescopes has ushered in an era of unprecedented data volumes in ra- dio astronomy, with facilities like the Square Kilometre Array (SKA) expected to generate exabytes of data an- nually [1]. While processing such vast data sets poses significant challenges, it also presents unique opportuni- ties for sere...

    work page

  4. [2]

    Anomaly detection in radio galaxy data with trainable COSFIRE filters

    Data and Methodology 2.1. Data set We evaluate our anomaly detection approach using theFIRST 1RadioGalaxyAnomalyDetectionBenchmark (FRGADB)data[9,10]. FRGADBiscomposedof ∼1000 samples with four radio galaxy classes: FRI, FRII, XRG, 1Faint Images of the Radio-Sky at Twenty centimeters arXiv:2505.18643v1 [astro-ph.IM] 24 May 2025 Table 1: Distribution of th...

    work page Pith review arXiv 2025

  5. [4]

    Discussion and Conclusion TheCOSFIRE-basedmethodologygeneratesrobust feature representations that effectively detect anomalies not present in the training data, while maintaining high performance despite the challenges of limited training data and class imbalance. This results in improved per- formance, with a G-Mean score of 79%, surpassing both the comp...

    work page

  6. [5]

    Sciencepipelinesforthesquarekilometrearray,

    J. Farnes, B. Mort, F. Dulwich, S. Salvini, and W. Armour, “Sciencepipelinesforthesquarekilometrearray,” Galaxies, 6, 4, 2018, pp. 120

    work page 2018

  7. [6]

    Prospecting for enigmatic radio sources with autoencoders: a novel approach,

    F. Ventura, “Prospecting for enigmatic radio sources with autoencoders: a novel approach,” Master’s thesis, Univer- sity of Pretoria, 2022

    work page 2022

  8. [7]

    CARA:convo- lutionalautoencodersforthedetectionofradioanomalies,

    K.Brand,T.L.Grobler,andW.Kleynhans,“CARA:convo- lutionalautoencodersforthedetectionofradioanomalies,” RASTI,4, 2025, pp. rzaf005

    work page 2025

Show all 19 references
  1. [8]

    Trainable COSFIRE filters for keypoint detection and pattern recognition,

    G. Azzopardi and N. Petkov, “Trainable COSFIRE filters for keypoint detection and pattern recognition,”PAMI, 35, 2, 2012, pp. 490-503

    2012

  2. [9]

    Classification of radio galaxies with trainableCOSFIREfilters,

    S.Ndung’u,T.Grobler,S.J.Wijnholds,D.Karastoyanova, and G. Azzopardi, “Classification of radio galaxies with trainableCOSFIREfilters,” MNRAS,530,1,2024,pp.783- 794

    2024

  3. [10]

    Content-BasedImageRetrievalUsingCOSFIREDescrip- tors with application to Radio Astronomy,

    S.Ndung’u,T.Grobler,S.J.Wijnholds,andG.Azzopardi, “Content-BasedImageRetrievalUsingCOSFIREDescrip- tors with application to Radio Astronomy,”MNRAS, 2025, pp. staf230

    2025

  4. [12]

    Morphologicalannotationsforgroupsinthe FIRST database,

    D.D.Proctor,“Morphologicalannotationsforgroupsinthe FIRST database,”ApJS,194, 2, 2011, pp. 31

    2011

  5. [13]

    The FIRST survey: faintimagesoftheradioskyattwentycentimeters,

    R. H. Becker, R. L. White, and D. J. Helfand, “The FIRST survey: faintimagesoftheradioskyattwentycentimeters,” AJ,450, 1995, pp. 559

    1995

  6. [14]

    FRGADB-FIRSTRadioGalaxyAnomalyDe- tection Benchmark,

    K.Brand, “FRGADB-FIRSTRadioGalaxyAnomalyDe- tection Benchmark,” Zenodo, version 1.0, 2024

    2024

  7. [15]

    Natural synthetic anomalies for self-supervised anomaly detection and localization,

    H. M. Schlüter, J. Tan, B. Hou, and B. Kainz, “Natural synthetic anomalies for self-supervised anomaly detection and localization,” ECCV, 2022, pp. 474-489

    2022

  8. [16]

    LOF:identifyingdensity-basedlocaloutliers,

    M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “LOF:identifyingdensity-basedlocaloutliers,”Proc.ACM SIGMOD Int. Conf. Manag. Data, 2000, pp. 93-104

    2000

  9. [17]

    An introduction to inferential statistics: A review and practical guide,

    G. Marshall and L. Jonker, “An introduction to inferential statistics: A review and practical guide,”Radiography, 17, 1, 2011, pp. e1-e6

    2011

  10. [18]

    Memorizing normal- ity to detect anomaly: Memory-augmented deep autoen- coder for unsupervised anomaly detection,

    D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh, and A. van den Hengel, “Memorizing normal- ity to detect anomaly: Memory-augmented deep autoen- coder for unsupervised anomaly detection,” ICCV, 2019, pp. 1705-1714

    2019

  11. [19]

    Detecting anomalous imagesinastronomicaldatasets,

    P. Alonso, J. Zhang, and X.-D. Li, “Detecting anomalous imagesinastronomicaldatasets,” AJ,167,3,2024,pp.129. University of Stellenbosch, Computer Science, Cnr Banhoek Road & Joubert Street, Stellenbosch 7600, South Africa; email:26846578@sun.ac.za. Steven Ndung’u; Universityof...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.