{"id":"09640f96-9728-4f93-9622-85e2c86a073d","arxiv_id":"2505.18643","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"COSFIRE descriptors plus Local Outlier Factor score 79% G-Mean on the FRGADB radio galaxy anomaly benchmark, marginally above the 77% of a deep autoencoder.","lead":"This paper combines COSFIRE image descriptors with a Local Outlier Factor detector to flag unusual radio galaxy shapes. It reports a 79% balanced accuracy score on a standard benchmark, two points above a deep learning autoencoder, but the margin is fragile.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 0.79 G-Mean is selected on a validation set containing real XRG/RRG anomalies; the leak-free tuning result is 0.78±0.03, overlapping the 0.77 autoencoder baseline, so the claimed superiority is unestablished.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: hyperparameter tuning on a validation set containing real XRG and RRG anomalies can inflate the test score, and the corrected synthetic-validation result overlaps the autoencoder baseline. My independent reading of the paper confirms this. The paper explicitly acknowledges the leakage risk in Section 2.1, yet the headline 0.79 comes from the leaky protocol, while the leak-free protocol yields 0.78±0.03. With no significance test or paired comparison, the difference between 0.78 and 0.77 is within reported uncertainty. I considered whether there are other internal inconsistencies, such as the selection of 90 filters per class on the same leaky validation set, but that is part of the same problem rather than a separate defect. The COSFIRE pipeline itself is clearly described, and prior work on COSFIRE descriptors provides support for the feature-extraction component. The central issue is therefore not the method's viability but the evidence for its headline advantage. The paper does provide code and data, which is a genuine strength, and a corrected protocol with proper significance testing could make the claim credible. For these reasons, I sustain the reader's REJECT verdict without change.","tokens_in":6974,"tokens_out":3052,"duration_ms":24300,"concrete_test":"Run the complete pipeline (COSFIRE filter configuration, number-of-filters selection, LOF grid search) using only the synthetic validation set for every model-selection decision. Then evaluate the selected model on the original 110-sample test set. Compute the paired difference in G-Mean between this leak-free COSFIRE model and SCAE/MemSCAE on the same test samples, with a 95% bootstrap or permutation confidence interval. If the interval includes 0, or if the leak-free model's G-Mean is not above 0.77, the claimed superiority over deep autoencoders is not established. The public GitHub repository should make this re-run straightforward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the model-selection protocol rather than the COSFIRE construction itself. In Section 2.1 the authors state that the original FRGADB validation split contains 22 XRG and 12 RRG real anomalies and that tuning on it 'introduces potential data leakage... can lead to an overestimation of its true generalization performance.' Yet the headline result in Table 3 (COSFIRE 0.79±0.01) is exactly the configuration selected on that leaky validation set. The authors' leak-avoiding alternative, COSFIRE* tuned on synthetic anomalies, gives 0.78±0.03 (Table 3), within error of SCAE 0.77±0.01 and MemSCAE 0.77±0.02. No significance test is reported anywhere that would separate 0.79/0.78 from 0.77. Since the only quantitative advantage over deep autoencoders rests on validation selection that is admitted to be leaky, and the corrected protocol removes that advantage, the central claim is not supported by the evidence in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes an anomaly-detection pipeline for radio galaxy images that combines trainable COSFIRE filters, which produce rotation-invariant morphological descriptors, with the Local Outlier Factor (LOF) algorithm for unsupervised anomaly scoring. The method is evaluated on the FRGADB benchmark, and the authors report a G-Mean of about 0.79, which they claim surpasses deep-learning autoencoder baselines (about 0.77) from prior work. They also describe an alternative validation protocol in which real anomalous classes are removed from the validation set and replaced by synthetic anomalies to avoid leakage; that protocol yields 0.78 ± 0.03. The manuscript includes links to code and data and positions the approach as a lightweight, interpretable alternative to deep autoencoders.","tokens_in":7154,"tokens_out":5722,"duration_ms":45467,"significance":"If the headline result were robust, COSFIRE + LOF would be a useful, interpretable, and computationally light baseline for anomaly detection in radio surveys. The paper has clear strengths: the code and data are made publicly available, the method is compared against several baselines, and the authors explicitly attempt to address validation leakage by constructing a synthetic validation set. However, the claimed advantage over deep autoencoders is not statistically established. The leak-free protocol gives a G-Mean of 0.78 ± 0.03, which overlaps with the reported 0.77 values for SCAE and MemSCAE, and the paper provides no significance test separating these numbers. As written, the contribution is therefore better characterized as 'competitive with deep autoencoders at lower computational cost' than as 'surpassing' them.","major_comments":[{"comment":"The central superiority claim rests on a validation procedure that the authors themselves identify as leaky. Section 2.1 states that the original validation set contains real XRG and RRG anomalies and that this 'introduces potential data leakage ... can lead to an overestimation of its true generalization performance.' Yet the headline result in Table 3 (COSFIRE = 0.79 ± 0.01) is obtained with hyperparameters selected on that original validation set. The leak-free alternative, COSFIRE* tuned on the synthetic validation set, gives 0.78 ± 0.03, which overlaps within uncertainty with SCAE (0.77 ± 0.01) and MemSCAE (0.77 ± 0.02). Because no significance test is reported to separate these values, the Abstract's claim of surpasssing the 77% deep autoencoder, and the similar claims in Sections 3.2 and 4, are not supported by the evidence. The leak-free protocol should be the primary result, and the claim should be scaled accordingly.","section":"Sections 2.1, 3.1, 3.2; Table 3"},{"comment":"No statistical comparison is made between the COSFIRE results and the baseline results. The baseline G-Mean values are taken from a separate study (reference [3]) with their own standard deviations, but the paper does not report a test (e.g., a paired comparison on the same test folds, or an appropriate re-sampling test) that would establish that 0.78–0.79 is meaningfully better than 0.77. Given the small test set (110 images, with only 22 XRG and 13 RRG anomalies), the reported differences are within the noise of the evaluation. The authors should either provide such a test or explicitly state that the method is statistically indistinguishable from the autoencoder baselines under the leak-free protocol.","section":"Section 3.2, Table 3"},{"comment":"The synthetic validation set is a load-bearing element of the leak-free evaluation, but the manuscript gives no evidence that rotating and superimposing normal FRI/FRII images produces anomalies whose difficulty is representative of real XRG/RRG anomalies. The paper cites reference [11] for 'natural synthetic anomalies,' but the construction described in Section 2.1 is substantially simpler and is not validated against the real test anomalies. At minimum, the authors should report both the leaky and leak-free protocols transparently, as they already do in Table 3, and discuss the proxy assumption as an explicit limitation; ideally they would provide a sensitivity analysis showing that ranking configurations on synthetic anomalies correlates with ranking on held-out real anomalies.","section":"Section 2.1"},{"comment":"The model-selection procedure is ambiguous about how many times the test set is used. The text says that for each of the two selected COSFIRE descriptors, 'the top 10 performing LOF hyperparameter sets were selected and evaluated on the test set,' and Table 3 reports a single G-Mean with a small standard deviation. If the reported number is the mean over those 10 configurations, the variance is over LOF hyperparameter sets, not independent data splits, and the standard deviation is not a measure of generalization error. If instead the best configuration on the test set was selected, the reported performance is optimistic. The authors should clarify the exact selection and reporting protocol and, if possible, use nested cross-validation or repeated splits to provide a meaningful performance estimate.","section":"Section 3.1"}],"minor_comments":[{"comment":"The claims 'achieving a G-Mean score of 79%, surpassing the 77% achieved by a deep learning autoencoder' should be revised to reflect the leak-free result and the absence of statistical significance; as written, the abstract overstates the finding.","section":"Abstract and Section 4"},{"comment":"The footnote for COSFIRE* is only defined at the table level; consider also defining it in Section 2.1 or Section 3.2, and explicitly stating that COSFIRE (without asterisk) uses the original validation set containing real anomalies.","section":"Table 3"},{"comment":"There is a typo in 'non-anamolous'; it should be 'non-anomalous.'","section":"Section 2.3"},{"comment":"The phrase 'surpassed these benchmarks' is used before the caveat about the synthetic validation set appears; reorder or qualify the sentence so the reader is not left with the impression that the leaky result is the final word.","section":"Section 3.2"},{"comment":"The connection between the proposed synthetic generation procedure and reference [11] is not clear; the paper should either explain how the cited method informs the current procedure, or cite a more directly relevant source for rotation/superimposition anomalies.","section":"Section 2.1 and Reference [11]"},{"comment":"The figure caption could state more explicitly which steps are performed once during training and which are applied to each test image; the current description is adequate but the inference path would be clearer with an explicit 'test image' arrow.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern in the reader's report is valid and is the decisive issue. I recommend major revision rather than rejection because the paper already contains a leak-free protocol (COSFIRE*) that can be made the primary evaluation, and the authors can reframe the contribution around efficiency and interpretability. However, the editor should be aware that after reframing, the claimed advantage over deep autoencoders may disappear; the paper's value would then rest on being a lightweight, explainable alternative that is competitive, not superior. The code and data availability are assets that make the required reruns feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine new application of COSFIRE descriptors to radio galaxy anomaly detection, and the authors are honest about the data leakage problem. But the headline result—0.79 G-Mean beating a 0.77 autoencoder—does not survive their own leak-free protocol, which gives 0.78±0.03, overlapping the baseline. So the central superiority claim is unestablished.\n\nWhat's new: the combination of trainable COSFIRE filters (rotation-invariant, interpretable morphological features) with LOF for anomaly detection on the FRGADB benchmark. The paper is clearly written, the method is refreshingly lightweight compared to deep autoencoders, and the authors explicitly identify the validation leakage in the standard FRGADB split and construct a synthetic-anomaly validation set to avoid it. That is the right instinct and it is commendable.\n\nThe soft spot is the way the headline is reported. In Section 2.1 they note that tuning on the original validation set (which contains 22 XRG and 12 RRG real anomalies) 'introduces potential data leakage' and can overestimate generalization. Yet the 0.79±0.01 in Table 3 is exactly the configuration selected on that leaky set. The leak-free alternative, COSFIRE* tuned on synthetic anomalies, gets 0.78±0.03—within error of SCAE and MemSCAE at 0.77. No significance test is reported that would separate 0.78/0.79 from 0.77, and the test set is only 110 images. The baselines come from a paper with overlapping authors, though the numbers appear independent. The corrected protocol should be the main result, not a robustness check.\n\nMinor: the COSFIRE hyperparameter grid is small, and the filter count (90 per class) is also selected on validation, but that is ordinary model selection, not a flaw.\n\nWho this is for: anyone doing anomaly detection in radio astronomy who wants a non-deep, interpretable baseline. It is a useful contribution even without a decisive win.\n\nI would send this to peer review. It deserves a serious referee. But the authors need to re-run with the leak-free validation as primary, add a significance test against the autoencoders, and fix the abstract and conclusion claims accordingly. As it stands, the paper overstates its case.","headline":"Lightweight COSFIRE+LOF anomaly detection on radio galaxies is a real but modest contribution; the headline superiority claim collapses under the authors' own leak-free protocol.","tokens_in":7727,"tokens_out":2453,"would_cite":false,"duration_ms":17224,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trainable COSFIRE filters combined with an unsupervised local outlier detector can spot unusual radio galaxy morphologies in the FRGADB benchmark, reaching a geometric-mean score of about 0.79 that edges out a deep autoencoder's 0.77…","keywords":["radio galaxies","anomaly detection","COSFIRE filters","local outlier factor","morphological descriptors","autoencoder","FRGADB benchmark","rotation invariance"],"falsifier":"Re-run the COSFIRE+LOF pipeline on the FRGADB test set using hyperparameters chosen exclusively with the synthetic validation set (no real anomalies seen during tuning), repeated over many random training/validation splits, and compare the G-Mean distribution to the autoencoder's 0.77; if the distributions overlap substantially or the autoencoder wins, the claimed superiority is not supported.","tokens_in":6688,"feed_emoji":"🔭","tokens_out":6643,"duration_ms":48476,"temperature":0.7,"pith_summary":"The paper sets out to show that trainable COSFIRE filters—automatically configured descriptors of the spatial arrangement of radio emission blobs—can replace heavy deep autoencoders for anomaly detection in radio galaxy surveys. On the FRGADB benchmark, the COSFIRE plus Local Outlier Factor (LOF) pipeline achieves an average geometric-mean score of about 0.79, above the 0.77 reported for convolutional autoencoders and the 0.60 for PCA. The method is semi-supervised: it learns what normal FRI and FRII galaxies look like and flags anything far from that in descriptor space, without ever seeing an anomalous example during training. The motivation is the coming flood of data from telescopes like SKA, where fast, explainable screening could surface rare or unknown morphologies.","feed_headline":"COSFIRE filters beat deep autoencoders at spotting odd radio galaxies","feed_subtitle":"Lightweight morphological descriptors reach 0.79 G-Mean on the FRGADB benchmark, rivaling deeper models.","key_machinery":"The machinery is the trainable COSFIRE filter, short for Combination of Shifted Filter Responses. For a training image, keypoints are detected as local maxima of Difference-of-Gaussians responses along concentric circles around the source center; each keypoint is encoded by its distance, angle, scale, and polarity, and a filter is built from these tuples. The filter response at a new image is the geometric mean of blurred, shifted DoG responses, and rotation invariance comes from taking the maximum over twelve rotated filter copies. The final descriptor is the L2-normalized vector of maximum responses from many such filters (90 per galaxy class in the main experiment), and anomalies are scored by the Local Outlier Factor, which measures how much a point's local density deviates from its neighbors in this descriptor space.","core_discovery":"The central claim is that a COSFIRE-based feature representation, followed by unsupervised Local Outlier Factor scoring, outperforms the published deep-learning and PCA baselines on the FIRST Radio Galaxy Anomaly Detection Benchmark (FRGADB). On the test set the authors report G-Mean of $0.79 \\pm 0.01$ with recall $0.89$ and specificity $0.70$, versus $0.77 \\pm 0.02$ for the best autoencoder variants and $0.60 \\pm 0.05$ for PCA+LOF. The paper also runs a leakage-aware variant in which hyperparameters are tuned on a synthetic validation set built from rotated and superimposed typical sources, yielding $0.78 \\pm 0.03$, which still sits near the autoencoder level. The authors argue the approach is efficient, rotation-invariant, and requires no anomalous training examples, making it suitable for discovering genuinely new radio source types.","pith_inferences":["The small gap between the headline 0.79 and the leakage-free 0.78, both within error of the autoencoder's 0.77, suggests the practical advantage over deep autoencoders may be modest; an independent, larger benchmark is needed before concluding that COSFIRE is the better detector.","The synthetic anomaly recipe—rotating and superimposing typical sources—could be standardized as a way to build leakage-free validation sets for other anomaly detection tasks in astronomy, where rare classes are hard to label.","Because the descriptors are cheap and explainable, they could be combined with more powerful downstream models, such as one-class SVMs or deep density estimators, potentially improving robustness on noisy or multi-scale images."],"forward_implications":["Radio survey pipelines could flag unusual galaxy morphologies without training deep neural networks, using simple convolutions and a nearest-neighbor density estimate.","Because COSFIRE descriptors are rotation-invariant, galaxies with identical intrinsic shapes will not be flagged as anomalies just because they appear at different orientations on the sky.","The method can in principle detect morphologies never seen during training, which is what matters for serendipitous discovery of new radio source classes.","The reported 0.78 G-Mean under the leakage-free protocol suggests the approach stays competitive with autoencoders even when real anomalies are kept out of validation.","Separating feature extraction (COSFIRE) from anomaly scoring (LOF) allows either stage to be swapped out, e.g., replacing LOF with another density estimator or adding new filter classes as surveys grow."],"supporting_citations":[{"why":"Defines trainable COSFIRE filters, the core feature detector used for anomaly detection.","marker":"[4]"},{"why":"Establishes the COSFIRE descriptor construction and L2 normalization for radio galaxy images, the representation the pipeline builds on.","marker":"[5]"},{"why":"Provides the deep autoencoder and PCA baselines and the evaluation protocol on the same FRGADB dataset that the paper compares against.","marker":"[3]"},{"why":"Introduces the FIRST survey, the source of the radio images in the FRGADB benchmark.","marker":"[9]"},{"why":"Defines the FRGADB dataset, its class distribution, and the training/validation/test splits used in the evaluation.","marker":"[10]"},{"why":"Supplies the method for generating synthetic anomalies used to build the leakage-free validation set.","marker":"[11]"},{"why":"Introduces the Local Outlier Factor algorithm used for unsupervised anomaly scoring on the COSFIRE descriptors.","marker":"[12]"}],"fun_headline_variants":["COSFIRE filters beat deep autoencoders on radio galaxy anomalies","Lightweight COSFIRE spots radio anomalies without labeled examples","Rotation-invariant COSFIRE filters find radio novelties without deep nets","Trainable COSFIRE filters rival deep nets on radio anomaly detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 0.79 result assumes that tuning hyperparameters on a validation set that already contains real examples of the anomalous classes (XRG and RRG) does not inflate the test score; the paper's own leakage-free variant, tuned on synthetic anomalies, scores 0.78, statistically overlapping with the autoencoder baseline.","fun_headline_variants_meta":{"raw":{"variants":["COSFIRE filters beat deep autoencoders on radio galaxy anomalies","Lightweight COSFIRE spots radio anomalies without labeled examples","Rotation-invariant COSFIRE filters find radio novelties without deep nets","Trainable COSFIRE filters rival deep nets on radio anomaly detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001075,"raw_usage":{"total_tokens":4493,"prompt_tokens":934,"completion_tokens":3559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":3484}},"tokens_in":550,"tokens_out":3559,"duration_ms":22492,"temperature":1.0,"reasoning_tokens":3484,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:27:51.071727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the COSFIRE+LOF pipeline on the FRGADB test set using hyperparameters chosen exclusively with the synthetic validation set (no real anomalies seen during tuning), repeated over many random training/validation splits, and compare the G-Mean distribution to the autoencoder's 0.77; if the distributions overlap substantially or the autoencoder wins, the claimed superiority is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines trainable COSFIRE filters, the core feature detector used for anomaly detection."},{"cited_title":"Sciencepipelinesforthesquarekilometrearray,","cited_arxiv_id":null,"evidence_quote":"Establishes the COSFIRE descriptor construction and L2 normalization for radio galaxy images, the representation the pipeline builds on."},{"cited_title":"ApplyCOSFIREfilters","cited_arxiv_id":null,"evidence_quote":"Provides the deep autoencoder and PCA baselines and the evaluation protocol on the same FRGADB dataset that the paper compares against."},{"cited_title":"Classification of radio galaxies with trainableCOSFIREfilters,","cited_arxiv_id":null,"evidence_quote":"Introduces the FIRST survey, the source of the radio images in the FRGADB benchmark."},{"cited_title":"Content-BasedImageRetrievalUsingCOSFIREDescrip- tors with application to Radio Astronomy,","cited_arxiv_id":null,"evidence_quote":"Defines the FRGADB dataset, its class distribution, and the training/validation/test splits used in the evaluation."},{"cited_title":"The morphology of extra- galacticradiosourcesofhighandlowluminosity,","cited_arxiv_id":null,"evidence_quote":"Supplies the method for generating synthetic anomalies used to build the leakage-free validation set."},{"cited_title":"Morphologicalannotationsforgroupsinthe FIRST database,","cited_arxiv_id":null,"evidence_quote":"Introduces the Local Outlier Factor algorithm used for unsupervised anomaly scoring on the COSFIRE descriptors."}],"review_version":1}