{"id":"561e93e4-0b21-4573-bcd2-e47470f8da5b","arxiv_id":"2506.15090","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning OpenCLIP on radio galaxy images and text descriptions enables a working image-and-text search engine over 170,000 EMU radio sources, though retrieval evaluation is qualitative.","lead":"This paper introduces EMUSE, a search engine that finds similar radio galaxies in the EMU survey using image or text queries. It fine-tunes the OpenCLIP vision-language model on 2,900 labeled radio galaxies and deploys it over roughly 170,000 sources.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval claim rests on four qualitative examples; all top-50 similarities are ≥0.89, suggesting a compressed embedding space, and no precision@k or unfine-tuned baseline is provided. A labeled retrieval benchmark is needed to validate cosine similarity as a morphological proxy.","rationale":"Good-faith reading: the authors build a usable tool, release code, report 84±3% classification accuracy from 10 repeated splits, and include an honest limitations section (Section 4.1 admits text-query sensitivity; Section 4.2 discusses screenshot/channel caveats; Data Availability makes exact training images available only on request). These are real positives. The load-bearing gap is that the central retrieval claim is not quantitatively tested. Classification accuracy on the fine-tuning set is not evidence that cosine similarity over 170k survey embeddings tracks morphology: the task is different, and the distribution shift from PS1 training cutouts to survey-wide cutouts is unquantified. The appendix tables show a striking compression of similarity scores (all top-50 entries ≥0.89, most ≥0.97 for each of four unrelated queries), which suggests the ranking is operating on tiny differences in a nearly saturated space—exactly the regime where spurious correlations (brightness, orientation, background) can dominate. The paper's own observations (orientation matching in the FR-II query, phrase-sensitivity in text queries, different result sets from different ORC screenshots) reinforce this concern. A labeled precision@k benchmark with an unfine-tuned baseline would settle the question; without it, CONDITIONAL remains the appropriate verdict.","tokens_in":23638,"tokens_out":8464,"duration_ms":90360,"concrete_test":"Build a labeled retrieval benchmark from the held-out test splits used for Figure 2 (or a fresh 20% split of RadioGalaxyNET): for each of, say, 200 test images, embed it with the fine-tuned model and retrieve top-10 from the 170k EMU embedding database; score each retrieved source by whether its RG-CAT/Gal-DINO category matches the query's known class. Report mean precision@10 and mean reciprocal rank against (i) random ranking and (ii) the unfine-tuned OpenCLIP baseline on the same queries. If precision@10 is not significantly above chance and above baseline, cosine similarity is not validated as a morphological similarity proxy on EMU data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that EMUSE retrieves morphologically similar radio sources via cosine similarity in the fine-tuned embedding space—is supported only by four selected queries. The classification accuracy (84±3%, Fig. 2) is measured on the fine-tuning dataset and does not transfer automatically to the 170k-source retrieval database. The appendix tables undercut the discriminative assumption: every top-50 result for every query, including the ORC image query, has similarity ≥0.89, and most are ≥0.97. If the embedding space were morphologically structured at the database scale, one would expect a wider dynamic range and query-dependent separation; instead the scores are compressed into a narrow band, so ranking may be dominated by source brightness, size, orientation, or image background rather than morphology. The paper itself notes that the FR-II query retrieved sources with matching orientation (Section 4.2), that text queries are sensitive to phrasing (Section 4.1), and that different screenshots of the same ORC yield different result sets. No precision@k, mean reciprocal rank, or baseline comparison (e.g., unfine-tuned OpenCLIP) is reported, so the reader cannot distinguish genuine morphological retrieval from nearest-neighbor artifacts in a compressed embedding space.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents EMUSE, a search engine for the Evolutionary Map of the Universe (EMU) radio survey. The authors fine-tune the OpenCLIP vision-language model on 2,900 radio galaxy images from the RadioGalaxyNET dataset, using adapter-based fine-tuning and expanded text descriptions, and report 84±3% classification accuracy on held-out test sets for the FR-I, FR-II, FR-x, R, and Peculiar categories. The fine-tuned model is used to embed approximately 170,000 extended radio sources from the EMU first-year data, together with infrared (AllWISE) cutouts and RG-CAT catalogue metadata. EMUSE supports zero-shot text and image queries by ranking database entries by cosine similarity to the query embedding. The paper demonstrates the tool with two text and two image queries, including an image query of an Odd Radio Circle that retrieves a known ORC candidate, and makes the search engine and code publicly available.","tokens_in":23921,"tokens_out":4552,"duration_ms":40708,"significance":"If the retrieval performance is validated quantitatively, EMUSE would be a valuable community tool for mining large radio surveys, with a sensible design: adapting a general foundation model to the radio domain via lightweight fine-tuning, and compressing survey images into a searchable embedding database. The authors provide open-source code, a public demo, and transparent reporting of the classification accuracy across multiple random splits. The paper also honestly reports limitations (rare-source retrieval, text-query sensitivity). What is missing is a rigorous evaluation of the retrieval claim, which is the central function of the tool.","major_comments":[{"comment":"The central retrieval claim is supported only by four qualitative query examples, with no precision@k, recall, mean reciprocal rank, or baseline comparison (e.g., unfine-tuned OpenCLIP). The similarity scores in Tables A1-A4 are almost all in the range 0.89–1.00, which suggests the embedding space is compressed; without ranking metrics, it is unclear whether the retrieved order reflects morphological similarity or other image properties such as brightness, orientation, or background. Please add a quantitative retrieval evaluation with labelled queries and report standard ranking metrics and a baseline.","section":"Section 4, Eq. (1)-(2), Tables A1-A4"},{"comment":"The 84±3% accuracy is measured on random splits of the 2,900 RadioGalaxyNET sources from EMU-PS1, and the final model is fine-tuned on the full dataset. However, the retrieval database comprises ~170,000 extended sources from the EMU main survey that are not independently labelled. The reported accuracy therefore does not demonstrate that the embedding space is morphologically structured at the scale of the actual retrieval database. Please evaluate on a labelled subset of the retrieval database, or otherwise provide evidence that the classifier/embedding generalises across the survey.","section":"Section 3.2, Fig. 2"},{"comment":"The validation of this key example relies on a known ORC candidate that is cross-identified using the authors' own catalogue (Gupta et al. 2025), and the training labels and descriptions derive from the same group's prior work. This limits the independence of the demonstration. The claim that EMUSE retrieves 'potential GLAREs' and 'half-ring-like structures' is based on the authors' visual classification rather than external validation. Please either provide independent confirmation (e.g., existing external catalogues, follow-up observations, or expert labels from outside the group) or temper the claims accordingly.","section":"Section 4.2, ORC image query"},{"comment":"The paper states in Section 4.1 that text queries currently underperform and are sensitive to phrasing, and that rare classes such as 'odd radio circle' return no results above a 0.9 threshold without a more descriptive prompt. Yet the abstract concludes the model is 'effective in retrieving and classifying radio sources' without these caveats. Please make the abstract consistent with the reported limitations, particularly regarding rare-source and text-query performance.","section":"Abstract and Conclusions"}],"minor_comments":[{"comment":"The phrase 'zero-shota learning' contains a stray footnote marker within the word; please fix it.","section":"Introduction"},{"comment":"The description 'Data clipping is applied between the 50th percentile level and the maximum values of the 99th and 99.9th percentiles for the first and second channels, respectively' is ambiguous; please clarify the clipping procedure for each channel.","section":"Section 3.2"},{"comment":"Several entries in the CatWISE Potential Host column contain 'nan'; please explain in the text or table caption that not all sources have a plausible infrared host.","section":"Tables A1-A4"},{"comment":"The Data Availability section says that fine-tuned models, embeddings, and metadata are accessible 'within main.py', which is not a direct access mechanism; please provide explicit links or download instructions.","section":"Data Availability"},{"comment":"The Conclusion contains two consecutive sentences beginning 'Future work should focus on...'; consider merging them for readability.","section":"Conclusions"},{"comment":"The reference list includes 'Norris et al. submitted' and 'Gupta et al. in preparation'; please update them or indicate their availability.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a journal like PASA and the authors have built a useful, publicly accessible tool. The main technical weakness is the absence of a quantitative retrieval evaluation, which is load-bearing for the central claim that EMUSE retrieves morphologically similar radio sources. The lack of a baseline comparison, ranking metrics, and independent validation of key retrieval examples should be addressed in revision. The public demo and code are strengths, and the classification accuracy reporting is well done."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EMUSE is a real tool: the authors fine-tune OpenCLIP on radio galaxy images with text descriptions, embed 170,000 EMU first-year sources, and provide a public Streamlit app, a GitHub repo, and the embeddings. That shipped artifact alone makes this worth reading. The held-out classification accuracy (84±3%, 10 random splits) is a legitimate positive result, and the paper is unusually honest about what it cannot do: text queries are phrasing-sensitive, rare classes like ORCs appear only because they were in the training set, and sources missing from Selavy are absent.\n\nThe soft spot is the retrieval claim. The paper presents two text and two image queries. There are no precision-at-k numbers, no baseline comparison (e.g., unfine-tuned OpenCLIP), and no ranking metrics. Appendix Tables A1–A4 show every top-50 similarity is ≥0.89, most ≥0.97, for every query. That compression is the real reason to worry: if the embedding space were morphologically structured at database scale, you'd expect query-dependent separation in the scores. Instead the ranking may be driven by brightness, orientation, background noise, or the radio/infrared channel mix. The authors themselves note the FR-II image query returned sources with matching sky orientation, and that different screenshots of the same ORC yield different result sets. So the concern holds. The classification accuracy on the fine-tuning set does not transfer automatically to the retrieval database; that is a separate claim needing separate evaluation.\n\nMinor: training labels and subcategory descriptions come from the authors' own prior catalogue work, and the ORC validation uses their own catalogue cross-identification. That is not fatal, because classification accuracy is on held-out splits, but it is worth flagging. Also, the exact expanded text descriptions are only available on request. The citation pattern is self-heavy but defensible, since the pipeline builds directly on their RG-CAT and RadioGalaxyNET work.\n\nWho is this for? EMU users who want to build morphological samples quickly, and anyone in astro-ML who wants a worked example of adapter-based CLIP fine-tuning on a 2,900-image dataset. It is not a claim about new astrophysics. I would send it to peer review, but with a request for a retrieval benchmark: at minimum precision@k on labeled queries, a comparison against unfine-tuned OpenCLIP, and some diagnostic on whether top-ranked neighbors remain stable under image preprocessing variations. Those are addressable.","headline":"A useful and honest tool paper: first CLIP-based radio morphology retrieval over EMU, but retrieval effectiveness is supported by four examples and the compressed similarity scores need a benchmark before I'd trust the search.","tokens_in":24441,"tokens_out":2679,"would_cite":false,"duration_ms":29492,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a general-purpose vision-language model lets astronomers search the radio sky by image or text.","keywords":["radio galaxies","multimodal foundation models","OpenCLIP","EMU survey","similarity search","morphological classification","Odd Radio Circles","zero-shot retrieval"],"falsifier":"Run a labelled retrieval benchmark: take held-out FR-I, FR-II, FR-x, R-type and Peculiar sources from RadioGalaxyNET, hide their labels, query EMUSE with each source image, and measure what fraction of the top-10 neighbours carry the same morphological label; if that fraction is no better than random or no better than the untuned OpenCLIP baseline, the cosine-similarity assumption fails. A sharper version is to query with a verified ORC and check whether the top-50 returns a statistically significant excess of confirmed ORCs and ORC candidates over chance, since the paper's own example already shows several non-ORC sources among the retrieved results.","tokens_in":1850,"feed_emoji":"🔭","tokens_out":3712,"duration_ms":80422,"temperature":0.7,"pith_summary":"This paper presents EMUSE, a search engine for the EMU radio survey that lets astronomers query roughly 170,000 extended radio sources by image or by a short text description. The authors show that fine-tuning the general-purpose vision-language model OpenCLIP on 2,900 labelled radio galaxies, with adapter-based training on a single GPU, pushes classification accuracy on the five main morphological classes to 84±3%. The fine-tuned model maps radio-infrared cutouts and natural-language labels into one embedding space, so that cosine similarity becomes the retrieval rule. The practical payoff is a tool that can assemble statistically robust samples of familiar classes and surface rare morphologies such as bent-tailed galaxies, star-forming ring galaxies, and Odd Radio Circle candidates without retraining per query.","feed_headline":"One fine-tuned model lets you search the radio sky by image or text","feed_subtitle":"EMUSE adapts a vision-language model to sort radio galaxies and retrieve rare rings like Odd Radio Circles.","key_machinery":"The load-bearing mechanism is the shared embedding space produced by contrastive fine-tuning of OpenCLIP. OpenCLIP, a CLIP-style model pre-trained on 2.32 billion LAION image-text pairs, is fine-tuned with adapters on the contrastive branch only, using 2,900 radio galaxy image-text pairs from RadioGalaxyNET with expanded labels such as 'An image of an FR-II or Fanaroff-Riley type II radio galaxy with edge-brightened lobes bent at an angle'. The fine-tuned image encoder converts each EMU source cutout into a compact vector, and retrieval is the $\\arg\\max$ of cosine similarity between a query embedding and the stored image embeddings, as given in Eqs. (1) and (2). The text encoder does the same for natural-language queries, which is what makes zero-shot retrieval possible without retraining on specific classes. A UMAP projection of the learned embeddings shows distinct clusters for the main categories, indicating that the embedding geometry carries the classification signal.","core_discovery":"The paper's central claim is that a foundation model pre-trained on ordinary internet image-text pairs can be cheaply adapted to radio astronomy and then serve as the core of a similarity search engine over survey-scale data. Fine-tuning only the contrastive branch of OpenCLIP on 2,900 RadioGalaxyNET sources, with radio and AllWISE infrared channels stacked into three-channel cutouts and paired with paraphrased morphological descriptions, yields a model that classifies held-out test sources into FR-I, FR-II, FR-x, R-type and Peculiar at 84±3% accuracy averaged over ten random splits. Deployed over first-year EMU data, EMUSE retrieves the top-$k$ sources by cosine similarity between the query embedding and precomputed image embeddings. The authors report that a text query for bent-tailed galaxies returns 50 visually consistent sources, and an image query using the known ORC J2103-6200 retrieves a known ORC candidate, half-ring structures, and potential GLAREs, even though only two ORCs were in the fine-tuning set. They also state that the same retrieval is imperfect: text queries are sensitive to phrasing, and rare or unseen classes such as supernova remnants and cluster relics are poorly handled.","pith_inferences":["If cosine similarity in the fine-tuned space is taken as the definition of morphological similarity, EMUSE's ranking should be validated with labelled retrieval metrics such as precision@k and mean reciprocal rank and compared against the untuned OpenCLIP baseline; the paper currently shows only four qualitative query examples.","The same approach should be testable on other survey-scale imaging problems: fine-tune a CLIP-style model on labelled cutouts, embed the full survey, and search, with labelled exemplars rather than compute or storage as the bottleneck.","Retrieval of ORC-like objects despite only two training examples hints that the model generalizes structural features such as closed rings and edge-brightened geometry rather than memorizing instances; a controlled study with deliberately sparse rare classes could test this directly.","Because the catalogue inherits Gal-DINO and Selavy detection limits, very faint sources are invisible to EMUSE; a catalogue-agnostic version that embeds all pixels or tiles would be a natural extension of the same pipeline."],"forward_implications":["Astronomers can query EMU survey data for a morphological class using plain language, such as 'a bent-tailed radio galaxy', and obtain ranked candidate lists with sky positions, integrated fluxes, and likely host galaxies.","Image-based queries let users find more examples of a known rare object: querying with ORC J2103-6200 surfaces an ORC candidate, half-ring structures, and potential GLAREs, suggesting a route to enlarge the ORC catalogue.","Replacing roughly 150 GB of survey cutouts with a few hundred megabytes of embeddings makes interactive similarity search feasible across the first-year EMU footprint of about 3 million detected sources.","The same fine-tuning recipe transfers to future EMU data releases and other multiwavelength surveys, since only new image embeddings and catalogue metadata need to be computed.","Classification of the five standard radio-galaxy families reaches 84±3% accuracy from 100 epochs of adapter training on one H100 GPU, so foundation-model adaptation is a practical path for other Big Data astronomy projects."],"supporting_citations":[{"why":"Introduces the CLIP contrastive language-image pre-training objective that EMUSE adapts for radio images and text.","marker":"Radford et al. (2021)"},{"why":"Provides the open-source OpenCLIP model pre-trained on 2.32 billion LAION image-text pairs that the paper fine-tunes.","marker":"Cherti et al. (2023)"},{"why":"Supplies the RadioGalaxyNET dataset of 2,800 FR-I, FR-II, FR-x and R-type galaxies used for fine-tuning.","marker":"Gupta et al. (2024a)"},{"why":"Defines the RG-CAT pipeline, extended to first-year EMU tiles to yield the roughly 170,000 extended sources searched by EMUSE.","marker":"Gupta et al. (2024b)"},{"why":"Describes the EMU survey whose first-year observations and Selavy catalogues are the search target.","marker":"Hopkins et al. (2025)"},{"why":"Presents EMU-PS1, the pilot survey from which the fine-tuning cutouts were drawn.","marker":"Norris et al. (2021a)"},{"why":"Reports ORC J2103-6200, the known Odd Radio Circle used as an image query and the class EMUSE aims to retrieve.","marker":"Norris et al. (2021b)"},{"why":"Provides CatWISE host galaxy names attached to retrieved sources in EMUSE output.","marker":"Marocco et al. (2021)"}],"fun_headline_variants":["Search the radio sky with text or images","AI tool finds rare radio galaxies in survey data","Multimodal search for the radio universe","Fine-tuned AI helps find odd radio circles","EMUSE: text or image search for the radio cosmos"],"cache_read_input_tokens":26624,"weakest_assumption_plain":"The whole retrieval story rests on the assumption that cosine similarity in the fine-tuned embedding space tracks astrophysical morphological similarity across the full survey; the paper validates this with only four qualitative query examples, with no ranking metrics and no comparison against the unfine-tuned model.","fun_headline_variants_meta":{"raw":{"variants":["Search the radio sky with text or images","AI tool finds rare radio galaxies in survey data","Multimodal search for the radio universe","Fine-tuned AI helps find odd radio circles","EMUSE: text or image search for the radio cosmos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1702,"prompt_tokens":1074,"completion_tokens":628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":690,"tokens_out":628,"duration_ms":6051,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:43:06.884287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a labelled retrieval benchmark: take held-out FR-I, FR-II, FR-x, R-type and Peculiar sources from RadioGalaxyNET, hide their labels, query EMUSE with each source image, and measure what fraction of the top-10 neighbours carry the same morphological label; if that fraction is no better than random or no better than the untuned OpenCLIP baseline, the cosine-similarity assumption fails. A sharper version is to query with a verified ORC and check whether the top-50 returns a statistically significant excess of confirmed ORCs and ORC candidates over chance, since the paper's own example already shows several non-ORC sources among the retrieved results.","supporting_citations":[{"cited_title":"W., Hallacy, C., et al","cited_arxiv_id":null,"evidence_quote":"Introduces the CLIP contrastive language-image pre-training objective that EMUSE adapts for radio images and text."},{"cited_title":"2023, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2818--2829","cited_arxiv_id":null,"evidence_quote":"Provides the open-source OpenCLIP model pre-trained on 2.32 billion LAION image-text pairs that the paper fine-tunes."}],"review_version":2}