{"id":"4bcc6eb4-5a8f-4a12-a752-ddf00677854c","arxiv_id":"2607.06305","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"Text captions of radio galaxy images can classify FR-I vs FR-II morphologies comparably to image embeddings, but LoRA fine-tuning improves local class coherence without improving global image-text alignment.","lead":"This paper tests whether text descriptions of radio galaxy images can carry the same morphological information as the images themselves, using a vision-language model. A smart generalist might read it to gauge whether natural-language labels could cheaply annotate astronomical survey data.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Text F1 ~0.90 is achieved even with incorrect control captions, indicating class-correlated vocabulary rather than genuine morphological information; near-zero Recall@1 confirms text embeddings lack instance-level detail.","rationale":"The reader correctly identified the load-bearing concern: binary F1 on a two-class task is insufficient to support the claim that captions 'encode meaningful morphological information,' and the near-zero Recall@1 scores actively contradict the stronger interpretation. My analysis sharpens this by pointing to the most damning piece of internal evidence: control captions with incorrect descriptions achieve the same ~0.90 F1 as curated captions (Table 1, parenthetical differences), which the authors themselves flag as surprising. This directly shows the F1 is driven by class-correlated vocabulary, not morphological detail. The reader's verdict of CONDITIONAL with MODERATE confidence is appropriate. The paper is a legitimate proof-of-concept showing that VLM-generated captions can separate FR-I from FR-II at the linear-probe level, but the claim's scope ('meaningful morphological information,' 'viable complement') exceeds what the evidence supports. Key additional weaknesses the reader also noted: 104 test samples with no error bars, proprietary captioning model, no shipped code or data, and manual editing of test captions. None of these rise to the level of requiring a REJECT — the empirical finding is real and the application is novel — but they do warrant the CONDITIONAL verdict. The concrete test I propose (keyword ablation) would definitively settle whether the F1 reflects vocabulary correlation or genuine morphological encoding; if F1 survives keyword removal, the claim strengthens considerably.","tokens_in":7095,"tokens_out":2316,"duration_ms":189412,"concrete_test":"Mask or remove class-discriminative keywords (e.g., 'lobe,' 'jet,' 'edge-brightened,' 'core-dominated,' 'collimated') from the curated test captions, then recompute linear probe F1 on the text embeddings. If F1 drops to near chance (~0.50), the original score was driven by vocabulary correlation with the binary label rather than by morphological information in the embeddings. As a complementary check, recompute all Table 1 metrics on unedited Gemini-generated test captions (no manual correction) to quantify the editing confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that captions 'encode meaningful morphological information' — rests on linear probe F1 ~0.90 for text matching images (Table 1). But Table 1 also shows that control captions, which the authors describe as having 'less morphological detail and often assumed incorrect descriptors,' achieve essentially the same F1 (differences of ±0.01–0.03 shown in parentheses). The authors themselves note this is 'surprising' and attribute it to Gemini sampling 'vocabulary that is already highly distinctive.' This is the crux: the high F1 is driven by class-correlated vocabulary (words that co-occur with FR-I vs FR-II descriptions) rather than by morphological detail encoded in the text embeddings. If incorrect captions perform as well as curated ones, the F1 is not measuring morphological information content — it is measuring whether Gemini's vocabulary happens to correlate with the binary label. This is further corroborated by two independent signals in Table 1: (1) Recall@1 is near zero (0.01–0.07), meaning text embeddings cannot retrieve their paired image, and (2) KNN F1 for text (0.69–0.73) is substantially lower than for images (0.81–0.88), indicating text embeddings do not form locally coherent morphological neighborhoods. Together these show that text embeddings encode just enough class-level signal for a linear probe to separate two categories, but not instance-level or fine-grained morphological structure. The claim 'encode meaningful morphological information' and 'viable complement to image-based representations' is overstated relative to what the metrics support: binary class separability, not morphological encoding. Additionally, the manual editing of test-set captions (§2) introduces a confound: a human editor aware of the classification task may unconsciously inject class-discriminative vocabulary, further inflating text F1.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This manuscript investigates whether text captions generated by a large language model (Gemini 2.5-Flash) can encode morphological information about radio galaxies comparable to image-based representations. Using the MiraBest dataset (833 training, 104 test images), the authors generate captions with both a domain-specific prompt and a generic control prompt, then evaluate image-text alignment through the SigLIP-2 vision-language model with and without LoRA fine-tuning. The central finding is that linear-probe F1 scores for text embeddings (~0.90) match those for image embeddings (~0.90) on binary FR-I/FR-II classification, leading the authors to conclude that captions 'encode meaningful morphological information' and are a 'viable complement' to image representations. The paper also reports that fine-tuning improves local coherence but not global alignment, and that control captions perform nearly as well as curated ones.","tokens_in":8034,"tokens_out":1178,"duration_ms":202850,"significance":"The question of whether natural language captions can serve as a useful modality for astronomical data is timely and relevant to the astro-ph.IM community. The systematic comparison of curated vs. control captions, multiple fine-tuning configurations, and a suite of metrics (linear probe, KNN, retrieval) is a reasonable experimental design. The honest reporting of near-zero Recall@1 and the surprising parity between curated and control captions are commendable, as these findings are informative even though they complicate the paper's narrative. However, the significance is limited by the small test set (104 samples), the binary classification task that reduces morphology to two classes, and the absence of error bars or confidence intervals on any reported metric. The claim that captions encode 'meaningful morphological information' is not adequately supported by the evidence presented, as detailed below.","major_comments":[{"comment":"§3, Table 1: No error bars or confidence intervals are reported on any metric. With a test set of only 104 samples, the differences between curated and control captions (shown in parentheses, typically ±0.01–0.03 F1) are well within expected statistical fluctuation. Without bootstrap confidence intervals or similar uncertainty estimates, it is impossible to determine whether any reported difference is meaningful. This is load-bearing because the paper's narrative relies on interpreting these small differences (e.g., 'scientific captions only provide a small boost in performance'). Adding bootstrap CIs would either confirm or refute this interpretation.","section":null},{"comment":"§4, Discussion: The paper's central claim that captions 'encode meaningful morphological information' is contradicted by its own evidence. Table 1 shows that control captions—which the authors describe as having 'less morphological detail and often assumed incorrect descriptors' (§2)—achieve essentially the same linear-probe F1 as curated captions. The authors acknowledge this is 'surprising' but do not reconcile it with their conclusion. If incorrect captions perform as well as detailed ones, the F1 is measuring class-correlated vocabulary, not morphological information content. This is further supported by near-zero Recall@1 (0.01–0.07), indicating text embeddings lack instance-level morphological detail, and by KNN F1 for text (0.69–0.73) being substantially lower than for images (0.81–0.88). The conclusion should be revised to accurately reflect what the metrics show: text embeddings","section":null},{"comment":"§2, Data: The curated captions for the test set were 'manually controlled, and if need be, edited' by the authors. This introduces potential evaluator bias, as the same individuals who designed the experiment curated the test-set captions. While the paper notes this was done only for the test set, no details are given on the extent or criteria for editing. A brief description of the editing protocol, or an acknowledgment of this as a limitation, would strengthen the manuscript's credibility.","section":null}],"minor_comments":[{"comment":"Table 1: The parenthetical differences (e.g., '(-0.01)', '(+0.03)') are ambiguous—do they represent curated minus control, or control minus curated? A footnote or caption clarification would improve readability.","section":null},{"comment":"§3: The validation subset of ~100 samples from the training set is small. The sensitivity of early stopping to this choice should be briefly discussed.","section":null},{"comment":"Appendix C: The metric definitions are helpful but could note that Recall@1 is computed over the test set (not the full dataset) to avoid ambiguity.","section":null},{"comment":"§4: The phrase 'asymmetrical semantic anchoring' is introduced without sufficient definition. A clearer explanation of this concept would help readers.","section":null},{"comment":"Figure 1: Axis labels and legends could be more explicit about which embeddings are being compared.","section":null},{"comment":"§1: The reference to 'AstroLLaMA' appears to have a formatting issue (missing space or citation delimiter).","section":null},{"comment":"§2: The Gemini model version string is very specific; a note on reproducibility given that API versions change would be appropriate.","section":null},{"comment":"The paper would benefit from a brief discussion of potential failure modes or biases when applying this approach to larger, less curated datasets (e.g., with artifacts, noise, or hybrid sources).","section":null}],"recommendation":"major_revision","confidential_remarks":"The core experimental design is sound and the question is worth pursuing, but the manuscript currently oversells its conclusions relative to what the metrics demonstrate. The parity between curated and control captions is the most interesting finding in the paper, yet the conclusion ignores its implications. If the authors reframe their claims to match the evidence—acknowledging that text embeddings capture class-level signal but not instance-level morphological detail—the paper could be a solid contribution. The small sample size and lack of uncertainty quantification are also significant but addressable concerns."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive reading of our manuscript. The report correctly identifies several genuine weaknesses—particularly the absence of uncertainty estimates and an overstatement of our central claim—that we will address in revision. We respond to each major comment below.","responses":[{"response":"The referee is correct. With 104 test samples, the differences reported in Table 1 (typically ±0.01–0.03 F1) are well within the range of statistical fluctuation, and we should not have drawn interpretive conclusions from them without uncertainty estimates. We will add bootstrap confidence intervals (1000 resamples, 95% CI) to all metrics in Table 1. This will allow readers to assess which differences are meaningful and which are not. We expect this will confirm that the curated-vs-control differences are not statistically significant, which is consistent with the narrative we already report (that the parity is 'surprising'). However, we agree that the current presentation does not support this interpretation quantitatively, and the revision will fix this.","revision_made":"yes","referee_comment":"§3, Table 1: No error bars or confidence intervals are reported on any metric. With a test set of only 104 samples, the differences between curated and control captions are well within expected statistical fluctuation. Without bootstrap confidence intervals or similar uncertainty estimates, it is impossible to determine whether any reported difference is meaningful."},{"response":"We largely accept this criticism. The referee is right that the parity between curated and control captions, combined with near-zero Recall@1 and substantially lower KNN F1 for text (0.69–0.73 vs. 0.81–0.88 for images), complicates the claim that captions encode 'meaningful morphological information.' The linear-probe F1 of ~0.90 for text likely reflects class-correlated vocabulary rather than instance-level morphological detail, as the referee suggests. We will revise the conclusion to state more precisely what the evidence supports: text embeddings carry class-level discriminative signal (sufficient for binary FR-I/FR-II separation) but do not preserve instance-level morphological detail (as shown by near-zero Recall@1 and lower KNN F1). The phrase 'meaningful morphological information' will be replaced with more precise language, and the claim that captions are a 'viable complement' to image representations will be qualified to refer specifically to class-level labeling rather than instance-level morphological encoding. We retain the point that class-level Recall@1 (0.53–0.77) does indicate some morphological structure in the text embedding space, but we will frame this more cautiously.","revision_made":"yes","referee_comment":"§4, Discussion: The paper's central claim that captions 'encode meaningful morphological information' is contradicted by its own evidence. Control captions achieve essentially the same linear-probe F1 as curated captions. If incorrect captions perform as well as detailed ones, the F1 is measuring class-correlated vocabulary, not morphological information content. Near-zero Recall@1 and lower KNN F1 for text further undermine the claim. The conclusion should be revised to accurately reflect what the metrics show."},{"response":"This is a fair point. We will add a description of the editing protocol: the test-set captions were reviewed by one author (EL), with corrections limited to fixing factually incorrect descriptors (e.g., wrong spatial locations of features, incorrect morphological terms) and removing speculative language about the nature of emission. No captions were rewritten wholesale; the goal was to ensure factual accuracy of morphological description, not to optimize for classification performance. We will also add an explicit acknowledgment of evaluator bias as a limitation, noting that an independent annotator or blinded protocol would be preferable for future work at scale. We agree this transparency is important for credibility.","revision_made":"yes","referee_comment":"§2, Data: The curated captions for the test set were 'manually controlled, and if need be, edited' by the authors, introducing potential evaluator bias. No details are given on the extent or criteria for editing. A brief description of the editing protocol, or an acknowledgment of this as a limitation, would strengthen the manuscript's credibility."}],"tokens_in":7003,"tokens_out":865,"duration_ms":173281,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Short version: this paper asks a genuinely new question — can text captions encode radio galaxy morphology the way images do? — and the answer it actually provides is narrower than the one it claims. The application of SigLIP-2 to radio galaxy captions on MiraBest is a legitimate first look at a problem nobody else has tackled in this domain. The authors also deserve credit for reporting multiple metrics that, read together, undermine their own headline claim. That honesty matters. The systematic comparison of curated vs. control captions is a good experimental design choice, and the t-SNE plots and cosine similarity analysis in the appendices are useful supporting material. The finding that LoRA fine-tuning sharpens class boundaries without improving global image-text alignment is a real and interesting observation about how contrastive loss behaves on small scientific datasets. Here is the problem. The central claim — that captions “encode meaningful morphological information” — does not survive the paper's own Table 1. Control captions, which the authors describe as having incorrect descriptors and less morphological detail, achieve essentially the same linear probe F1 as curated captions (differences of 0.01–0.03). The authors call this “surprising,” but it is actually the key result: the F1 is measuring whether Gemini's vocabulary happens to correlate with the binary label, not whether the text encodes morphology. Two other metrics in the same table confirm this. Recall@1 is near zero (0.01–0.07), meaning text embeddings cannot retrieve their paired image. KNN F1 for text (0.69–0.73) is well below images (0.81–0.88), meaning text embeddings do not form locally coherent morphological neighborhoods. So what the paper actually shows is that text embeddings carry enough class-level signal for a linear probe to separate two categories on 104 test samples — not that they encode morphological information in any meaningful sense. The manual editing of test-set captions is a second concern: a human editor aware of the FR-I/FR-II distinction may inject class-discriminative vocabulary, which would inflate text F1 independently of any morphological content. No error bars are reported on any metric, and with 104 test samples, the 0.01–0.03 differences between curated and control captions are almost certainly noise. No code or data is shipped, and the captioning model (Gemini) is proprietary, so the pipeline is not independently reproducible. This is a reasonable proof-of-concept by people asking the right question. The conclusions need to be scaled back to match what the metrics actually show: binary class separability, not morphological encoding. A serious referee should push the authors to (1) report confidence intervals, (2) use unedited test captions, (3) test on a larger dataset, and (4) reframe the claim. The paper is worth refereeing because the application is novel and the empirical work, while limited, is transparent enough to critique productively.","headline":"Text captions match images on binary FR-I/FR-II F1, but the evidence points to class-correlated vocabulary, not morphological encoding.","tokens_in":7910,"tokens_out":1314,"would_cite":false,"duration_ms":97807,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":["95.75.Mn","95.80.+p"],"model":"glm-5.2","headline":"Text captions match images on radio galaxy classification","keywords":[],"falsifier":"If text embeddings continue to match image embeddings on binary classification but fail on any finer morphological task (e.g., multi-class morphology, regression of physical parameters, or instance retrieval), then the comparable F1 scores reflect class-label encoding rather than morphological information transfer.","tokens_in":7153,"feed_emoji":"📡","tokens_out":1200,"duration_ms":101357,"temperature":0.7,"pith_summary":"This paper asks whether natural language descriptions of radio galaxy images can carry the same morphological information as the images themselves. Using the MiraBest dataset of Fanaroff-Riley Type I and Type II radio galaxies, the authors generate text captions with a commercial vision-language model (Gemini 2.5-Flash) using both a generic prompt and a domain-specific prompt designed to elicit expert morphological descriptions. They then feed images and captions through the SigLIP-2 vision-language model, with and without lightweight LoRA fine-tuning, and evaluate whether text embeddings alone can distinguish FR-I from FR-II galaxies as well as image embeddings can. The central finding is that linear-probe F1 scores for text embeddings (~0.90) match those for image embeddings (~0.90) on this binary classification task, suggesting that captions encode enough morphological signal to serve as a complement to image-based representations. However, instance-level retrieval metrics (Recall@1 of 0.01–0.07) remain near zero, meaning the model can separate broad morphological classes but cannot reliably match a specific caption to its specific image. Fine-tuning with LoRA sharpens class boundaries and improves local clustering but does not improve global image-text alignment; the sigmoid loss decreases without pulling paired image-text embeddings closer together. A secondary finding is that scientifically detailed captions only marginally outperform generic, sometimes incorrect ones — Gemini's default vocabulary already contains enough discriminative signal for the binary task.","feed_headline":"Text captions match images on radio galaxy classification","feed_subtitle":"Generated descriptions of radio galaxy morphology classify FR-I vs FR-II as well as the images themselves — but can't match specific images","key_machinery":"The central object is the SigLIP-2 vision-language model, which produces joint embedding spaces for images and text. The authors use linear probes (lightweight classifiers on frozen embeddings) to test whether each modality preserves enough information to separate FR-I from FR-II galaxies, and retrieval metrics (Recall@1, class-level Recall@1) to test whether specific image-text pairs can be matched. LoRA fine-tuning adapts a small fraction (0.16–0.33%) of model parameters to the radio galaxy domain.","core_discovery":"Text embeddings derived from model-generated captions of radio galaxy images achieve the same linear-probe F1 (~0.90) as image embeddings on FR-I/FR-II binary classification, but instance-level alignment between specific images and their captions remains essentially absent (Recall@1 of 0.01–0.07). Fine-tuning improves class separability but not cross-modal retrieval, revealing a gap between class-level discrimination and instance-level morphological fidelity.","pith_inferences":[],"forward_implications":["If text embeddings genuinely carry morphological signal, natural language captions could serve as a lightweight labeling layer for large radio surveys, complementing tabular source catalogs that capture only patch-level Gaussian fits.","The gap between class-level F1 and instance-level Recall@1 suggests that current caption-based embeddings encode coarse categorical information but not fine-grained morphological detail, limiting their use for similarity-based discovery or individual source characterization.","The finding that generic captions perform nearly as well as expert ones implies that the discriminative signal may come from broad vocabulary differences between FR-I and FR-II descriptions rather than from genuine morphological reasoning, which would narrow the scope of the claim.","If the approach scales to larger and more morphologically diverse datasets, caption-based embeddings could enable zero-shot or few-shot classification pipelines for upcoming surveys without requiring labeled training images."],"fun_headline_variants":["Captions classify radio galaxies as well as images do","Text matches images on FR-I vs FR-II classification but not instance retrieval","Caption embeddings match image embeddings on galaxy morphology classification","Radio galaxy captions rival images for classification, fail on specific matching","Domain-specific captions match images on galaxy class but not instance alignment"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper treats binary FR-I/FR-II classification as a sufficient proxy for 'morphological information.' If the text embeddings merely encode which of two classes a galaxy belongs to — something a captioning model can infer from gross visual cues — rather than genuine morphological detail (lobe structure, jet orientation, diffuse emission patterns), then the claim that captions 'encode meaningful morphological information' is stronger than the evidence supports. The near-zero","fun_headline_variants_meta":{"raw":{"variants":["Captions classify radio galaxies as well as images do","Text matches images on FR-I vs FR-II classification but not instance retrieval","Caption embeddings match image embeddings on galaxy morphology classification","Radio galaxy captions rival images for classification, fail on specific matching","Domain-specific captions match images on galaxy class but not instance alignment"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":440,"prompt_tokens":374,"completion_tokens":66,"prompt_tokens_details":null},"tokens_in":374,"tokens_out":66,"duration_ms":46131,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T10:12:44.165719+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If text embeddings continue to match image embeddings on binary classification but fail on any finer morphological task (e.g., multi-class morphology, regression of physical parameters, or instance retrieval), then the comparable F1 scores reflect class-label encoding rather than morphological information transfer.","supporting_citations":[],"review_version":1}