{"id":"8a487fad-2078-4da1-85a0-b5893c63b400","arxiv_id":"2505.01711","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A text-only language model, fed structured text extracted from chest X-rays by an undisclosed vision pipeline, is reported to outperform multimodal baselines on the authors' new CXR-ClinEval benchmark.","lead":"This paper proposes CXR-TextInter, which interprets chest X-rays by converting images into structured text and feeding that text to a text-only large language model. It reports state-of-the-art results on a new benchmark, but the crucial image-to-text step is not described, so the claim cannot yet be verified.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is untestable as written: the undisclosed f_img2text pipeline supplies all visual information, so the reported F1 and VQA gains may only reformat the pipeline's own labels.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the hidden f_img2text could be doing the visual work. I agree because the paper does not even sketch the pipeline's architecture or training data, and the evaluation benchmark is not released, so there is no way to disentangle the LLM's reasoning contribution from the upstream detector's output. The ablation of the knowledge module (Table 2) shows an LLM-side effect, but it does not test whether the 14-class F1 reflects visual interpretation; a text model can add knowledge and still be downstream of a detector that has already named every finding. The human evaluation is suggestive but small (three raters, no agreement statistics). These are structural evidence problems, so keeping the rejection is appropriate. The alternative text-only paradigm is worth pursuing, but this manuscript does not yet establish the central SOTA claim.","tokens_in":10908,"tokens_out":4612,"duration_ms":49016,"concrete_test":"Recompute Table 1's pathology and VQA rows with f_img2text replaced by a public, disclosed CXR tagger (e.g., a released CheXpert/MIMIC-CXR labeler), and also compute an oracle ceiling by feeding ground-truth structured annotations into the same LLM. If the oracle run approaches ceiling F1 while the public-tagger run falls below CheXagent, the headline result is an artifact of the hidden pipeline; if the public-tagger run reproduces the gain, the paradigm survives and only needs the pipeline to be disclosed for verification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1's central comparison with multimodal baselines is only meaningful if f_img2text (Section 3.1, 'not detailed herein') converts CXRs into structured text that preserves visual information without already containing the evaluation answers. The manuscript never states what the pipeline detects, how it was trained, or what its error profile is. Since the structured input evidently lists findings, locations, and relationships, the pathology-detection task is potentially circular: Macro/Micro F1 would then measure the LLM's ability to echo labels that f_img2text already produced, and the comparison would credit the unknown pipeline plus a text model, not an image-interpretation system. The paper's own Section 4.8 confirms that missing-finding errors are 'primarily dependent on the upstream image-to-text pipeline's detection capabilities,' an admission that the system cannot recover information absent from Timg. With CXR-ClinEval unreleased and no error bars or significance tests, the quantitative SOTA claim cannot be independently verified from the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CXR-TextInter, a framework that replaces direct image understanding with a two-stage pipeline: an upstream image analysis system f_img2text converts chest X-rays into structured text Timg (Section 3.1), and a text-only LLM, augmented by a learned knowledge-graph integration module, is fine-tuned to produce reports and answers from Timg. The authors introduce the MediInstruct-CXR training dataset and the CXR-ClinEval benchmark, and report results on pathology detection, report generation, VQA, differential diagnosis ranking, and a human preference study, claiming state-of-the-art performance over multimodal baselines such as CheXagent and Med-PaLM-M. The paper acknowledges in Sections 4.8 and 5 that performance is constrained by the upstream pipeline, but the pipeline itself is not described.","tokens_in":11100,"tokens_out":5158,"duration_ms":48807,"significance":"If the claims were fully supported, this would establish a useful alternative architectural paradigm for medical imaging AI, showing that strong text-only LLMs can interpret images once a sufficiently rich structured text representation is available. The manuscript has clear strengths: the LLM-side architecture is formally specified with equations for knowledge integration (Section 3.2), the evaluation includes several tasks and a blinded human study, and the authors are explicit about the dependency on the upstream representation. However, the central quantitative claim is unverifiable in the submitted form because the upstream f_img2text pipeline is undisclosed; regardless of its internal quality, the pathology detection numbers cannot be attributed to the LLM. The paper therefore does not currently support the abstract's claim of state-of-the-art CXR interpretation.","major_comments":[{"comment":"The core of the method, f_img2text, is explicitly 'not detailed herein', yet it is the only component that converts pixels into the findings, locations, and relationships used in every evaluation task. Because Timg is the sole input to the LLM, the reported Macro F1/Micro F1 and VQA numbers in Tables 1-4 measure the combined pipeline-plus-LLM system, and if the pipeline already emits the pathology labels, the detection task is circular: the LLM would only need to echo Timg. The manuscript must specify the pipeline's architecture, training data, detection performance, and, crucially, demonstrate that Timg does not contain the test labels; without this, the SOTA claim in Table 1 cannot be assessed.","section":"Section 3.1"},{"comment":"The error analysis states that missing findings are 'primarily dependent on the upstream image-to-text pipeline's detection capabilities.' This is a direct admission that the LLM cannot recover information absent from Timg, so any improvement in missing-finding rates cannot be attributed to the proposed framework. Similarly, the reduced hallucination rate in Table 6 is expected by construction, since the LLM only generates from a list of detected entities; it does not show that CXR-TextInter interprets images better than multimodal baselines. The paper should report the pipeline's standalone finding-detection performance and measure the information loss between image and Timg.","section":"Section 4.8"},{"comment":"All quantitative results are point estimates. There are no error bars, confidence intervals, significance tests, or multiple training seeds, and the ablation study (Table 2) compares single runs. Given the modest differences between models (e.g., Report Score 0.848 vs. 0.835 in Table 1), the claimed superiority of CXR-TextInter and the contribution of the knowledge module in Section 4.4 are not statistically supported. The authors should provide variance estimates and significance tests, or at least multiple seeds.","section":"Tables 1-6"},{"comment":"The human evaluation uses three radiologists on 100 CXRs but reports only average Likert scores and raw preference percentages. No inter-rater agreement (e.g., Cohen's kappa or Fleiss' kappa), confidence intervals, or significance tests are given, so the statement that radiologists 'significantly' prefer CXR-TextInter is unsupported. Report agreement and a proper test of preference (e.g., Wilcoxon signed-rank on per-case scores).","section":"Section 4.7"},{"comment":"The knowledge integration module is a central contribution, but the mapping φ(S) that identifies relevant concepts from the input sequence, the construction of the Radiology Knowledge Graph G, and the choice of aggregation (Equation 4) are not specified. As a result, the ablation in Table 2 cannot be interpreted: the improvement could come from any number of implementation choices. The authors should define φ(S), state how G is built (including whether it uses the CXR-ClinEval or MediInstruct-CXR data), and report the retrieval/aggregation design.","section":"Section 3.2"}],"minor_comments":[{"comment":"The text introduces 'CXR-TextInter (placeholder name)', indicating the name is not final; remove this before publication.","section":"Section 1"},{"comment":"There are formatting errors in model names, e.g., 'LLaV A-Rad' and 'CXR-T extInter'; fix spacing.","section":"Tables 1-6"},{"comment":"A 'composite average score' is mentioned but never defined or reported; either define it or remove the reference.","section":"Section 4.2"},{"comment":"The Differential Diagnosis Ranking task is called 'hypothetical' and MRR is reported without a description of the candidate set or ground truth; specify the task protocol.","section":"Section 4.5"},{"comment":"The decoding strategy used in the reported experiments is not stated (greedy, beam, or sampling); specify it.","section":"Section 3.4"},{"comment":"References [27] and [29] are the same paper (Singhal et al.), and several baselines (MARIA, LLaVA-Rad) are described without citations; add the appropriate references.","section":"References"},{"comment":"The paper does not state whether MediInstruct-CXR and CXR-ClinEval will be released; for reproducibility, release details are needed.","section":"Datasets"}],"recommendation":"reject","confidential_remarks":"The reference list in Sections 2.1 and 2.2 contains a concentrated cluster of citations to [8]-[16] from the same research group that are not central to the method. This does not affect my recommendation, but the editor may wish to check citation practice. The decisive issue is that the reported experiments cannot validate the stated claims without the upstream pipeline; even with major additions, the evaluation design would need to be substantially reworked to support the abstract's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper has a plausible idea and some honest acknowledgments, but it hides the component that actually does the visual work, so the central SOTA claim can't be assessed from the manuscript. The reader's rejection is right, and I'd desk-reject it too.\n\nWhat's genuinely new: the proposal to decouple CXR interpretation into an upstream image-to-text stage and a text-only LLM stage, and the creation of a large instruction dataset and a multi-task benchmark to support that paradigm. The knowledge integration module (average of concept embeddings added to all token embeddings) is simple, but the ablation suggests it helps, at least on this benchmark. Those are legitimate contributions if the pipeline is made public and evaluation fixed.\n\nWhere it falls apart: Section 3.1 defines f_img2text and says 'not detailed herein.' That's the whole ballgame. The LLM only sees Timg, so the pathology detection F1 scores in Table 1 measure how well the LLM echoes labels that the pipeline already produced. The paper's own Section 4.8 admits missing findings are 'primarily dependent on the upstream image-to-text pipeline's detection capabilities'—so the LLM can't recover what isn't in the text. Comparing against multimodal baselines that read pixels is apples-to-oranges unless the pipeline's error profile is characterized.\n\nThe rest is consistent with that story: no error bars or significance tests anywhere, single-run ablation, three radiologists with no inter-rater agreement or confidence intervals, and a benchmark that isn't released. The citation list leans heavily on a particular group's papers, which looks like padding rather than engagement.\n\nI'm not accusing the authors of anything. The conclusion and error analysis do mention the pipeline dependency as a limitation. But the abstract and introduction still claim SOTA without that caveat, and the evidence as written can't support it.\n\nWho's it for? Researchers working on structured-input LLM methods might find the framework ideas worth a skim, but I wouldn't build on these numbers. It doesn't deserve referee time until the pipeline is disclosed and the evaluation is made fair.\n\nRecommendation: desk reject as submitted, with an invitation to resubmit after releasing the pipeline and running a proper comparison.","headline":"The text-centric idea is worth a look, but with the image-to-text pipeline undisclosed, the headline SOTA numbers are not trustworthy.","tokens_in":11647,"tokens_out":2995,"would_cite":false,"duration_ms":29084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured text lets a 1B text-only LLM top multimodal X-ray models","keywords":["chest X-ray interpretation","large language models","structured textual representation","radiology knowledge graph","report generation","visual question answering","multimodal foundation models","instruction tuning"],"falsifier":"Take a set of CXRs with ground-truth findings, run the upstream pipeline, and check whether each pathology label appears in the resulting structured text; any pathology missing from the text should also be missing from CXR-TextInter's reports and VQA answers. A direct version of the test is to feed the model the ground-truth structured description from a radiologist's report instead of the pipeline output: if scores rise substantially, the pipeline is the binding constraint. Alternatively, delete one finding from $T_{\\text{img}}$ and verify that the model's accuracy on questions about that finding collapses.","tokens_in":10687,"feed_emoji":"🩻","tokens_out":8265,"duration_ms":73293,"temperature":0.7,"pith_summary":"CXR-TextInter claims that a text-only large language model, given a structured textual description of a chest X-ray instead of the image itself, can outperform larger multimodal foundation models on pathology detection, report generation, and visual question answering. The authors fine-tune a 1-billion-parameter LLM on the MediInstruct-CXR instruction dataset, add radiology knowledge-graph embeddings to every input token, and evaluate on the CXR-ClinEval benchmark. There the model reaches 47.1 Macro F1 over 14 pathologies, a report score of 0.848, and 0.537 VQA accuracy, ahead of CheXagent and Med-PaLM-M. A blinded panel of radiologists rated its outputs higher on accuracy, completeness, and clinical relevance and preferred them in 60 percent of cases. The paper's broader claim is that visual information can be separated from reasoning: structure the image into text, and text-centric LLMs can do medical image interpretation.","feed_headline":"Structured text lets a 1B text-only LLM top multimodal X-ray models","feed_subtitle":"Feeding X-ray findings as structured text plus medical knowledge beats larger multimodal rivals on clinical tasks.","key_machinery":"The load-bearing component is the structured textual representation $T_{\\text{img}}$: an upstream image-analysis pipeline $f_{\\text{img2text}}$, not detailed in the paper, converts a chest X-ray into tokens that list findings, attributes, anatomical locations, and spatial relationships, and this text is the LLM's only view of the image. Around it, the method adds a Radiology Knowledge Graph $G=(V,E)$ whose concept embeddings are averaged into a sequence-level vector and added to every input token embedding, injecting medical context. The LLM is a decoder-only Transformer fine-tuned with cross-entropy on instruction-response pairs. The mechanism is that the structured text supplies the visual facts, the knowledge vector biases reasoning toward medically consistent conclusions, and the pretrained text-only model contributes its generative and reasoning abilities.","core_discovery":"The central claim is that a text-only LLM becomes the best CXR interpreter when the image is replaced by a rich structured text representation produced by an upstream vision pipeline. The paper argues that entities, attributes, anatomical locations, and spatial relations encoded as tokens are sufficient input for a fine-tuned LLM to detect pathologies, generate reports, and answer clinical questions, and that injecting a radiology knowledge graph improves performance most on rare and critical findings. The ablation, 47.1 versus 42.5 Macro F1 without the knowledge module, 0.848 versus 0.810 report score, and 0.537 versus 0.485 VQA accuracy, attributes a major share of the gain to that module. The authors also report a lower hallucination rate, 4.8 percent versus 8.1 percent for CheXagent, because the model reads explicitly detected entities rather than raw pixels. They present the results as evidence that decoupling visual analysis from language reasoning is a viable alternative to end-to-end multimodal models.","pith_inferences":["If the paradigm holds, the bottleneck shifts from model scale and multimodal fusion to the quality of image-to-text translation; improving the upstream pipeline should raise the ceiling directly, since the LLM cannot recover findings the text omits.","The same decoupling could transfer to ultrasound, mammography, or pathology slides whenever a reliable structured descriptor exists, letting medical imaging interpretation track text-LLM progress rather than multimodal training.","A direct test of the knowledge module's contribution would be to replace averaged knowledge embeddings with per-case retrieval of relevant concepts; if per-case retrieval further lifts rare-pathology F1, the current averaging under-uses the graph.","Combining structured image text with patient history and laboratory values is a natural next step, since both are text and the current design already accepts additional textual input alongside $T_{\\text{img}}$."],"forward_implications":["A 1B text-only LLM can outperform 3B-84B multimodal foundation models on CXR detection, report generation, and VQA when the image is pre-structured into text.","Adding medical knowledge-graph embeddings improves detection of rare pathologies and reduces medically inconsistent statements, as shown by the ablation and error analysis.","Hallucinated findings drop to 4.8 percent versus 8.1 percent for CheXagent because the LLM reads detected entities instead of raw pixels.","The CXR-ClinEval benchmark provides a reusable evaluation suite for LLM-centric medical image interpreters across classification, generation, and question answering.","Radiologists preferred CXR-TextInter outputs in 60 percent of blinded comparisons, suggesting the quality difference is clinically visible."],"supporting_citations":[{"why":"Defines the deep-learning CXR pathology-detection task and metrics this work builds on for the 14-class evaluation.","marker":"[2]"},{"why":"Supplies CheXagent, the main multimodal baseline CXR-TextInter must beat on CXR-ClinEval.","marker":"[5]"},{"why":"Establishes the generalist biomedical multimodal model family (Med-PaLM-M) used as a competing baseline.","marker":"[6]"},{"why":"Provides the Transformer architecture underlying the decoder-only LLM and its attention blocks.","marker":"[17]"},{"why":"Establishes the text-to-text paradigm that motivates feeding image content to the LLM as text.","marker":"[19]"},{"why":"Supplies the open-source Llama text-only LLM family that the framework fine-tunes.","marker":"[23]"},{"why":"Shows medical LLMs can reach expert-level performance on text medical questions, motivating the knowledge-augmented approach.","marker":"[27]"}],"fun_headline_variants":["1B text LLM tops larger multimodal X-ray rivals on chest findings","Text-only LLM beats multimodal X-ray models with structured text","Structured X-ray text + medical knowledge powers a small LLM to SOTA","No pixels needed: text-only LLM champions chest X-ray interpretation","Knowledge-augmented text LLM surpasses multimodal rivals on CXRs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the unstated assumption that the undisclosed upstream image-to-text pipeline converts a chest X-ray into structured text that preserves every clinically relevant finding, location, and relationship; if that pipeline omits or misstates a finding, the LLM cannot recover it, so the reported scores partly measure the pipeline's detection ability rather than the LLM's interpretation.","fun_headline_variants_meta":{"raw":{"variants":["1B text LLM tops larger multimodal X-ray rivals on chest findings","Text-only LLM beats multimodal X-ray models with structured text","Structured X-ray text + medical knowledge powers a small LLM to SOTA","No pixels needed: text-only LLM champions chest X-ray interpretation","Knowledge-augmented text LLM surpasses multimodal rivals on CXRs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000753,"raw_usage":{"total_tokens":3384,"prompt_tokens":1014,"completion_tokens":2370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2275}},"tokens_in":630,"tokens_out":2370,"duration_ms":15259,"temperature":1.0,"reasoning_tokens":2275,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:11:53.368086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of CXRs with ground-truth findings, run the upstream pipeline, and check whether each pathology label appears in the resulting structured text; any pathology missing from the text should also be missing from CXR-TextInter's reports and VQA answers. A direct version of the test is to feed the model the ground-truth structured description from a radiologist's report instead of the pipeline output: if scores rise substantially, the pipeline is the binding constraint. Alternatively, delete one finding from $T_{\\text{img}}$ and verify that the model's accuracy on questions about that finding collapses.","supporting_citations":[{"cited_title":"Nejm Ai 1(3), AIoa2300138","cited_arxiv_id":null,"evidence_quote":"Establishes the generalist biomedical multimodal model family (Med-PaLM-M) used as a competing baseline."},{"cited_title":"Advances in neural information pro- cessing systems 30 CXR-TextInter 15","cited_arxiv_id":null,"evidence_quote":"Provides the Transformer architecture underlying the decoder-only LLM and its attention blocks."},{"cited_title":"Journal of machine learning research21(140), 1–67","cited_arxiv_id":null,"evidence_quote":"Establishes the text-to-text paradigm that motivates feeding image content to the LLM as text."}],"review_version":1}