{"id":"409ec7b8-dc2d-43dd-b1ec-62f493fbed69","arxiv_id":"2509.10873","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A topic-keyword semantic guidance framework improves automated radiology report generation and reaches state-of-the-art on two public chest X-ray benchmarks.","lead":"An automated chest X-ray report writer pulls similar past cases and uses detected disease topics plus symptom keywords to steer its text decoder. It reports higher BLEU and clinical accuracy scores than prior systems on IU X-ray and MIMIC-CXR, with code released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IU X-ray hyperparameter search appears to select on the test set; reported SOTA may be inflated by selection bias.","rationale":"The reader's verdict is CONDITIONAL and lists several concerns, with the weakest assumption being retrieval leakage. I agree that retrieval leakage is possible, but it is not confirmed by the text and would require the retrieval set to include test images' reports, which is unusual. A more immediately load-bearing concern is the hyperparameter selection procedure: the paper reports 108 experiments on IU X-ray and selects the best configuration, but never states that a validation split was used. If the test split was used for selection, the reported SOTA numbers are the best-of-108, which is a well-known source of optimistic bias. This directly affects the central claim because the IU X-ray improvements over the second-best are very small, and the same configuration is applied to MIMIC-CXR. The CheXbert circularity is also real but only affects CE metrics, not the NLG metrics that constitute the core SOTA claim. The retrieval leakage, if present, would be severe but is speculative and testable via code inspection; the hyperparameter issue is already evident from the paper's own description. Therefore, the most load-bearing concern is the lack of a stated validation split for hyperparameter tuning, and the concrete test is to inspect the code and rerun with a proper split. This does not change the CONDITIONAL verdict but tightens the condition: the authors must clarify the split and provide error bars before the SOTA claim can be accepted.","tokens_in":10111,"tokens_out":9310,"duration_ms":97349,"concrete_test":"Inspect the released code and configuration files to determine whether the 108 hyperparameter runs in Figure 3 were evaluated on the IU X-ray test split or on a held-out validation split. If they used the test split, define a standard train/validation/test split for IU X-ray, select NR and NK using only the validation set, retrain TKSG with that configuration, and evaluate on the test set. Compare the resulting BLEU-1/ROUGE-L to the reported Table I values. If the retrained model's BLEU-1 drops below the second-best baseline (0.504) or the improvement margin shrinks to near zero, the SOTA claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim of SOTA on IU X-ray rests on the configuration (NR=30, NK=20) chosen after 108 hyperparameter trials presented in Figure 3 and Section IV-C. The paper never states that these trials were evaluated on a held-out validation split; it says only that experiments were run on the IU X-ray dataset and that the best configuration was selected. If the 108 runs were scored on the test split, the reported numbers in Table I are the maximum over 108 tries, which is statistically biased upward and not directly comparable to baselines that used a single pre-specified configuration. This matters because TKSG's margins over the second-best on IU X-ray are tiny (e.g., BLEU-1 +0.007, BLEU-2 +0.017, METEOR +0.001), well within the range of selection-induced optimism. The same selected configuration is then transferred to MIMIC-CXR, so the concern indirectly affects both datasets. Even if the architecture is sound, the SOTA claim cannot be evaluated without knowing whether hyperparameter selection used test data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TKSG, a radiology report generation framework that augments a Swin-Transformer-based encoder-decoder with two semantic guidance signals: a global topic vector predicted from image features, and local keyword embeddings selected from predicted concept probabilities. Similar historical reports are retrieved with BiomedCLIP/MedCLIP/CLIP; topic labels are produced by CheXbert; keyword labels are multi-hot concept indicators. The model is evaluated on IU X-ray and MIMIC-CXR against seven baselines with NLG and CheXbert-derived clinical metrics, and the paper claims state-of-the-art performance in almost all metrics. Ablations (BASE, TSG, KSG-B, TKSG-B) and retrieval-model comparisons are also reported.","tokens_in":10365,"tokens_out":4209,"duration_ms":48979,"significance":"If the results hold, the paper makes a useful contribution by demonstrating that explicitly injecting topic- and keyword-level prior knowledge from retrieved historical reports can improve radiology report generation. The architecture is straightforward and reproducible, and the code release is a concrete strength. The ablation study consistently shows that each proposed component improves over the paper's own BASE model. However, the stronger SOTA and clinical-accuracy claims are not fully supported as written: the hyperparameter selection protocol is unclear, retrieval candidate sets are not specified, no error bars or significance tests are given, and the same CheXbert tool generates the topic-detector training targets and computes the clinical metrics, partially conflating training signal with evaluation.","major_comments":[{"comment":"The paper reports 108 hyperparameter trials on the IU X-ray dataset and states that the best configuration (N_R=30, N_K=20) was selected, but it never states whether selection used a held-out validation split. If these trials were scored on the same test set used for Table I, the reported numbers are the best of 108 attempts and are not comparable to baselines evaluated with a single configuration. This concern is load-bearing because TKSG's margins over the second-best IU X-ray results are tiny (e.g., BLEU-1 +0.007, BLEU-2 +0.017, METEOR +0.001, ROUGE-L +0.004). Please specify the selection protocol and report test-set results only for the single configuration chosen by validation.","section":"Section IV-C, Figure 3"},{"comment":"The candidate report set Z used for image-to-report retrieval is never defined. In particular, the paper does not state whether Z excludes the query image's own paired ground-truth report. Since the Keyword Detector uses features R from the retrieved reports (Section III-D, Eq. 8–9), inclusion of the paired report would let the detector directly see the ground-truth disease phrases, making the keyword guidance partially oracle and inflating both NLG and clinical metrics. Please define Z explicitly for training and evaluation, and confirm that the query's own report and any same-patient reports are excluded from retrieval.","section":"Section III-B, Section III-D"},{"comment":"The SOTA claim rests on point estimates without error bars or significance tests. Several IU X-ray improvements over the second-best baseline are extremely small (BLEU-4 +0.002, METEOR +0.001), and on MIMIC-CXR TKSG is actually worse than FSE/MA on BLEU-1 and BLEU-2. To support the 'current SOTA' claim, report means and standard deviations over multiple seeds and, where possible, paired significance tests (e.g., bootstrap or permutation tests).","section":"Table I"},{"comment":"CheXbert is used both to generate pseudo-labels for the Topic Detector (Eqs. 4 and 7) and to compute the clinical accuracy metrics (Precision/Recall/F1) on generated reports. Consequently, part of the reported CE improvement may simply reflect better agreement with the CheXbert labeling function used during training, not necessarily improved clinical correctness. The authors should either evaluate with an independent labeler (e.g., a different CheXpert-based model, an external rule-based labeler, or human clinician assessment) or explicitly discuss this limitation and temper the clinical-accuracy claim.","section":"Section III-C, Section IV-B"}],"minor_comments":[{"comment":"The text states that the text encoders of CLIP/BiomedCLIP/MedCLIP are 'decoder-only Transformers.' These models are typically bidirectional (encoder) transformers for text. Please correct this description or clarify what is meant.","section":"Section III-B"},{"comment":"The axis label and text repeatedly use 'BLUE-4'; this should be 'BLEU-4'.","section":"Figure 3, Section IV-C"},{"comment":"The sentence 'TKSG achieves the best performance across all evaluation metrics except RG-L' is contradicted by Table I, where TKSG has the highest ROUGE-L (0.394) on IU X-ray. Please correct the text.","section":"Section IV-D"},{"comment":"The first sentence of the methodology section refers to 'radioactivity report generation'; this should presumably be 'radiology report generation.'","section":"Section III"},{"comment":"The construction of the keyword vocabulary (N_W=100 concepts) is not described in detail. Please clarify from which corpus/split the frequency-based concept vocabulary is built, and whether this vocabulary is shared between datasets.","section":"Section III-D"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a useful incremental contribution to radiology report generation. TKSG combines a topic vector from CheXbert-derived disease probabilities injected into decoder embeddings with keyword-guided attention over retrieved similar reports. That is a sensible combination of known ingredients. What earns credit: the ablations are internally consistent; every added module improves on their BASE model across NLG and CE metrics on both IU X-ray and MIMIC-CXR. They also release code and compare three retrieval backbones. That is real, reproducible work.\n\nThe soft spots are real but not disqualifying. The strongest claim—state-of-the-art on two benchmarks—rests on point estimates without error bars or significance tests. More concerning, the hyperparameter search on IU X-ray runs 108 configurations and reports the best result, but the paper never states that a validation split was used. If that search was scored on the test set, the reported BLEU gains are selected maxima and not comparable to baselines that used a single configuration. The margins over the second best are tiny on several metrics (e.g., BLEU-1 +0.007). The same configuration is then transferred to MIMIC-CXR, so the issue is not confined to one dataset.\n\nAlso worth flagging: the image-to-report retrieval never explicitly says the query image's own paired ground-truth report is excluded from the candidate set. If it is not excluded, the keyword detector gets the ground-truth disease phrases, and the clinical metric gains become partly a leakage artifact. And since CheXbert generates both the topic-detector training targets and the clinical evaluation labels, part of the CE improvement may measure agreement with the training labeler rather than clinical correctness.\n\nNone of this sinks the architecture. The central design claim—that topic and keyword guidance help—is supported by the ablations. But the SOTA claim cannot be evaluated without a stated split, error bars, and an explicit retrieval exclusion rule. I'd recommend accepting for peer review with these concerns addressed; the paper is worth engaging, especially for the evaluation-methodology discussion.","headline":"Solid incremental architecture with consistent ablations; SOTA claim needs error bars and a stated hyperparameter split.","tokens_in":10822,"tokens_out":2004,"would_cite":true,"duration_ms":22620,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Injecting disease topics and symptom keywords into a decoder improves automated chest X-ray reports.","keywords":["radiology report generation","chest X-ray","topic-keyword semantic guidance","CheXbert","BiomedCLIP","multimodal retrieval","Swin Transformer","clinical metric evaluation"],"falsifier":"Run TKSG with the query's own ground-truth report explicitly excluded from the retrieval candidate set Z. If BLEU-4 and CheXbert F1 on MIMIC-CXR drop by a large margin relative to the reported numbers (or if a manual inspection shows that the top retrieved reports frequently equal the query's own report), the central claim that semantic guidance drives the gains collapses.","tokens_in":10012,"feed_emoji":"🩻","tokens_out":7001,"duration_ms":65929,"temperature":0.7,"pith_summary":"The paper proposes a framework called TKSG that injects two levels of semantic guidance into a standard Transformer encoder-decoder for chest X-ray report generation. A topic detector turns CheXbert's 14 disease labels into a probability vector that is added to every decoder input embedding, giving global disease context; a keyword detector fuses image features with features of historically similar reports retrieved by BiomedCLIP and selects top symptom words, which are then used as extra keys and values in a semantic-guided attention layer. The authors show on IU X-ray and MIMIC-CXR that each guidance module improves BLEU, METEOR, ROUGE, and CheXbert-based clinical F1 over a base model, and that the combined model achieves the best reported scores against seven published baselines. If correct, this is evidence that cheap automatic labels plus retrieved examples can push free-text radiology report generation closer to clinical use.","feed_headline":"Topic and keyword guidance boosts radiology report accuracy","feed_subtitle":"Disease-level and symptom priors combined yield top clinical and lexical scores on two chest X-ray benchmarks.","key_machinery":"The core mechanism is the two-level semantic guidance: (1) Topic Semantic Guidance (TSG) - a topic detector with a sigmoid layer over 14 CheXbert disease labels, whose predicted probabilities are mean-pooled from Swin Transformer features and mapped through one fully connected layer into a 512-d topic vector that is added to the word and positional embeddings of every decoder step; (2) Keyword Semantic Guidance (KSG) with Semantic-Guided Attention - a keyword detector that concatenates mean-pooled image features with mean-pooled features of the top-30 retrieved reports (from a 100-word concept vocabulary) to predict keyword probabilities, selects top-20 keywords, embeds them plus a rank embe","core_discovery":"The central claim is that explicitly modeling both the global disease state and local symptom-level content of a chest X-ray before decoding improves both the language quality and the clinical correctness of generated reports. The disease state is encoded by a topic vector predicted from image features against CheXbert labels, and the symptom content is encoded by keyword embeddings taken from a multimodal combination of image features and the most similar historical reports retrieved by BiomedCLIP. The topic vector is injected into the decoder's input embeddings at every timestep, and the keyword embeddings are attached to visual features as key/value pairs in a semantic-guided attention mo","pith_inferences":["The paper does not state whether the query image's own ground-truth report is removed from the candidate retrieval set Z. If it is not, the keyword detector may encounter the correct disease phrases for the very image during training, and part of the reported gains (particularly the keyword ablation and the clinical F1 jump) could be leakage rather than genuine semantic guidance. A simple masking ","Because the topic vector is derived from CheXbert, the model's disease-state prior is bounded by CheXbert's own accuracy; improving the labeler or using expert-annotated labels would likely further raise the ceiling, but also make the system less automatic.","The retrieval step is a training-time prior; a natural extension is to test TKSG with the retrieval corpus updated at inference time, or with retrieval restricted to the same patient's historical reports, which might be more clinically plausible than random similar cases.","The same two-level guidance could transfer to other structured reporting tasks beyond chest X-rays, provided a labeler and a retrieval model exist for that modality."],"forward_implications":["Adding a topic vector derived from automatic disease labels to the decoder input embedding improves all NLG and clinical metrics over the base encoder-decoder on both datasets.","Adding keyword guidance from retrieved historical reports improves report precision, with the best results when both topic and keyword guidance are used (e.g., CheXbert F1 rises from 0.387 to 0.467 on MIMIC-CXR).","The choice of retrieval model matters: BiomedCLIP-based retrieval yields higher BLEU and METEOR than MedCLIP or CLIP, indicating that domain-specific image-text retrieval is important for transmitting prior knowledge.","The framework's performance is sensitive to the number of retrieved reports (N_R) and keywords (N_K), so these hyper-parameters need tuning; the paper's default is 30 retrieved reports and 20 keywords.","Clinical accuracy gains exceed lexical gains on MIMIC-CXR, suggesting the guidance improves the factual content of reports more than just n-gram overlap."],"fun_headline_variants":["Topic and keyword priors sharpen X-ray reports","Disease and symptom guidance improves chest X-ray reports","Two-level semantic guidance ups radiology report fidelity","Global and local semantics boost X-ray report accuracy","Keyword-guided decoding sharpens automated radiology reports"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The candidate report set Z used in image-to-report retrieval is drawn from the training corpus, and the paper never states that the query image's own paired ground-truth report is excluded from Z; if that report can be retrieved, the keyword detector sees the correct disease phrases during training and the reported BLEU and clinical improvements are partly leakage artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Topic and keyword priors sharpen X-ray reports","Disease and symptom guidance improves chest X-ray reports","Two-level semantic guidance ups radiology report fidelity","Global and local semantics boost X-ray report accuracy","Keyword-guided decoding sharpens automated radiology reports"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":1741,"prompt_tokens":672,"completion_tokens":1069,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":997}},"tokens_in":416,"tokens_out":1069,"duration_ms":9297,"temperature":1.0,"reasoning_tokens":997,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:28:28.113837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TKSG with the query's own ground-truth report explicitly excluded from the retrieval candidate set Z. If BLEU-4 and CheXbert F1 on MIMIC-CXR drop by a large margin relative to the reported numbers (or if a manual inspection shows that the top retrieved reports frequently equal the query's own report), the central claim that semantic guidance drives the gains collapses.","supporting_citations":[],"review_version":1}