{"id":"e5b0dc03-fede-4593-94f7-6fd9299b6813","arxiv_id":"2412.12001","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A flexible radiology report generator and a rewritten MIMIC-CXR dataset reduce input-agnostic hallucinations to about 1.5 percent while keeping state-of-the-art clinical metrics on four input scenarios.","lead":"Radiology reports are usually generated from a single X-ray image, even though real radiologists work from different views and earlier exams. This paper introduces a cleaned dataset and a model that adapt to four input scenarios and makes far fewer input-agnostic mistakes than current open-source systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hallucination scores are self-confirming: DiscBERT, trained on Llama3 judgments within the same cleaning pipeline that creates MIMIC-RG4, is also the metric used to report hall.","rationale":"The reader's weakest assumption already identifies the load-bearing premise: the cleaning pipeline, and the DiscBERT discriminator trained inside it, are what define both the training data and the hallucination metric. My stress-test pass converges on the same point and sharpens it: the reported hall. numbers are not an external check on the model's factual grounding; they are a consistency check with the pipeline's own notion of 'clean'. This is not an accusation of dishonesty—the paper openly describes DiscBERT as a byproduct of the pipeline and reports its 200-case validation—but it means the headline claim 'minimizes input-agnostic hallucinations' is currently supported only by a self-referential measurement. The proposed radiologist-annotation test would settle whether the low DiscBERT scores reflect genuine input-groundedness or merely distributional mimicry of the cleaning process. Since the reader's CONDITIONAL verdict already requires independent validation of DiscBERT and release of the dataset, my concern does not move the verdict; it reinforces the conditionality. I also credit the paper's internal consistency: the ablations and mixed-training tables are coherent, the architectural contribution (adaptive token fusion) is clearly described, and the CheXbert-based CE evaluation is a standard community metric. The unresolved risk is specifically that the hallucination and cleaning evaluations share a single learned discriminator, and that risk is testable as outlined.","tokens_in":19773,"tokens_out":3928,"duration_ms":39552,"concrete_test":"Sample 200 generated reports per scenario from Table 3 (800 total). Have two board-certified radiologists, blinded to model identity and to DiscBERT's outputs, annotate each report for the paper's four categories of input-agnostic information (prior comparison, prior procedure, view, communication), after being shown the actual inputs (image, prior report, indication/history) for that scenario. Compare radiologist hall. rates and per-category agreement with DiscBERT. If radiologist-assessed hall. exceeds DiscBERT by more than 2 absolute percentage points, or if DiscBERT's per-category F1 against the radiologists falls materially below the Table 6 level, the 'minimal hallucination' claim is not established and the paper should report the radiologist-derived estimate instead.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLM-RG4 has 'minimal input-agnostic hallucinations' (Tables 2–3) is measured with DiscBERT, a BERT discriminator trained on Llama3-70B pseudo-labels generated during the same reconstruction pipeline that produced MIMIC-RG4 (Dataset Generation Pipeline; Table 6). The only external check is 200 manually labeled reports, but that validates DiscBERT against Llama3-style judgments, not against the true input-support relation for the four scenarios. Because LLM-RG4 is trained on reports that survived exactly this discriminator's notion of 'clean', low DiscBERT hall. scores may reflect the model matching the cleaning distribution rather than being genuinely input-grounded. The risk is concrete: if DiscBERT misses a class of input-agnostic content, that content survives cleaning and is reproduced by the model without being counted. Table 6 shows non-trivial error, e.g., prior-procedure recall of 79.3% and F1 of 86.0%, so the blind spot is not hypothetical. The same circularity can inflate CE: CheXbert-label consistency is the only content check in the pipeline, and CheXbert is also the CE evaluator, so rewritten reports that preserve CheXbert labels score well even if clinical nuance is distorted. Finally, for sw/mn/mw, a report containing prior comparisons is not necessarily hallucinated because longitudinal input supplies that information; the paper does not specify how DiscBERT's per-category outputs are conditioned on the input scenario, making the reported sw/mn/mw hall. numbers hard to interpret independently.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses radiology report generation from diverse input contexts. The authors construct MIMIC-RG4 from MIMIC-CXR using Llama3-70B to rewrite reports and a BERT-based discriminator (DiscBERT) to identify input-agnostic content (prior comparisons, prior procedures, view descriptions, communications) across four scenarios: single view with/without longitudinal data and multi-view with/without longitudinal data. They propose LLM-RG4, a Vicuna-7B-based model with an adaptive token fusion module that maps frontal, lateral, and previous-report inputs into a fixed number of tokens, and a token-level loss weighting scheme that up-weights sentences containing positive or uncertain CheXbert findings. Experiments on MIMIC-CXR and MIMIC-RG4 compare clinical efficiency (CheXbert F1), natural language generation (BLEU/ROUGE/METEOR), and hallucination (DiscBERT) against open-source baselines and retrained CXRMate, reporting state-of-the-art CE and low hallucination rates. Ablations show that ATF reduces token count while retaining performance, and that TLW improves CE.","tokens_in":20098,"tokens_out":7162,"duration_ms":64329,"significance":"If validated, the paper would make a useful practical contribution: a multi-scenario RRG benchmark, a flexible model that handles varying inputs in one framework, and a token-compression design that reduces the computational burden of additional inputs. The strengths are concrete: the MIMIC-RG4 dataset is a potentially reusable resource, the authors retrain CXRMate on the new benchmark, they apply Wilcoxon signed-rank tests, the code is released, and there is some manual validation of DiscBERT and of reconstruction quality. However, the central factual/hallucination claim relies on an evaluator (DiscBERT) that is trained inside the same data-cleaning pipeline that produced the training set, and the clinical-efficiency metric (CheXbert) is the same labeler used to filter training data. The external manual validation is small and does not break this circularity. The paper's significance therefore hinges on whether those claims survive independent, scenario-conditioned evaluation.","major_comments":[{"comment":"The central hallucination claim is partially self-referential and needs external validation.","section":"Dataset Generation Pipeline; Tables 2-3, 6"},{"comment":"The CE metric and the training-data filter share the same labeler, so an independent clinical check is needed.","section":"Dataset Generation Pipeline; Experimental Setup"},{"comment":"Single-run results are not sufficient for the strength of the SOTA and ablation claims.","section":"Tables 2-5; Table 4"}],"minor_comments":[{"comment":"The phrase 'perfectly corresponded input and output' overstates the dataset: Table 7 shows residual uninferable content even under the single-view no-longitudinal setting (e.g., 5.5% View), and Tables 8 and 10 show that prior comparisons are intentionally retained in sw and mw. Suggest using 'aligned' or 'minimized' instead of 'perfectly corresponded'.","section":"Abstract; Tables 7-10"},{"comment":"It is unclear whether the hallucination scores for the baseline models were computed by the authors using DiscBERT or quoted from the original papers; please specify the evaluation protocol for each baseline.","section":"Table 2"},{"comment":"There is a typo: 'as shown in Tabel 15' should read 'as shown in Table 15'.","section":"Appendix; Table 15"},{"comment":"The phrase 'radiologists tail content' should be 'radiologists tailor content'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about evaluator circularity is real and is not resolved by the paper's current manual validation. Table 6 itself documents nontrivial DiscBERT errors, and the scenario-conditioning issue for sw/mn/mw is not addressed. I would recommend requiring independent, scenario-conditioned human evaluation of hallucinations and an external clinical-efficiency check before acceptance, in addition to multi-seed results for the main claims. The dataset and code release are valuable, and the paper's core direction is sound, but the current evidence does not yet support the strength of the stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mark,\n\nThe thing to know: this is a solid, citable engineering paper with one load-bearing claim that is currently self-referential. The MIMIC-RG4 dataset paradigm — four realistic input scenarios (sn/sw/mn/mw) with paired inputs and outputs — is a real step beyond the single-image, full-findings mold. The adaptive token fusion module (ATF) that compresses multi-view plus longitudinal features into a fixed 204-token budget using frontal-image queries is a clean idea, and the ablation showing ATF matches interleaved inputs at 204 vs 502 tokens is encouraging. The token-level loss weighting (TLW) with CheXbert + Integrated Gradients is a simple loss-layer trick that their ablations show improves F1 in both training stages. They also did more than the usual baseline work: they retrained CXRMate, ran Wilcoxon signed-rank tests, and gave hyperparameters and prompts in the appendix.\n\nThe soft spot is the hallucination metric. DiscBERT is trained on Llama3-70B judgments produced while reconstructing MIMIC-RG4, and then the same DiscBERT is used to measure 'input-agnostic hallucinations' in generated reports. Low hall. scores may simply mean the model learned the cleaning distribution, not that it is genuinely input-grounded. The external validation is only 200 reports, and Table 6 shows nontrivial error (prior-procedure recall 79.3%, F1 86.0%). The CheXbert label-consistency filter in the pipeline plus CheXbert as the CE evaluator is a milder version of the same circularity, though the TLW ablation does show the weighting helps, so CE is not purely self-confirming. There are also no error bars or multiple seeds, and the clean-NLG comparison is scored against the same cleaned ground truth the model was trained on, which favors the authors' setup. The sw/mn/mw hall. numbers are hard to interpret: a prior comparison is legitimate when longitudinal input is present, and the paper doesn't say how DiscBERT conditions on scenario.\n\nNone of this sinks the architecture contributions. The ATF efficiency result and the four-scenario benchmark are valuable even if the hallucination claim is overstated. But the 'minimal input-agnostic hallucinations' sentence in the abstract should be softened until the metric is validated on an external label set.\n\nMy recommendation: send to peer review. The reviewer should ask for the dataset and code (with commit hash), multiple seeds, a larger external DiscBERT validation, and a scenario-conditioned hallucination protocol. If those come back clean, this is an accept. If not, the hallucination claims should be removed and the paper still has value as an efficiency-focused RRG method.\n\nSerious thinker: yes — the citations are genuinely engaged and the limitations (e.g., missing pneumonia mention in the case study) are acknowledged.","headline":"A genuinely useful four-scenario dataset and an efficient fixed-token fusion idea, but the 'minimal hallucination' headline is measured by a discriminator trained inside the same cleaning pipeline that built the training data.","tokens_in":20640,"tokens_out":3740,"would_cite":true,"duration_ms":32185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One 7B model, trained on an input-aligned cleaned dataset, generates radiology reports across four scenarios with only 0.2–2.1 percent input-agnostic hallucinations.","keywords":["radiology report generation","input-agnostic hallucination","multimodal large language model","adaptive token fusion","token-level loss weighting","MIMIC-RG4","clinical efficacy","longitudinal chest X-ray"],"falsifier":"Take a random sample of several thousand MIMIC-RG4 reports, have radiologists compare each reconstructed report with the original MIMIC-CXR report, and count how often a CheXbert-positive finding is dropped, added, or changed; if that rate is non-negligible, the reported CE and hallucination gains are attributable to the cleaning pipeline rather than to LLM-RG4. A cheaper check is to run LLM-RG4 on original un-cleaned MIMIC-CXR inputs and see whether the hallucination rate stays near 1.5 percent.","tokens_in":19533,"feed_emoji":"🩻","tokens_out":8849,"duration_ms":72741,"temperature":0.7,"pith_summary":"Radiology report generation today is usually trained as a fixed mapping from one frontal X-ray to the full findings section, even though real reports routinely draw on lateral views and prior examinations, and even though much of the text—comparisons, procedures, communications—cannot be inferred from a single image. The paper's proposal is to reorganize both data and model around four actual drafting scenarios (single or multi-view, with or without longitudinal reports) and to guarantee that every generated sentence is inferable from the supplied inputs. To that end it constructs MIMIC-RG4, a cleaned 172.6K-report dataset produced by an iterative Llama3-70B/DiscBERT rewriting pipeline, and presents LLM-RG4, a 7B instruction-tuned model whose adaptive token fusion keeps the token count fixed across input combinations and whose token-level loss weighting emphasizes positive and uncertain diagnoses. Across all four scenarios the model reports clinical-efficiency F1 of 0.559–0.610 and input-agnostic hallucination rates of 0.2–2.1 percent, compared with 54.9–91.7 percent for open baselines. The practical stake is a report generator that adapts to whatever data the radiologist has and does not invent priors.","feed_headline":"One model writes X-ray reports with near-zero invented findings","feed_subtitle":"Four input scenarios, one 7B model: MIMIC-RG4 and token fusion keep reports factual when views or priors vary.","key_machinery":"The argument is carried by three mechanisms. The adaptive token fusion module (ATF) uses the frontal image's perceiver output as query tokens for the lateral image and prior-report text streams, then concatenates the compressed token features along the feature dimension and projects them back to a fixed length, so the LLM always sees the same number of tokens no matter how many inputs are present; the paper reports roughly 60 percent fewer tokens than interleaved inputs with similar or better scores. The token-level loss weighting strategy (TLW) runs CheXbert to flag positive or uncertain disease labels, uses Integrated Gradients to attribute those labels to tokens, smooths the attributions, and raises the loss weight to λ=1.75 for entire sentences whose attribution exceeds a threshold, pushing the model to spend capacity on diagnostic content. The dataset pipeline matters as the third mechanism: Llama3-70B rewrites reports in up to three iterative rounds judged by DiscBERT, and any rewrite that changes CheXbert's disease labels is discarded, which is what makes low hallucination rates measurable and trainable.","core_discovery":"The central claim is that the standard single-image-to-full-report paradigm causes input-agnostic hallucinations, and that replacing it with a scenario-aware paradigm removes most of them. In MIMIC-RG4, each report is reconstructed so that its content is a function of the four possible input combinations—frontal image, lateral image, prior report text, and indication/history—with prior comparisons, prior procedures, view statements, and communications deleted or rewritten when the input cannot support them. LLM-RG4 then learns all four scenarios in one model: the adaptive token fusion module compresses available modalities into a fixed number of tokens seeded by the frontal image, and the token-level loss weighting uses CheXbert and Integrated Gradients to raise the training weight of sentences carrying positive or uncertain diagnoses. The reported outcome is state-of-the-art clinical-efficiency and language scores on both MIMIC-RG4 and the conventional MIMIC-CXR task, with essentially no input-agnostic hallucinations; the only exception the authors acknowledge is that cross-scenario consistency (e.g., mentioning pneumonia in one setting but not another) is not yet fully solved.","pith_inferences":["A natural extension is to apply the same cyclic Llama3/DiscBERT reconstruction to other imaging modalities (CT, MRI, ultrasound), where input-dependent content rules differ but the need to remove uninferable text is identical; DiscBERT could then serve as a real-time hallucination monitor during deployment.","The fixed-token ATF bottleneck suggests a testable hypothesis the paper leaves implicit: in narrowly specialized medical tasks, token-count compression does not cost accuracy, so the same design could fuse non-image inputs such as lab values or ECG traces without retraining the decoder.","Because original-MIMIC-CXR NLG scores stay competitive despite training on cleaned reports, one could test whether MIMIC-RG4-trained models adapt to institution-specific report styles with only light fine-tuning, which would bear on real deployment.","The paper's own case study shows cross-scenario inconsistency (pneumonia mentioned in some settings but not others); a concrete next step is combining TLW with decoding-time constraints that align diagnoses across the four scenarios."],"forward_implications":["A single 7B model can serve all four input scenarios with CE F1 0.559–0.610 and hallucination rates 0.002–0.021, outperforming per-task baselines, so hospitals would not need separate models for each workflow.","Adaptive token fusion keeps the LLM's token count constant, cutting tokens by about 60 percent when multi-view and longitudinal data are present, with similar or better scores than interleaved inputs—so adding modalities need not proportionally increase compute.","Token-level loss weighting improves CE F1 on both ATF and interleaved architectures, meaning loss-layer emphasis on positive and uncertain diagnoses is a transferable way to improve clinical accuracy without reinforcement learning or inference-time classifiers.","Mixed training across the four scenarios acts like data augmentation, with the largest CE gains in the harder multi-view and longitudinal settings.","Training on reconstructed reports makes generated reports nearly free of prior-comparison, prior-procedure, view, and communication hallucinations, while remaining competitive on original MIMIC-CXR NLG metrics against closed-source and specialized models."],"supporting_citations":[{"why":"Supplies MIMIC-CXR, the only public dataset with multi-view and longitudinal chest X-rays and free-text reports, from which MIMIC-RG4 is generated.","marker":"(Johnson et al. 2019)"},{"why":"Defines the report information categories (positive/negative mentions, prior comparisons, prior procedures, views, communications) that the pipeline judges and rewrites.","marker":"(Nguyen et al. 2023)"},{"why":"Provides Llama3-70B, the generator that judges and reconstructs reports in the cyclic data pipeline.","marker":"(AI@Meta 2024)"},{"why":"Supplies CheXbert, used to discard pipeline rewrites that change disease labels and to identify positive/uncertain labels in the token-weighting strategy.","marker":"(Smit et al. 2020)"},{"why":"Supplies the Perceiver architecture used in the adaptive token fusion module to compress each modality into a fixed number of tokens.","marker":"(Jaegle et al. 2021)"},{"why":"Provides RAD-DINO, the frozen image encoder for frontal and lateral chest X-rays.","marker":"(Pérez-García et al. 2024)"},{"why":"Provides BiomedVLP-CXRBERT, the frozen text encoder for prior report text.","marker":"(Boecking et al. 2022)"},{"why":"Provides Vicuna-7B v1.5, the LLM backbone that is instruction-tuned to generate the reports.","marker":"(Chiang et al. 2023)"},{"why":"Supplies Integrated Gradients, used to attribute CheXbert disease labels to tokens so that positive/uncertain sentences can be upweighted in the loss.","marker":"(Sundararajan, Taly, and Yan 2017)"}],"fun_headline_variants":["One model, four input scenarios, near-zero hallucinated findings","Four radiology input contexts, one LLM, near-zero invented findings","One 7B model drafts factual X-ray findings from any input combo","Scenario-aware radiology report generation: four inputs, one LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic report-cleaning pipeline removes input-agnostic content from 172.6K training reports without distorting clinical meaning or injecting its own artifacts, since only 200 reports were manually validated and a CheXbert label-consistency filter is the main large-scale guard.","fun_headline_variants_meta":{"raw":{"variants":["One model, four input scenarios, near-zero hallucinated findings","Four radiology input contexts, one LLM, near-zero invented findings","One 7B model drafts factual X-ray findings from any input combo","Scenario-aware radiology report generation: four inputs, one LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3121,"prompt_tokens":1054,"completion_tokens":2067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":1991}},"tokens_in":670,"tokens_out":2067,"duration_ms":15140,"temperature":1.0,"reasoning_tokens":1991,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:23:23.595367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of several thousand MIMIC-RG4 reports, have radiologists compare each reconstructed report with the original MIMIC-CXR report, and count how often a CheXbert-positive finding is dropped, added, or changed; if that rate is non-negligible, the reported CE and hallucination gains are attributable to the cleaning pipeline rather than to LLM-RG4. A cheaper check is to run LLM-RG4 on original un-cleaned MIMIC-CXR inputs and see whether the hallucination rate stays near 1.5 percent.","supporting_citations":[],"review_version":1}