{"id":"0ad2c2f7-f3fc-4084-b9c2-d203e6c1326a","arxiv_id":"2608.10505","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RadFusion makes radiology reports threshold-controllable by having an LLM rewrite classifier-guided findings, yet the reported ROC conformance follows by construction.","lead":"RadFusion hooks a radiology image classifier to a report-writing AI so doctors can dial a report's sensitivity up or down for triage versus confirmation. On chest X-ray data, rewritten reports track the classifier's accuracy curve, but this behavior is wired in by the rewriting instructions rather than being an emergent finding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ROC conformance is largely a closed-loop artifact: the rewriter is instructed to match thresholded classifier labels, GPT-5 extracts labels back from template phrases, and both training and evaluation use VisualCheXbert ground truth; independent labels are needed before the clinical accuracy…","rationale":"The reader's weakest assumption identifies the same load-bearing weakness I find: VisualCheXbert labels serve as ground truth both for training the MI2 classifier and for computing every ROC curve, so the classifier's AUC and RadFusion's report ROC are measured against the same automatic labeler rather than against clinical truth. I add one reinforcing observation: the evaluation protocol is a closed loop. The LLM rewriter is explicitly instructed to make the report's stated diagnoses match the thresholded classifier outputs, and GPT-5 is then used to convert the rewritten report back into class labels. If the rewriter follows instructions and the extractor recognizes the template phrases, the extracted labels equal the classifier's thresholded decisions, making the reported conformance near-tautological. This does not destroy the engineering contribution, which is a portable mechanism for threshold-controllable reporting, but it does mean the empirical ROC curves and the matched sensitivity and specificity gains cannot, by themselves, establish clinical or regulatory validity. The paper's own Limitations section acknowledges the automatic-label-extraction risk, which strengthens rather than weakens this concern. Because the reader's conditional verdict already requires clarification and independent label validation, my analysis does not move the verdict; it sharpens the specific condition under which the central claim would be trustworthy. The proposed radiologist-annotation check would settle whether the conformance and accuracy improvements survive an independent ground truth.","tokens_in":17603,"tokens_out":6048,"duration_ms":65926,"concrete_test":"Recompute the entire threshold-control evaluation (classifier ROC, report ROC, and the matched sensitivity and specificity comparisons in Section 4.3) on a held-out subset of the MIMIC-CXR test split for which two board-certified radiologists independently annotate the 13 CheXpert classes, with adjudication and with the classifier never trained on those annotations. Use the radiologist labels as ground truth for both the classifier and the extracted report labels. If the report ROC still tracks the classifier ROC and the 6.9% and 20.7% improvements persist, the conformance claim is robust to label noise and the closed-loop concern is not fatal. If the curves separate or the gains shrink, the reported conformance and accuracy improvements are artifacts of the VisualCheXbert labeler and the GPT-5 extraction loop.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RadFusion's headline conformance result is close to a closed-loop identity. In Section 3.4 the rewriting instruction forces the LLM to assert every class with yhat_i=1 and remove or negate every class with yhat_i=0, using template text segments; in Section 4.2 the rewritten report is converted back to labels with GPT-5. If the rewriter obeys and GPT-5 reads the templates, the extracted labels are exactly the thresholded classifier outputs, so the 'report ROC' is the classifier ROC by construction. The green-cross comparison then measures how well GPT-5 can extract the original report's findings, not the clinical accuracy of either system. The clinical and regulatory value of the ROC therefore depends entirely on the ground truth against which the classifier ROC is computed. That ground truth is not independent: Section 3.2 trains the MI2 classifier with VisualCheXbert labels, and the Figure 4 caption in Section B states that ground-truth labels are obtained from reference reports via VisualCheXbert. The classifier's AUC and every RadFusion ROC point are measured against the same automatic labeler used to supervise the classifier. If VisualCheXbert disagrees with a radiologist, both curves are biased in the same direction, and the reported 6.9% sensitivity and 20.7% specificity gains may reflect extraction bias (template phrases are easier for GPT-5 to label) rather than real diagnostic improvement. The paper's own Limitations section concedes that 'our evaluation relies on automatic label extraction from reports, which may itself introduce error.' This is the load-bearing weakness: the central claim of 'validated ROC performance' holds only conditional on VisualCheXbert being a faithful proxy for clinical truth and on GPT-5 extraction being unbiased, neither of which is established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RadFusion is a framework for threshold-controllable radiology report generation. It fuses a multi-label classifier (default: a fine-tuned MedImageInsight model) that provides per-disease confidence scores, a VQA-based report generator (QRad Auto-VQA) that produces detailed findings, and an off-the-shelf LLM rewriter that rewrites the generated report so that its stated diagnoses match the classifier's binary decisions at a chosen threshold. On MIMIC-CXR, the paper reports that sweeping the threshold and mapping the rewritten reports back to class labels reproduces the classifier's ROC curve, and that threshold-controlled reports improve sensitivity by 6.9% at matched specificity and specificity by 20.7% at matched sensitivity over uncontrolled generation. The paper also compares alternative classifier implementations (MI2 fine-tuned, QRad linear probe, QRad token logit) and alternative LLM rewriters, and describes an extension to three-dimensional threshold control.","tokens_in":17907,"tokens_out":6518,"duration_ms":57481,"significance":"The underlying idea—coupling a perception model's operating point to free-text radiology reporting—is genuinely novel and potentially useful for clinical deployment, where the sensitivity–specificity trade-off matters. The framework is portable across classifiers and generators, and the ablation studies provide useful engineering evidence about which components matter. However, the headline empirical claims are undermined by two related circularities. First, the ROC conformance result is largely guaranteed by construction: the rewriter is instructed to assert every class with yhat_i=1 and negate every class with yhat_i=0, and the evaluation extracts labels back from the rewritten text. Second, the ground-truth labels for both classifier training and ROC evaluation are the same automatic labeler (VisualCheXbert), so the reported accuracy improvements may reflect extraction bias rather than genuine diagnostic gain.","major_comments":[{"comment":"The ROC conformance result is a closed-loop artifact. The rewriting instruction in Section 3.4 explicitly requires the LLM to add or remove disease statements so the report matches the thresholded classifier decisions, using template text segments. Section 4.2 then converts the rewritten report back to class labels with GPT-5. If the rewriter obeys the instruction and GPT-5 reads the template phrases, the extracted labels are exactly the thresholded classifier outputs, so the 'report ROC' is the classifier ROC by construction. This makes the conformance claim a sanity check of instruction following, not an independent validation of the report's diagnostic content. The paper should present this as a design guarantee and focus the evaluation on the non-trivial aspects (e.g., how often the rewriter fails, extraction reliability), rather than claiming that the ROC reproduction is an empirical finding.","section":"Section 3.4 and Section 4.2"},{"comment":"The ground-truth labels are not independent of the classifier being evaluated. Section 3.2 states that the MI2 classifier is trained using class labels from VisualCheXbert, and the Figure 4 caption in Section B states that ground-truth labels are obtained from the reference reports via VisualCheXbert. Consequently, the classifier's ROC and every RadFusion ROC point are measured against the same automatic labeler used for supervision. If VisualCheXbert disagrees with a radiologist, both the classifier and the report evaluation are biased in the same direction, and the reported sensitivity and specificity improvements may reflect easier extraction of template phrases rather than real diagnostic improvement. Independent radiologist labels, or at least a separately validated labeler not used in training, are required to support the accuracy claims.","section":"Section 3.2 and Section B (Figure 4 caption)"},{"comment":"The procedure for computing 'matched specificity' and 'matched sensitivity' is not described. The non-controlled report is a single operating point, while the threshold-controlled reports span a curve; matching requires specifying which thresholds are used and how the matching is performed (e.g., interpolation, nearest point, or threshold on the classifier scores). Without this detail, the headline numbers of 6.9% sensitivity gain and 20.7% specificity gain are not reproducible. This is a load-bearing methodological omission because the accuracy improvement is a central claimed contribution.","section":"Section 4.2"}],"minor_comments":[{"comment":"The abstract and method describe 14 CheXpert classes, but the evaluation uses 13 classes because 'No Finding' is excluded. This should be stated consistently in the main text, not only in Section 4.2.","section":"Abstract and Section 4.2"},{"comment":"The green cross is labeled 'Original Report' in Figure 2 but 'Non-controlled reports' in Section 4.2. Use consistent terminology throughout.","section":"Figures 2 and 4"},{"comment":"Equation (2) uses a learnable temperature τtemp, and then a separate temperature scaling scalar T is introduced for calibration. Clarify the relationship between these two parameters and whether T is applied on top of the already temperature-scaled logits.","section":"Equation (2) and Section 3.2"},{"comment":"The column header 'A VG' should be 'AVG' or 'Avg' for readability.","section":"Tables 1-3"},{"comment":"The statement that 'the instruction was tuned on GPT-5' lacks details on the tuning procedure and the amount of validation data used. This is relevant for interpreting the cross-model comparisons.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The central evaluation is close to circular: the conformance result is guaranteed by the rewriting instruction and the closed-loop label extraction, and the ground truth is the same automatic labeler used for training. I recommend requesting a substantive revision that either (a) reframes the paper as a systems/engineering contribution with the conformance result presented as a design property, and adds an independent evaluation (e.g., radiologist-labeled ground truth or a validated labeler not used in training) for the accuracy claims, or (b) narrows the claims to those supported by the current evaluation. The idea itself is publishable if the framing is corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things before anything else. The headline conformance result is largely a closed-loop artifact, and the paper would be more honest if it said so. Section 3.4 instructs the LLM to assert or deny each class according to the thresholded classifier, using template phrases; Section 4.2 then uses GPT-5 to read labels back out of those same template phrases. Of course the report ROC tracks the classifier ROC—the extraction is almost the inverse of the rewriting. This does not sink the framework, but it turns “conformance validates the reports” into “conformance is the output of the instruction.” Separately, both training and evaluation lean on VisualCheXbert: the classifier is supervised with it, and all ROC ground truth comes from it. If VisualCheXbert is biased relative to a radiologist, the classifier’s ROC and every RadFusion point inherit that bias. The paper’s own Limitations section admits automatic label extraction may introduce error, but the abstract and Section 4 still present the results as validated accuracy.\n\nWhat is genuinely useful: the system is clearly described, the ablations across classifier implementations and LLM rewriters are sensible, and the UI plus three-zone threshold extension are thoughtful touches. Combining classifier decisions with VQA-driven evidence retrieval before rewriting is a reasonable engineering contribution, and it likely does improve fidelity to the labeler’s notion of truth. The citation pattern is unremarkable; they engage the relevant prior work.\n\nSoft spots, in rough order. First, the 6.9% and 20.7% improvements over uncontrolled generation may be inflated by extraction bias: template phrases are easier for GPT-5 to label than free-text original reports. Second, there are no error bars anywhere; with 2,347 test studies, that matters. Third, the three-dimensional urgency extension is presented without evaluation—fine as a sketch, but it should not read as a result. Fourth, there is no code or data release. None of this is disqualifying, but the central claim needs reframing.\n\nThis paper is for people building deployable radiology AI who care about operating-point selection and regulatory ROC arguments. A serious referee should be assigned; it deserves peer review, not desk rejection. My recommendation is conditional acceptance with explicit revision requirements: state that conformance is by construction, validate labels against an independent reference such as a small radiologist panel, add confidence intervals, and release code or detailed examples.","headline":"Useful systems paper, but the headline conformance result is largely by construction and the evaluation leans on the same automatic labeler for training and ground truth; conditional acceptance with reframing and independent validation would be appropriate.","tokens_in":18479,"tokens_out":2800,"would_cite":false,"duration_ms":28140,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RadFusion makes generated radiology reports threshold-controllable: sweeping the threshold and mapping reports back to labels reproduces the classifier's ROC curve.","keywords":["threshold control","radiology report generation","ROC curve","sensitivity-specificity trade-off","visual question answering","large language model rewriting","multi-label classification","chest X-ray imaging"],"falsifier":"Take a held-out set of chest X-ray images from a public dataset, have a panel of radiologists independently label the disease classes, and compute the threshold-controlled reports' ROC curve against those radiologist labels; if the report ROC no longer closely tracks the classifier ROC—for example, AUC falling several points behind on multiple classes—the claimed conformance would be falsified.","tokens_in":17429,"feed_emoji":"🩻","tokens_out":15424,"duration_ms":111739,"temperature":0.7,"pith_summary":"The paper argues that automated radiology report generation should offer the same threshold control that image classifiers have, and introduces RadFusion to provide it. RadFusion combines a multi-label chest X-ray classifier, a visual-question-answering (VQA) based report generator, and an LLM rewriter, so that the report's stated diagnoses follow the classifier's decisions at any chosen threshold while retaining clinical detail. On the MIMIC-CXR dataset, sweeping the threshold and converting the rewritten reports back to class labels reproduces the classifier's ROC curve, giving generated reports a measurable sensitivity–specificity trade-off. This matters because clinical settings differ: emergency triage wants high sensitivity, confirmatory assessment wants high specificity, and regulatory approval typically expects ROC-based evidence.","feed_headline":"Sweep one threshold and radiology reports track the classifier's ROC","feed_subtitle":"On MIMIC-CXR, sweeping one threshold reproduces the classifier's ROC, enabling operating-point selection.","key_machinery":"The load-bearing mechanism is a closed-loop threshold–rewrite–extract protocol. A calibrated multi-label classifier produces per-class confidence scores; a threshold $\\tau$ binarizes them into positive and negative lists; an LLM rewriter, guided by a template dictionary that maps each disease class to natural-language expressions, rewrites the report generated by a visual-question-answering (VQA) based report generator so that positive classes are asserted and negative classes are denied, while non-class content and clinical attributes such as location and severity are preserved. Follow-up VQA queries supply grounded descriptions when a low threshold flips a class to positive. The rewritten report is then converted back to class labels to compute ROC curves, which the authors show track the classifier's ROC.","core_discovery":"The paper's central claim is that a radiology report generator can be made threshold-controllable by fusing a perception model with a generative model: a classifier supplies per-disease confidence scores, a VQA-based report generator supplies grounded descriptive text, and an LLM rewrites the report so that its diagnoses are consistent with the thresholded classes. On MIMIC-CXR, over 13 CheXpert disease classes, the ROC curve computed from the rewritten reports closely tracks the classifier's own ROC curve with matched AUC values, and threshold control improves diagnostic accuracy over uncontrolled generation: sensitivity increases by 6.9% at matched specificity and specificity by 20.7% at matched sensitivity. The framework also extends to three-dimensional thresholds that create negative, semi-positive, and positive zones with urgency emphasis, all adjustable through rewriting instructions alone.","pith_inferences":["The same closed-loop rewriting could extend to other clinical documents (e.g., pathology reports or discharge summaries) wherever a classifier's decisions must be reflected in free text, making threshold control a general pattern rather than a radiology-specific trick.","The LLM rewriter is the most likely failure point: conformance depends on rewriting fidelity, so one testable extension is to measure how tightly report ROC tracks classifier ROC as rewriter size and instruction quality vary.","Because the label set is fixed, findings outside the predefined classes remain uncontrolled; an open-set extension that lets thresholded decisions cover free-form findings could broaden the framework's clinical scope.","The reported accuracy gains may partly reflect the rewriter correcting generator omissions, not just threshold alignment; a reader could test this by comparing reports rewritten with a permissive threshold (all positives) against the generator's original reports."],"forward_implications":["Clinicians can pick an operating point—low threshold for triage, high threshold for confirmation—and the generated report will match the corresponding sensitivity–specificity trade-off.","Regulators can evaluate generated reports with ROC/AUC statistics in the same way diagnostic devices are assessed, since report behavior tracks the classifier's validated curve.","Fusing the classifier with the generator improves diagnostic accuracy over uncontrolled generation: sensitivity rises 6.9% at matched specificity and specificity rises 20.7% at matched sensitivity on MIMIC-CXR.","The framework is modular: swapping the classifier, report generator, or LLM rewriter changes performance predictably, with ablations identifying which configurations preserve conformance."],"supporting_citations":[{"why":"It supplies the MIMIC-CXR dataset, the source of all chest X-ray images and reference reports used for training and evaluation.","marker":"Johnson et al., 2019"},{"why":"It provides the pre-trained medical image-text encoder from which the classifier is fine-tuned.","marker":"Codella et al., 2024"},{"why":"It provides the automatic labeler whose labels serve as ground truth for training the classifier and computing ROC curves.","marker":"Jain et al., 2021"},{"why":"It provides the VQA-based report generator whose follow-up query capability supplies grounded evidence for rewritten sentences.","marker":"Jin et al."},{"why":"It provides the LLM used both as the rewriter that conforms reports to thresholded classes and as the label extractor for ROC evaluation.","marker":"Singh et al., 2025"},{"why":"It defines the 14 CheXpert disease classes that constitute the threshold-controllable label set.","marker":"Irvin et al., 2019"},{"why":"It supplies temperature scaling, the calibration step that makes confidence thresholds map predictably to operating points.","marker":"Guo et al., 2017"}],"fun_headline_variants":["Radiology reports get a sensitivity-specificity dial","Threshold control makes report generation ROC-verifiable","Flip a threshold, reports follow the classifier's ROC","RadFusion: tune report sensitivity like a classifier","One dial for radiology reports: match the ROC curve"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the automatic label-extraction tool applied to reference reports as ground truth, both for training the classifier and for computing every ROC curve; if that tool disagrees with clinical reality, the reported conformance and accuracy gains are measured against the tool's labels, not against truth.","fun_headline_variants_meta":{"raw":{"variants":["Radiology reports get a sensitivity-specificity dial","Threshold control makes report generation ROC-verifiable","Flip a threshold, reports follow the classifier's ROC","RadFusion: tune report sensitivity like a classifier","One dial for radiology reports: match the ROC curve"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1505,"prompt_tokens":995,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":611,"tokens_out":510,"duration_ms":4847,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:18:14.630143+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of chest X-ray images from a public dataset, have a panel of radiologists independently label the disease classes, and compute the threshold-controlled reports' ROC curve against those radiologist labels; if the report ROC no longer closely tracks the classifier ROC—for example, AUC falling several points behind on multiple classes—the claimed conformance would be falsified.","supporting_citations":[],"review_version":1}