{"id":"0455f47e-666c-4b5b-a561-d96276a178c8","arxiv_id":"2411.18672","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A modular query-code-update pipeline reduces measurement hallucinations in chest X-ray reports by replacing model-generated numbers with measurements from specialized vision tools.","lead":"FactCheXcker is a tool pipeline that corrects wrong numbers, such as endotracheal tube distances, in AI-generated chest X-ray reports by converting measurements into code that calls specialized vision models. It improved measurement accuracy in 10 of 11 report-generation models on the MIMIC-CXR dataset, but the evaluation has gaps that limit how strongly the result can be trusted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline '135.0% average improvement' is inflated by an inverted denominator: Table 4 reports (Original−Updated)/Updated instead of (Original−Updated)/Original; recomputed average is ≈43%.","rationale":"The reader identified a plausible concern about the private CarinaNet+ module and evaluation transparency, but the single most load-bearing issue is more concrete and internal: the paper's headline improvement metric is computed with the post-intervention value in the denominator. The reader's rationale mentions a 'nonstandard and easy-to-misread improvement metric' without pinpointing the inversion. This matters because the Abstract and conclusion rest on '135.0% average improvement', which is not a relative reduction from the original error. Recomputed correctly, the average improvement is approximately 43%, and RadFM remains a negative case (−38%). The claim that 10/11 models improve still holds, so rejection is not warranted; however, the quantitative headline must be corrected and re-stated. This is an internal inconsistency (Section 4.4 defines MAE but never defines the improvement percentage), not a disagreement with external consensus. The conditional verdict remains appropriate: the paper needs a corrected metric, per-model confidence intervals, and a clearer separation of the LLM orchestration from the underlying measurement modules before the headline claims can be accepted as written.","tokens_in":18222,"tokens_out":10064,"duration_ms":91494,"concrete_test":"Recompute every 'Improvement' entry in Table 4 and the Abstract's 135.0% using (Original − Updated)/Original with the exact MAE and Composite values already reported. Verify that the reported 191%, 503%, 1024%, etc. are (Original − Updated)/Updated. If the corrected average falls to ≈43%, the headline claim must be revised; also re-run the CheXpert Plus supplementary results with the corrected formula.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 4's 'Improvement' column is not a standard relative reduction from the original error. For CheXagent, MAE goes 1.98 → 0.68; the paper reports 191.0%, which equals (1.98−0.68)/0.68, not (1.98−0.68)/1.98 = 65.7%. The same pattern holds across rows: GPT-4V 2.35→0.39 is reported as 503% rather than 83.4%; LLM-CXR composite 18.54→1.65 is reported as 1024% rather than 91.1%. The Abstract's 'average improvement of 135.0%' is the mean of these inverted ratios. Recomputing each row with the standard formula (Original−Updated)/Original gives per-model reductions of roughly 66, 54, 77, 83, 62, 31, 20, 52, 13, −60, and 72 percent, averaging ≈43%, not 135%. The '10/11 models improve' claim survives, but the headline quantitative claim is overstated by a factor of three. This is a correctness issue in the central evidence, not a stylistic choice: the paper explicitly calls it 'relative performance gain', and the Abstract uses the inflated number.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FactCheXcker, a modular query-code-update pipeline that corrects measurement hallucinations in chest X-ray report generation. Given a model-generated report and the chest X-ray, FactCheXcker generates measurement queries, writes and executes Python code against an API implemented by fine-tuned tool modules (ResNet-50+ for ETT presence, CarinaNet+ for ETT/carina coordinates, TorchXRayVision for anatomy), and uses a Report Updater to revise the report. The method is evaluated on the MIMIC-CXR test set for endotracheal tube placement with 11 report-generation models, reporting presence precision, measurement MAE, a composite MAE/F1 score, and placement precision. The authors claim that FactCheXcker improves 10 of 11 models and achieves an average MAE improvement of 135.0%.","tokens_in":18475,"tokens_out":11384,"duration_ms":87294,"significance":"If the findings hold, the framework offers a practical, model-agnostic approach to reducing quantitative errors in generated radiology reports, with no retraining of the original model and a modular design that can be extended to other measurement tasks. The evaluation across 11 models is a substantial strength, and the paper ships code. The central direction is credible, but the headline quantitative claim is miscalculated: the reported improvement percentages use an inverted denominator, and the average is a pooled ratio rather than a per-model mean. Several internal inconsistencies in the reported metrics also need to be resolved before the paper's core evidence can be accepted.","major_comments":[{"comment":"The 'Improvement' column in Table 4 is not the standard relative reduction from the original error. For CheXagent, the reported 191.0% equals (1.98−0.68)/0.68, whereas the standard relative reduction is (1.98−0.68)/1.98 = 65.7%. The same inverted denominator is used in every row, and the Abstract's 135.0% is the pooled ratio (1.93−0.82)/0.82 rather than an average of per-model improvements; recomputing each row with the standard formula gives a mean of approximately 43%. The 10-of-11 improvement claim survives, but the headline quantitative claim in the Abstract, Section 5.3, Table 4, Table 8, and Table 10 is overstated by roughly a factor of three. Please state the formula for improvement, report standard relative reductions, and clarify whether pooled or per-model averaging is intended.","section":"Abstract and Table 4"},{"comment":"The reported original MAE for RadFM (0.65 cm) is inconsistent with the original failure rate (44.4%) shown in Figure 4. With a failure threshold of 1.5 cm, the MAE must be at least 0.444 × 1.5 = 0.666 cm if all failed cases have errors just above the threshold and all non-failed cases have zero error. Additionally, the updated MAE worsens from 0.65 to 1.04 while the failure rate drops from 44.4% to 11.1%, which is difficult to reconcile. Please specify the exact case populations used for MAE and failure rate (e.g., all studies with ground-truth ETT measurement versus only those with a model-generated measurement) and recompute.","section":"Table 4 and Figure 4"},{"comment":"The MAE definition sets missing model measurements to 0, but the paper does not state over how many cases each model's MAE is averaged, nor how the 45 radiologist-annotated ground-truth cases are distributed among models. This matters because CarinaNet+, the module that provides the corrected measurement, has a reported MAE of 0.94 cm on the MIMIC-CXR test set (Table 2), yet several updated model MAEs (e.g., GPT4V 0.39, CheXpertPlus 0.66) are lower than this. The reader cannot determine whether these numbers are computed on different subsets or whether the pipeline sometimes retains the original (inaccurate) measurement. Please report the case counts and explain the relationship between CarinaNet+'s MAE and the updated MAE values.","section":"Section 4.4 and Table 2"},{"comment":"The claim that FactCheXcker 'maintains the quality of the original reports' is not supported by any evaluation. The paper evaluates ETT presence, measurement, and placement, but does not measure readability, clinical consistency, or preservation of non-ETT findings in the updated reports. The Report Updater prompt instructs the model to retain formatting and tone, but this is not a measurement. Please add a quantitative or human evaluation of updated-report quality, or revise the claim to reflect that only measurement-focused metrics were assessed.","section":"Abstract and Section 3.2"},{"comment":"The asterisks in Figure 3 indicate statistical significance with p<0.1, but the test is not described. It is unclear whether the comparisons are paired across the same studies, which test (e.g., Wilcoxon signed-rank) was used, and whether any multiple-comparison correction was applied across the 11 models. A threshold of p<0.1 is also unusually lenient for a claim of 'significant' improvement. Please specify the statistical procedure or remove the significance claims.","section":"Section 5.3 and Figure 3"}],"minor_comments":[{"comment":"The text states that LLM-CXR shows the most significant precision improvement 'with an increase of 1024%,' but the 1024% figure in Table 4 is the composite-score improvement; the precision improvement is from 0.08 to 0.58 (roughly a six-fold relative increase). Please correct this misattribution.","section":"Section 5.2"},{"comment":"The header 'Presense' is a typo for 'Presence'.","section":"Table 4"},{"comment":"The Composite metric MAE/F1 is unusual because a low F1 amplifies MAE; its interpretation and choice should be justified, and the improvement percentages for it suffer from the same denominator issue as the MAE improvements.","section":"Section 4.4"},{"comment":"The failure threshold of 1.5 cm is introduced without clinical justification or sensitivity analysis; please cite a source or provide a sensitivity analysis.","section":"Section 4.4 and Figure 4"},{"comment":"The assumption that a report with an ETT mention but no placement detail is 'correct' biases placement precision upward; please quantify this effect or use a more neutral handling.","section":"Section 4.4"},{"comment":"The fine-tuned CarinaNet+ uses a private dataset from 22 hospitals that is not released; given that this module is the measurement engine of the pipeline, a public version or detailed protocol would improve reproducibility.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the evaluation effort is substantial, but the headline improvement metric is miscalculated and there are unresolved internal inconsistencies (notably RadFM's MAE versus failure rate). These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The private CarinaNet+ dataset is a reproducibility concern; I encourage the authors to release model weights or a detailed validation protocol. The inclusion of the authors' own MedVersa and CheXagent baselines is not problematic, but independent implementations would strengthen the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful engineering contribution. FactCheXcker takes a model-generated chest X-ray report, generates measurement queries via an LLM, writes code against a modular API to invoke specialized vision modules, and rewrites the report with verified numbers. That is a real post-processing tool that doesn't require retraining. The evaluation across 11 models on ETT placement is broad, and the consistent reduction in MAE (10/11 models) is believable and internally consistent.\n\nBut the headline claim needs a correction. The 'relative performance gain' in Table 4 is computed as (Original − Updated)/Updated, not the usual (Original − Updated)/Original. For CheXagent that turns a 65.7% MAE reduction into '191.0%'; the abstract's 'average improvement of 135.0%' is the mean of these inverted ratios and corresponds to roughly 43% average relative reduction. That is a three-fold overstatement of the central result. The direction and qualitative conclusion survive, but the number should be reported as ~43% or as a ratio (e.g., updated error is 2.4× smaller) with the formula stated.\n\nOther soft spots: the CarinaNet+ module is fine-tuned on a private 1,100-image dataset from 22 hospitals, and neither weights nor data are released. That makes the measured accuracy hard to check; the pipeline's ceiling is exactly that module's accuracy, so replication risk is real. No confidence intervals or per-model sample sizes are reported, though the violin plots in Figure 3 suggest many of the differences are real. The composite metric MAE/F1 is also nonstandard and not clearly motivated.\n\nOn the positive side, the paper draws the right citations (ViperGPT, FacTool, etc.), the API is clean, and the code is public. The authors are also transparent about the rule-based report updater and the failure cases. This is not circular: the updated measurements come from an external module trained independently of the report generation models.\n\nBottom line: this deserves peer review, but the 135% figure needs to be corrected before publication, and the authors should provide error bars or at least per-model sample sizes. If you read it, skip the abstract and look at Table 4 plus Figure 3.","headline":"Useful framework, but the 135% improvement headline is miscomputed—standard relative reduction is about 43%.","tokens_in":19030,"tokens_out":3064,"would_cite":false,"duration_ms":24899,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Measurement hallucinations in chest X-ray reports can be corrected post-hoc by a modular pipeline that re-measures the image and rewrites the report.","keywords":["measurement hallucination","radiology report generation","chest X-ray","endotracheal tube placement","query-code-update","tool-augmented LLM","MIMIC-CXR","hallucination mitigation"],"falsifier":"Replace CarinaNet+ with any publicly available carina and tube-tip localizer, run FactCheXcker on the MIMIC-CXR test set, and check whether the updated reports still beat the original models on mean absolute error; if the improvement disappears, the 135% reduction is an artifact of the unreleased private module.","tokens_in":17995,"feed_emoji":"🩻","tokens_out":9114,"duration_ms":72396,"temperature":0.7,"pith_summary":"FactCheXcker is a modular pipeline that corrects measurement hallucinations in radiology reports after they are generated, without retraining or modifying the underlying vision-language model. The framework reads a model-written report, turns any quantitative claim into a measurement query, writes and executes Python code against a domain-specific API backed by specialist vision tools, and then rewrites the report with the verified numbers. On the MIMIC-CXR dataset, applying FactCheXcker to endotracheal tube placement improved 10 of 11 report generation models, reducing measurement error by an average of 135% and cutting clinically significant errors (over 1.5 cm) roughly in half, while keeping the original reports' wording and quality intact. The paper positions this as a general prescription: specialized modules can de-hallucinate any report generator, and each new measurable object only requires adding a tool module.","feed_headline":"One add-on fixes wrong measurements in 10 of 11 X-ray report models","feed_subtitle":"It re-measures the breathing tube's position on chest X-rays and rewrites the report, cutting average error by 135%.","key_machinery":"The load-bearing mechanism is the query-code-update loop over a domain-specific API. A query generator (GPT-4o mini) converts report sentences into structured measurement requests, such as \"measure the distance between the endotracheal tube and the carina.\" A code generator writes Python functions that call API methods—exists, find, distance, width, diameter, dimensions, within—each implemented by swappable specialist tools: a fine-tuned ResNet-50+ for tube presence, a fine-tuned CarinaNet+ that outputs carina and tube-tip coordinates, chest anatomical segmentation models for regions such as heart and lungs, and the image's pixel-spacing metadata to convert pixel distances into centimeters. A report updater then merges the executed results back into the text, deleting hallucinated objects and flagging misplaced tubes using a rule-based interpretation of the 3–7 cm \"correct placement\" guideline.","core_discovery":"The paper's central claim is that measurement hallucinations in chest X-ray report generation are not an unavoidable failure of the report model but a separate, correctable output defect. FactCheXcker demonstrates this by decomposing measurement into sub-tasks—deciding whether an object is present, localizing it, locating anatomical landmarks, converting pixels to centimeters, and judging clinical placement—and handing each sub-task to a specialized module. The strongest quantitative evidence is on endotracheal tube placement: across 11 models, the framework raises ETT detection precision from 0.49 to 0.65, lowers measurement mean absolute error from 1.93 cm to 0.82 cm (a 135.0% average improvement), improves the composite MAE/F1 score by 186%, and raises placement-correctness precision from 0.84 to 0.94. The authors also report that this is achieved while retaining the style and readability of the original report.","pith_inferences":["If the unreleased CarinaNet+ weights and its 22-hospital training data were made public, the framework could be independently replicated and stress-tested on new institutions; without them, the 135% figure cannot be separated from the quality of that one private module.","The evaluation metric penalizes a missing measurement as a zero (yielding an error equal to the ground-truth value), so a fraction of the measured improvement comes from deleting hallucinated ETT mentions rather than correcting numbers; a per-case decomposition of deletion versus correction would clarify the mechanism.","A natural next experiment is to apply the same loop to lesion size measurement or cardiothoracic ratio, where the tool modules are weaker today; the paper predicts the framework would still reduce errors, but only if a reliable segmenter exists."],"forward_implications":["Any existing report generation model can be de-hallucinated post-hoc without weight access, making the fix applicable to closed and open models alike.","The pipeline's measurement accuracy is upper-bounded by its best tool module; improving modules such as CarinaNet+ directly improves all downstream reports.","The framework, in principle, extends beyond endotracheal tubes to lesions, catheters, pneumothorax and other measurable findings by adding new tool modules to the same API.","Adding a reliable detector for every image (not just reports that mention an ETT) would reduce false negatives, but the paper's ablation shows precision drops, so production use needs a better presence detector or confidence gating.","For already-accurate models such as MedVersa and RaDialog the gains are modest (15–32% MAE improvement), so the marginal value of correction is largest for weak generators."],"supporting_citations":[{"why":"Supplies the MIMIC-CXR images and reports used to develop and test the pipeline.","marker":"[17]"},{"why":"CarinaNet is the base model fine-tuned into CarinaNet+ that localizes the carina and endotracheal tube tip.","marker":"[29]"},{"why":"Provides the clinical guideline (correct ETT placement at 5 ± 2 cm above the carina) used by the report updater.","marker":"[10]"},{"why":"GPT-4o mini is the large language model that generates measurement queries and executable code.","marker":"[14]"},{"why":"ImageNet-pretrained ResNet-50 is the backbone fine-tuned for the ETT presence module.","marker":"[13]"},{"why":"Provides the chest anatomical segmentation maps used by the API's region-based methods.","marker":"[7]"}],"fun_headline_variants":["Cut X-ray measurement errors by 135% with this add-on","This tool fixes wrong measurements in 10 of 11 X-ray report models","X-ray report measurement errors drop 135% with this add-on","One add-on fixes chest X-ray measurements in 10 of 11 models","How this add-on cuts X-ray report measurement errors by 135%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire correction rests on CarinaNet+, a model fine-tuned on a private 1,100-image dataset from 22 hospitals whose weights are not released; if its localization accuracy does not transfer to new hospitals or to the specific images where the baseline models hallucinate, the reported improvement will not replicate.","fun_headline_variants_meta":{"raw":{"variants":["Cut X-ray measurement errors by 135% with this add-on","This tool fixes wrong measurements in 10 of 11 X-ray report models","X-ray report measurement errors drop 135% with this add-on","One add-on fixes chest X-ray measurements in 10 of 11 models","How this add-on cuts X-ray report measurement errors by 135%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001146,"raw_usage":{"total_tokens":4751,"prompt_tokens":940,"completion_tokens":3811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":3715}},"tokens_in":556,"tokens_out":3811,"duration_ms":23083,"temperature":1.0,"reasoning_tokens":3715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:04:24.908131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace CarinaNet+ with any publicly available carina and tube-tip localizer, run FactCheXcker on the MIMIC-CXR test set, and check whether the updated reports still beat the original models on mean absolute error; if the improvement disappears, the 135% reduction is an artifact of the unreleased private module.","supporting_citations":[{"cited_title":"Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports","cited_arxiv_id":null,"evidence_quote":"Supplies the MIMIC-CXR images and reports used to develop and test the pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CarinaNet is the base model fine-tuned into CarinaNet+ that localizes the carina and endotracheal tube tip."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the clinical guideline (correct ETT placement at 5 ± 2 cm above the carina) used by the report updater."}],"review_version":1}