{"id":"c9a17f4e-90cc-4bbf-a601-d48007ac98e6","arxiv_id":"2507.06937","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new public dataset of 144 critical retained foreign object chest X-rays is introduced, with benchmark results showing physics-based synthetic augmentation improves detector performance.","lead":"This paper releases Hopkins RFOs Bench, a new set of 144 chest X-ray images with retained surgical objects, plus benchmark results for four object detectors. It also tests whether synthetic X-rays can improve detection, finding physics-based simulation helps while diffusion-generated images do not.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that physics-based augmentation consistently improves critical-RFO detection is not statistically grounded: the critical test subset is roughly 29 images and Table 2 reports no confidence intervals or seed variability.","rationale":"The reader's weakest assumption was the small critical test split and the absence of confidence intervals; I agree and elevate this as the single load-bearing issue. The paper's central scientific contribution beyond the dataset itself is the comparison of physics-based versus diffusion-based synthetic augmentation, and that comparison is made on point estimates from what is effectively a few dozen positive test cases. This is not a disagreement with field consensus; it is an internal statistical-support gap: the reported improvements are too small relative to the test size to be distinguished from split noise. The DDPM comparison is additionally confounded by zero-shot use of RoentGen, but the statistical fragility already undercuts the positive physics-based claim. Conditional acceptance remains appropriate: the dataset and benchmark are valuable resources, but the synthetic-data conclusions need added rigor, including uncertainty estimates and repeated runs, before the headline claims can be taken at face value.","tokens_in":15284,"tokens_out":4814,"duration_ms":57632,"concrete_test":"Extract the exact test split from the released code; then run each Table 2 configuration (Base, +1000, +2000, +4000) for both synthetic pipelines under five random seeds. Bootstrap the held-out test set, stratified by class, to obtain 95% confidence intervals for Delta = Base+2000 minus Base for ACC, AUC, and FROC. If any interval contains zero, or fewer than three of five seeds show positive Delta across all models, the 'consistently improves' claim is not supported by the data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 describes a 70/10/20 patient-level split; under any consistent reading, the held-out set contains only about 29 critical RFO images. Table 2's physics-based improvements are therefore in the range of one to three test examples: for instance, FCOS ACC goes 75.1 to 76.4, Faster-RCNN FROC goes 50.5 to 54.3, and the Base+2000 versus Base+4000 differences are even smaller. The paper reports a single split, a single training run per configuration, and no bootstrap or confidence intervals. Consequently the central 'consistently improves across all metrics' claim in Section 6.2, and the contrasting DDPM conclusion in Section 6.3, rest on point estimates whose rankings could easily reverse under resampling or retraining. The dataset contribution may survive this concern, but the most actionable finding—the physics-based synthetic augmentation advantage—does not yet support the strength of the stated conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Hopkins RFOs Bench, a retrospectively collected dataset of 144 chest X-ray images containing critical retained foreign objects (RFOs) from the Johns Hopkins Health System, together with image- and object-level annotations. The authors benchmark four object detectors and compare two synthetic-data augmentation strategies: DeepDRR-RFO, a physics-based pipeline, and RoentGen-RFO, a diffusion-based pipeline. The central claims are that the dataset is the first publicly available resource dedicated to critical RFOs and that physics-based synthetic augmentation consistently improves detection performance across all reported metrics, while DDPM-based augmentation does not.","tokens_in":15404,"tokens_out":5171,"duration_ms":53939,"significance":"If the dataset is released as described, it would fill a real gap: the public RFO datasets listed in Table 1 contain only non-critical objects. The curation process is described carefully, including IRB approval, radiologist second reads, and patient-level data splits, and the authors commit to open release of code and (upon acceptance) data. The paper also provides a useful side-by-side comparison of physics-based and diffusion-based synthetic augmentation. However, the central quantitative claim is not yet statistically grounded: the held-out critical test set is small, Table 2 reports no confidence intervals or seed variability, and the same baseline condition is reported with different values in Table 2 and Table 3. These issues must be resolved before the benchmark results can be relied upon.","major_comments":[{"comment":"The held-out critical test set contains roughly 29 images (20% of 144), and Table 2 reports only single point estimates with no confidence intervals, bootstrap resampling, or multiple-seed variability. The observed improvements are small in absolute terms (e.g., FCOS ACC 75.1 to 76.4 and FROC 52.0 to 55.1; Faster-RCNN FROC 50.5 to 54.3), so a change of one or two test cases could alter the rankings. The Section 6.2 conclusion that physics-based augmentation 'consistently improves model performance across all metrics' is therefore not statistically supported as stated.","section":"Section 3 and Section 6.2, Table 2"},{"comment":"The baseline condition is reported inconsistently across the manuscript: Table 2 'Base' for Faster-RCNN gives ACC 74.0, AUC 0.62, FROC 50.5, while Table 3 'Hopkins RFOs Bench' gives ACC 74.3, AUC 0.73, FROC 49.8; FCOS likewise differs (ACC 75.1 vs 71.4, AUC 0.61 vs 0.67). Since these appear to describe the same training condition, the discrepancy must be reconciled before any augmentation comparison is interpretable.","section":"Table 2 vs Appendix G, Table 3"},{"comment":"The task definition in Section 4.1 requires object-level predictions with a category ci ∈ {non-critical, critical}, but Appendix D describes binary training that assigns 'class label 1 for all RFOs' and a classification head that distinguishes images with and without RFOs. It is therefore unclear whether any baseline model is trained or evaluated to separate critical from non-critical objects. Because the paper's stated focus is critical RFO detection, the label space used in the benchmark and the metrics reported for critical-object localization must be clarified, or the benchmark should be described as any-RFO detection.","section":"Section 4.1 and Appendix D"},{"comment":"The authors acknowledge in Section 7 that synthetic RFOs 'often appear with lower image resolution and are overly contrasted against the surrounding anatomical background,' and the Limitations paragraph states that synthetic images 'may still lack certain clinical subtleties found in real data.' These statements qualify the generalizability of the Table 2 gains, since the improvement may be driven by high-contrast synthetic objects rather than by features that transfer to subtle real-world critical RFOs. The discussion should temper the 'consistently improves' claim, or provide per-type or difficulty-stratified evidence that the gains are not an artifact of synthetic object appearance.","section":"Section 7 and Limitations paragraph"}],"minor_comments":[{"comment":"The text and figure captions use 'No-critical' where 'non-critical' is intended; please correct this terminology consistently.","section":"Section 3 and Figure 2"},{"comment":"The Abstract and Section 3 state that the dataset is publicly available/open at GitHub, while Appendix B.1 says the dataset 'will be made fully and publicly available upon acceptance of the corresponding manuscript'; please reconcile these availability statements.","section":"Abstract and Appendix B.1"},{"comment":"Table 3 is captioned as results on 'different synthetic datasets,' but the rows describe real-data training settings (Object-CXR, Hopkins RFOs Bench, and pretrain-plus-finetune); the caption should be corrected.","section":"Table 3 caption"},{"comment":"The last line of Appendix E, 'The rndring volume of each RFO are open access on our Hopkins RFO Bench,' contains a typo and unclear wording; please revise it.","section":"Appendix E"},{"comment":"The definition of DDPMs is given twice in Section 2.3 ('iteratively denoising random inputs' and 'iteratively denoising random noise'); please remove the redundancy.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is valuable and likely within scope, but the benchmark's statistical grounding and internal consistency need substantial work before publication. I would also verify with the editor that the data-sharing timeline is clearly stated, since the 'first publicly available' claim depends on release timing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the dataset is real and fills a genuine gap: Hopkins RFOs Bench is the first public chest X-ray collection dedicated to critical retained foreign objects, with 144 images, radiologist annotations, and a class-balanced set that includes no-RFO and non-critical cases. Second, the paper's headline result—that physics-based synthetic augmentation consistently helps—rests on point estimates from a small test set, so treat the rankings as preliminary.\n\nThe curation is done properly: IRB approval, keyword screening of 50,000+ studies, radiologist review, second read, and a clear annotation protocol. The benchmark is straightforward but useful, and the synthetic comparison is a reasonable first look at two very different augmentation strategies. I also credit the authors for stating limitations plainly: they acknowledge the single-institution source, the small size, and that synthetic RFOs look overly contrasted.\n\nThe soft spot is the statistics. With a 70/10/20 split, the held-out critical set is roughly 29 images. Table 2 reports ACC, AUC, and FROC without confidence intervals, from a single split and a single run per configuration. Differences like Faster-RCNN FROC going from 50.5 to 54.3 come down to a handful of test cases. The \"consistently improves across all metrics\" claim in Section 6.2 is stronger than the evidence supports. The DDPM conclusion is also somewhat overgeneralized: RoentGen was used zero-shot without fine-tuning, so \"diffusion doesn't help\" is really \"this off-the-shelf model, prompted, didn't help.\" Both issues are fixable—add bootstrapped CIs, multiple seeds, and either fine-tune RoentGen or soften the claim.\n\nWho is this for? Groups working on rare finding detection in medical images, or using synthetic data for augmentation. The dataset alone justifies referee time. The benchmark numbers will be superseded, but that is fine for a resource paper.\n\nRecommendation: send it to peer review. The dataset contribution is valuable enough that the statistical weakness, while real, is a revision issue rather than a rejection issue.","headline":"The dataset is the real contribution; the synthetic augmentation comparison is useful but rests on point estimates from a tiny test set, so the headline claim needs statistical grounding before this is final.","tokens_in":16009,"tokens_out":1600,"would_cite":true,"duration_ms":17541,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Hopkins RFOs Bench, the first publicly available chest X-ray dataset dedicated to critical retained foreign objects, and shows that physics-based synthetic X-ray augmentation consistently improves detector…","keywords":["retained foreign objects","critical RFOs","chest X-ray","object detection benchmark","physics-based synthetic X-rays","diffusion models","dataset release","medical imaging AI"],"falsifier":"Train the four detectors with and without +2,000 physics-based synthetic images exactly as described, but evaluate on a separate external test set of 100 or more critical-RFO chest X-rays from another institution; if the synthetic-augmented models do not beat the real-data baseline on AUC and FROC by a margin larger than the run-to-run variance across several random seeds, the paper's central empirical claim would be undermined.","tokens_in":15063,"feed_emoji":"🩻","tokens_out":3546,"duration_ms":40392,"temperature":0.7,"pith_summary":"Critical retained foreign objects (RFOs) such as surgical sponges and needles are rare but life-threatening \"never events,\" and AI detection has been stymied by the lack of public datasets that actually contain them. This paper builds Hopkins RFOs Bench, a curated set of 144 chest X-rays with critical RFOs collected over 18 years, plus 150 no-RFO and 150 non-critical-RFO images, all with radiologist annotations. It benchmarks four object detectors and finds that augmenting training with physics-based simulated X-rays (DeepDRR-RFO) consistently improves accuracy, false-negative rate, AUC, and localization (FROC), with the best results at 2,000 added synthetic images. In contrast, adding diffusion-generated X-rays (RoentGen-RFO) generally degrades performance. If correct, the paper supplies the first public resource for training and evaluating critical-RFO detectors and gives concrete evidence about which synthetic-data strategy works for this rare-object problem.","feed_headline":"Physics-based synthetic X-rays boost critical RFO detection","feed_subtitle":"A 144-image open benchmark shows simulated radiographs help detectors, while diffusion-generated ones do not.","key_machinery":"The load-bearing machinery is the dataset itself plus two synthetic-generation pipelines that are directly compared. DeepDRR-RFO is a physics-based pipeline that segments CT volumes into air, soft tissue, and bone, reconstructs 3D models of real surgical items from single photographs, embeds them into the CT volumes, and simulates X-ray formation with material-specific attenuation to produce automatically annotated radiographs. RoentGen-RFO is a diffusion-based pipeline that adapts a pretrained DDPM through carefully designed text prompts, without fine-tuning on RFO images, to generate synthetic chest X-rays containing critical foreign objects. The comparison of these two pipelines, evaluated with the same detectors and metrics on the same held-out test set, carries the paper's central argument about which synthetic-data strategy is effective for rare critical findings.","core_discovery":"Hopkins RFOs Bench is claimed to be the first publicly available dataset dedicated explicitly to critical RFO cases, containing 144 critical-RFO chest X-ray images from distinct patients, annotated at both image level and object level with bounding boxes or polygons and a critical/non-critical label. On this benchmark, all four tested detectors (Faster R-CNN, FCOS, RetinaNet, YOLO) improve substantially when pretrained on the non-critical Object-CXR dataset and fine-tuned on Hopkins RFOs Bench, e.g., Faster R-CNN's AUC rises from 0.62 to 0.80. When additional physics-based synthetic radiographs are added to the training set, every model improves across ACC, FNR, AUC, and FROC, peaking at 2,000 synthetic images (e.g., RetinaNet reaches 79.5% ACC, 0.23 FNR, 0.78 AUC, 63.5 FROC) and declining slightly at 4,000. Diffusion-based synthetic images, by contrast, generally reduce performance, illustrating the authors' conclusion that physically grounded simulation currently offers more useful training data for critical RFO detection than zero-shot diffusion generation.","pith_inferences":["An implication the paper leaves implicit is that the reported performance gains rest on a small test set: with a 70/10/20 split of 144 critical cases, the critical test partition contains roughly 29 images, so the superiority of +2,000 physics-based images over +1,000 or +4,000 could shift with a few difficult cases; the authors report no confidence intervals or multi-seed variance.","A testable extension is to evaluate the trained detectors on external chest X-rays from another institution to see whether the physics-based synthetic gains transfer beyond the single health system used to build the benchmark.","The physics-based pipeline's acknowledged limitation that synthetic RFOs appear overly contrasted suggests a concrete improvement: applying appearance randomization or domain adaptation to the synthetic images before training, then measuring whether gains persist on subtle real-world objects.","Because the dataset includes non-critical RFOs and no-RFO images alongside critical cases, it could support multi-class training and calibrated screening workflows, not just binary critical-versus-absent detection."],"forward_implications":["Researchers gain a public, IRB-approved benchmark with image- and object-level annotations, enabling direct comparison of future critical-RFO detection models.","Physics-based synthetic augmentation at moderate scale (around 2,000 images) is a viable strategy to mitigate extreme data scarcity for rare radiological findings.","Diffusion-based synthetic data, at least in zero-shot form, is not yet useful for training critical-RFO detectors and may harm performance by introducing distribution shift.","Pretraining on a large non-critical RFO dataset (Object-CXR) followed by fine-tuning on a small critical-RFO dataset is an effective transfer recipe, at least on this benchmark.","The same benchmark can be used to fine-tune diffusion models on real critical-RFO examples, potentially improving their ability to generate useful synthetic cases."],"supporting_citations":[{"why":"The Object-CXR dataset is the main existing public RFO-containing dataset (non-critical objects) and is used for pretraining and as a comparison baseline.","marker":"[14]"},{"why":"The DeepDRR physics-based X-ray simulator is the foundation of the DeepDRR-RFO synthetic pipeline.","marker":"[32]"},{"why":"RoentGen, a diffusion-based chest X-ray generator, is adapted into RoentGen-RFO for the DDPM synthetic arm.","marker":"[4]"},{"why":"TotalSegmentator performs the CT volume segmentation (air, soft tissue, bone) used in the physics-based synthetic pipeline.","marker":"[37]"},{"why":"TripoSR reconstructs 3D models of real surgical items from single photographs, providing the RFO shapes embedded into CT volumes.","marker":"[29]"},{"why":"This prior deep-learning study for detecting retained surgical items defines the clinical task context and motivates the need for critical-RFO datasets.","marker":"[17]"}],"fun_headline_variants":["Physics-based synthetic X-rays outperform diffusion for RFO detection","Hopkins RFO Bench: 144 images show simulated X-rays boost detectors","First critical RFO dataset reveals physics sim beats diffusion for detection","Simulated radiographs help AI spot retained objects; diffusion doesn't","Physics-based X-ray synthesis improves critical RFO detection AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative conclusions rest on a small held-out test set (about 29 critical-RFO images) with no reported confidence intervals or multiple training runs, so a few hard images could change the ranking of training strategies, and the synthetic RFOs themselves are acknowledged to be overly contrasted.","fun_headline_variants_meta":{"raw":{"variants":["Physics-based synthetic X-rays outperform diffusion for RFO detection","Hopkins RFO Bench: 144 images show simulated X-rays boost detectors","First critical RFO dataset reveals physics sim beats diffusion for detection","Simulated radiographs help AI spot retained objects; diffusion doesn't","Physics-based X-ray synthesis improves critical RFO detection AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1463,"prompt_tokens":1040,"completion_tokens":423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":335}},"tokens_in":656,"tokens_out":423,"duration_ms":4494,"temperature":1.0,"reasoning_tokens":335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:50:24.612059+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the four detectors with and without +2,000 physics-based synthetic images exactly as described, but evaluate on a separate external test set of 100 or more critical-RFO chest X-rays from another institution; if the synthetic-augmented models do not beat the real-data baseline on AUC and FROC by a margin larger than the run-to-run variance across several random seeds, the paper's central empirical claim would be undermined.","supporting_citations":[{"cited_title":"Object-cxr: Automatic detection of foreign objects on chest x-rays, 2020","cited_arxiv_id":null,"evidence_quote":"The Object-CXR dataset is the main existing public RFO-containing dataset (non-critical objects) and is used for pretraining and as a comparison baseline."},{"cited_title":"Deepdrr–a catalyst for machine learning in fluoroscopy-guided procedures","cited_arxiv_id":null,"evidence_quote":"The DeepDRR physics-based X-ray simulator is the foundation of the DeepDRR-RFO synthetic pipeline."},{"cited_title":"A deep learning model based on fusion images of chest radiography and x-ray sponge images supports human visual character- istics of retained surgical items detection","cited_arxiv_id":null,"evidence_quote":"This prior deep-learning study for detecting retained surgical items defines the clinical task context and motivates the need for critical-RFO datasets."}],"review_version":1}