{"id":"914e311f-fcd5-4a43-9672-f46a0786ecc1","arxiv_id":"2507.17508","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across six X-ray baggage datasets, no detector family dominates, and X-ray-specific modules attached to YOLOv8 consistently reduce accuracy.","lead":"This paper compares ten deep learning object detectors on six public X-ray security screening datasets, measuring both accuracy and speed. It finds that no single detector family wins on every benchmark, and that simple YOLOv8 often outperforms X-ray-specific add-on modules.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that X-ray-specific modules (CHR/DOAM/LIM) consistently underperform the YOLOv8 baseline rests on confounded training configurations; matched-optimizer re-runs are needed before the 'architectural disharmony' conclusion can stand.","rationale":"The reader's weakest-assumption identification is exactly the load-bearing point: per-dataset training configurations are not held constant across the detectors being compared. The paper's headline finding that no single architecture class dominates is a negative claim and would likely remain true even with corrected hyperparameters, since Table 5 shows different winners per dataset. However, the paper's most actionable and literature-challenging conclusion is that X-ray-specific attention modules (CHR, DOAM, LIM) consistently degrade a modern YOLOv8 detector, attributed to 'architectural disharmony.' This claim is used to caution the field against domain-specific modules and to motivate future hybrid designs. Because the baseline and the custom-module detectors are trained with different optimizers and learning rates on three datasets, and because no variance information is provided, the observed gaps could plausibly result from suboptimal optimization of the modified architectures rather than from the modules themselves. The paper does provide code and weights, which is commendable and enables the proposed test, but reproducibility of the pipeline does not by itself remove the confound. The internal table inconsistencies (for example, PIDray overall numbers differing between Table 5 and Table 11, and the PIDray test-subset counts in Section 4.1 not matching Table 6) further weaken confidence, though they are secondary to the training-protocol confound. For these reasons, the reader's CONDITIONAL verdict is appropriate: the central negative claim is plausible, but the strong positive claim about architectural disharmony requires matched-configuration, multi-seed re-evaluation before it can be accepted.","tokens_in":35060,"tokens_out":6169,"duration_ms":64696,"concrete_test":"Re-run D(YOLOv8, CSPDarkNet53) and D(YOLOv8+CHR/DOAM/LIM, CSPDarkNet53) on OPIXray, EDS, and HiXray using a strictly identical training protocol (same optimizer, learning-rate schedule, number of epochs, augmentation, and at least three random seeds), and report mean and standard deviation of mAP50/mAP50:95. If any custom-module detector comes within one standard deviation of the baseline or wins on any dataset, the 'consistent under-performance' and 'architectural disharmony' findings in Section 5.1 are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.6 and Table 4 show that the YOLOv8 baseline and the custom-module detectors (D(YOLOv8+CHR/DOAM/LIM, CSPDarkNet53)) are trained with different optimizers and learning rates on three of six datasets: on OPIXray the baseline uses SGD (lr=0.01) while the custom detectors use AdamW (lr=0.000714); on EDS the baseline uses AdamW (lr=0.000714) while the custom detectors use SGD (lr=0.01); on HiXray the baseline uses SGD while CHR/LIM use AdamW and DOAM uses SGD. No hyperparameter search is reported and each configuration is run once with no seeds or error bars. The observed gaps that Section 5.1 interprets as 'architectural disharmony' (e.g., OPIXray 0.868 vs 0.835, EDS 0.547 vs 0.416, HiXray 0.845 vs 0.811) are therefore confounded with optimization choices and run-to-run variance. The central negative claim—that no single detector type wins everywhere—would likely survive, but the paper's more consequential positive finding that X-ray-specific modules consistently harm YOLOv8 is not demonstrated. The 'architectural disharmony' explanation is an untested interpretation, not an ablation result, so the load-bearing causal conclusion is unsupported by the current experimental design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic comparative evaluation of ten object detectors on six public X-ray security benchmarks (OPIXray, CLCXray, SIXray, EDS, HiXray, PIDray), spanning generic CNN, custom X-ray-specific CNN, generic transformer, and hybrid CNN-transformer architectures. It reports detection performance (mAP50 and mAP50:95), inference time, parameter count, and GFLOPS, and it derives conclusions about the relative merits of detector families. The central claim is that no single detector type or architecture class is clearly advantageous across all benchmarks (Section 5.1). A second, more specific finding is that the X-ray-specific modules CHR, DOAM, and LIM, when integrated into YOLOv8, consistently underperform the generic D(YOLOv8, CSPDarkNet53) baseline, an effect the paper attributes to 'architectural disharmony' (Section 5.1). The authors release evaluation code and model weights at a public repository.","tokens_in":35383,"tokens_out":4961,"duration_ms":45939,"significance":"If the results are reliable, this study would be a useful reference for practitioners and researchers in X-ray security screening, owing to its unusual breadth: six widely used benchmarks, ten detector configurations covering four architectural families, and a uniform set of detection and efficiency metrics. The public release of code and weights is a concrete reproducibility asset that should be credited. The central negative result—that no single detector type dominates—is plausible and consistent with the averages reported in Table 5. However, the more consequential positive claim, that X-ray-specific modules are consistently harmful to a modern YOLOv8 detector, is currently confounded by unequal training configurations, and the paper contains internal inconsistencies in the PIDray results that need to be resolved before the findings can be taken at face value.","major_comments":[{"comment":"The comparison between D(YOLOv8, CSPDarkNet53) and D(YOLOv8+CHR/DOAM/LIM, CSPDarkNet53) is confounded by per-dataset optimizer and learning-rate differences. On OPIXray the baseline uses SGD (lr=0.01) while the custom-module detectors use AdamW (lr=0.000714); on EDS the baseline uses AdamW while the custom detectors use SGD (lr=0.01); on HiXray the baseline uses SGD while CHR and LIM use AdamW and DOAM uses SGD. Since each configuration is run once with no reported seeds or error bars, the observed gaps (e.g., OPIXray 0.868 vs 0.835, EDS 0.547 vs 0.416, HiXray 0.845 vs 0.811) cannot be attributed to architectural merit. The 'architectural disharmony' claim in Section 5.1 is an untested interpretation, not an ablation result. Please re-run the affected configurations with matched optimizers and learning rates, or provide a sensitivity analysis showing that the conclusions are robust to optimization choices.","section":"Section 4.6 and Table 4"},{"comment":"The PIDray 'overall' results are inconsistent between Table 5 and Table 11 for identical detector configurations. For example, D(YOLOv8, CSPDarkNet53) is reported as 0.897/0.807 in Table 5 but 0.874/0.780 in Table 11; D(YOLOv8, HGNetV2) is 0.902/0.796 vs 0.880/0.767; D(RT-DETR, HGNetV2) is 0.835/0.720 vs 0.803/0.684; D(YOLOv8, Next-ViT-S) is 0.898/0.801 vs 0.842/0.736; and D(RT-DETR, Next-ViT-S) is 0.879/0.773 vs 0.858/0.746. These discrepancies affect the average values in Table 5 and the dataset-specific conclusions in Section 5.3; please reconcile the tables or state explicitly which set of numbers is authoritative.","section":"Table 5 and Table 11"},{"comment":"Several comparative claims are made without any variance estimate or significance test. For instance, D(YOLOv8, Next-ViT-S) is said to exhibit the 'best overall performance on average' with 0.813 vs 0.799 for the next-best configuration, and D(Co-DETR, Swin-B) is called 'superior in half of the datasets' when compared with D(YOLOv8, CSPDarkNet53). With a single run per configuration, differences of this magnitude can be within run-to-run noise. Please report multiple seeds with means and standard deviations, or at minimum state explicitly that the observed differences are not statistically tested and should be interpreted as indicative only.","section":"Section 5.1"}],"minor_comments":[{"comment":"The abbreviation 'HR' appears twice (e.g., 'YOLOv8, RT-DETR, HR, DOAM, and LIM were trained for 100 epochs' and 'HR, DOAM, and LIM were trained using AdamW') and should be corrected to 'CHR'.","section":"Section 4.6"},{"comment":"The GFLOPS entry for D(YOLOv8+DOAM, CSPDarkNet53) is a dash, which may be read as 'not applicable' or 'not measured'; please state explicitly whether this value was not computed and why.","section":"Table 12"},{"comment":"The dataset name is spelled 'HIXray' in Table 5 and in parts of the text (e.g., Section 5.1), while the rest of the paper uses 'HiXray'; please unify the spelling.","section":"Table 5 and Section 5.1"},{"comment":"In the description of DINO, 'builts upon the DETR model' should be 'builds upon the DETR model'.","section":"Section 4.2.3"}],"recommendation":"major_revision","confidential_remarks":"The authors' own hybrid detector, D(YOLOv8, Next-ViT-S), is reported as the best on average; this is a natural point of attention, but it is evaluated under the same protocol as the other baselines, so I do not see circular reasoning. The most urgent issue is the inconsistency between Tables 5 and 11 for PIDray, which must be resolved before the paper can be considered reliable. The confounded training configurations are fixable within the manuscript's scope by re-running with matched hyperparameters or by softening the causal claims. The paper fits an empirical evaluation venue well."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. It is a genuinely useful common-protocol benchmark across six public X-ray datasets, with code and weights released, and its central negative claim—no single detector family wins everywhere—probably survives scrutiny. But the paper's more eye-catching assertion, that X-ray-specific modules like CHR/DOAM/LIM hurt YOLOv8 due to 'architectural disharmony,' is not actually demonstrated; the relevant comparisons are confounded by different optimizers.\n\nWhat is actually new: ten detector configurations run under one protocol, with per-class, per-size, per-subset, and cross-domain results not available previously in the literature. The efficiency analysis and the object-size/material-density breakdowns are useful. Releasing code and model weights is real support for reproducibility, and the survey of methods and datasets is thorough. That portion is solid work.\n\nThe soft spots are concentrated in the interpretation. First, Table 5 and Table 11 disagree on PIDray overall numbers for identical configurations: the YOLOv8 baseline appears as 0.897/0.807 in Table 5 but 0.874/0.780 in Table 11. That looks like a reporting error, but it needs to be fixed. Second, no error bars or multiple seeds are reported anywhere, so we cannot tell which gaps are stable.\n\nThird, and most important, the custom-module comparison is confounded. Table 4 shows that the YOLOv8 baseline and the CHR/DOAM/LIM variants use different optimizers and learning rates on three of six datasets. On OPIXray the baseline gets SGD with lr=0.01 while the custom modules get AdamW with lr=0.000714; on EDS the assignment is reversed; on HiXray it is mixed within the custom group. The gap the authors interpret as 'architectural disharmony' could easily be an optimizer artifact. The stress-test note has this right: the causal conclusion is an untested interpretation, not an ablation result.\n\nThe paper's general negative claim—that no single detector type dominates across all benchmarks—is well supported by the rank patterns and would survive a matched-optimizer rerun. The more specific message that domain-specific modules are systematically harmful, and the speed penalty claims, need the matched re-run before they can stand.\n\nThe SIXray/SIXray-D naming is disclosed, so that is fine. The 'HR' typo for CHR in Section 4.6 is cosmetic.\n\nFor a researcher in X-ray screening, this is a useful reference with practical guidance per dataset. I would send it to peer review, ask for the PIDray correction, seeds, and a matched-optimizer or at least sensitivity analysis, and let the 'architectural disharmony' conclusion be softened. The benchmark contribution deserves referee time.","headline":"Useful common-protocol benchmark with released code, but the 'architectural disharmony' claim is confounded by per-dataset optimizer choices and a PIDray table inconsistency.","tokens_in":35868,"tokens_out":3567,"would_cite":true,"duration_ms":32930,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that no single family of object detectors wins across X-ray benchmarks: the best architecture depends on the dataset's dominant challenge, such as occlusion, hidden items, or scanner domain shift.","keywords":["X-ray imaging","object detection","security screening","CNN detectors","transformer detectors","hybrid CNN-transformer","comparative evaluation","mean average precision"],"falsifier":"Retrain each custom module (CHR, DOAM, LIM) inside its originally matched detector, such as SSD, YOLOv3, or YOLOv5, using the same data splits and hyperparameters, and also give the YOLOv8 variants a per-dataset hyperparameter search; if the modules recover their originally reported gains over the new generic baseline, the architectural-disharmony explanation would be contradicted.","tokens_in":34937,"feed_emoji":"🛃","tokens_out":8150,"duration_ms":85036,"temperature":0.7,"pith_summary":"The paper establishes that no one family of object detectors---CNN, transformer, or hybrid CNN-transformer---consistently outperforms the others on X-ray illicit-item detection. Across six public benchmarks and ten detector/backbone combinations scored by mean average precision (mAP50 and mAP50:95), the winning architecture changes with the dataset's main difficulty: occlusion, domain shift, hidden objects, object size, or clutter. The paper also establishes a second, counter-intuitive result: bolting the X-ray-specific modules CHR, DOAM, and LIM onto a modern YOLOv8 detector consistently degrades detection relative to the plain generic baseline. This matters because it contradicts the common assumption that domain-specific modules automatically help, and it suggests that reported gains in the literature may be tied to the older detector generation those modules were originally attached to.","feed_headline":"No single detector class wins across six X-ray benchmarks","feed_subtitle":"Six datasets, ten detectors: the best choice depends on the task, and added X-ray modules often hurt.","key_machinery":"The mechanism is the controlled comparison grid itself, not a single mathematical identity. Six datasets were selected to isolate distinct challenges: occlusion (OPIXray, CLCXray), scanner domain shift (EDS), deliberately hidden items (PIDray), fine-grained electronic objects (HiXray), and realistic subway scans (SIXray), then crossed with ten detectors covering generic CNN, custom CNN, transformer, and hybrid families, all trained with publicly available implementations and reported under the same mAP definitions plus inference time, parameter count, and GFLOPS. The load-bearing comparative device is the notation D(head, backbone), which isolates the contribution of each detector head and backbone swap, and the per-dataset training recipes that determine whether differences are attributable to architecture or to tuning.","core_discovery":"On its own terms, the paper's central discovery is negative: after aligning datasets, protocols, and evaluation metrics, there is no universal winner. The generic CNN detector D(YOLOv8, CSPDarkNet53) achieves the highest mAP50 on three of six benchmarks, the transformer detector D(Co-DETR, Swin-B) is best on the domain-shift benchmark EDS and ties for best recognition rates on two others, and the hybrid D(YOLOv8, Next-ViT-S) has the best average performance overall and leads on the heavily occluded OPIXray dataset. The paper further claims that all three custom X-ray modules, CHR, DOAM, and LIM, underperform the generic YOLOv8 baseline on every dataset, an effect attributed to architectural disharmony: modules designed for older detection frameworks such as SSD, YOLOv3, and YOLOv5 do not integrate cleanly with modern YOLOv8. At the object level, the paper finds that dense, high-attenuation objects and larger objects are detected better across all architecture families, while low-density items and small or highly articulated objects consistently suffer.","pith_inferences":["An untested but plausible consequence is that combining the two winning components, Co-DETR's assignment strategy with a hybrid Next-ViT-style backbone, could yield a detector that is robust to both domain shift and occlusion; the grid here does not include that combination.","The architectural-disharmony account predicts that simply updating the custom modules' training recipes, such as optimizer or epoch count, will not close the gap, because the gap comes from the integration itself; this is testable through an ablation that matches hyperparameters exactly.","Given the reported size-dependent results, a detector that selects its backbone based on estimated object scale in the image, CNN for small objects and transformer for large ones, is a natural next experiment not performed here.","The paper's central negative result implies that one-benchmark leaderboards in X-ray detection may be misleading, since the reported winner is largely a property of the benchmark rather than of the architecture family."],"forward_implications":["Deployment should be scenario-dependent: real-time screening favors generic CNN detectors, cross-scanner or heavily occluded scenes favor transformer or hybrid detectors, and no single model can be recommended across all checkpoints.","Custom X-ray modules developed for earlier detector generations cannot be assumed to improve modern detectors; gains reported with their original frameworks should not be extrapolated to newer architectures.","Dataset size and number of classes are weak predictors of detector ranking; benchmark difficulty, such as occlusion, clutter, domain shift, and hidden objects, drives performance, so evaluations should be reported per dataset rather than as a single average.","Practical latency cannot be read off parameter count or GFLOPS: some custom modules make YOLOv8 roughly two to four times slower despite comparable theoretical cost.","Object-level and size-level results imply that material density and geometric complexity should be part of any X-ray detection evaluation protocol, because they dominate per-class mAP variance across every architecture family."],"supporting_citations":[{"why":"Supplies the SIXray benchmark (used in its SIXray-D annotated form here) and the CHR module; CHR's underperformance is a key result.","marker":"(Miao et al., 2019)"},{"why":"Supplies the OPIXray occlusion benchmark and the DOAM module; both are central to the occlusion analysis.","marker":"(Wei et al., 2020)"},{"why":"Supplies the HiXray benchmark and the LIM module; LIM's underperformance and latency cost are analyzed.","marker":"(Tao et al., 2021)"},{"why":"Supplies the PIDray hidden-object benchmark whose easy, hard, and hidden splits are compared per detector.","marker":"(Wang et al., 2021)"},{"why":"Supplies the EDS domain-shift benchmark; Co-DETR's superiority there is a central claim.","marker":"(Tao et al., 2022a)"},{"why":"Supplies the CLCXray overlapped-object benchmark used in the comparison.","marker":"(Zhao et al., 2022)"},{"why":"Provides the YOLOv8 model that serves as the generic CNN baseline and as the host for the custom modules.","marker":"(Jocher et al., 2023)"},{"why":"Provides the DINO transformer detector compared within the transformer family.","marker":"(Zhang et al., 2022)"},{"why":"Provides Co-DETR, the strongest transformer detector in the evaluation.","marker":"(Zong et al., 2023)"},{"why":"Provides RT-DETR, used in hybrid configurations with HGNetV2 and Next-ViT-S backbones.","marker":"(Zhao et al., 2024)"}],"fun_headline_variants":["No universal winner in X-ray object detection benchmarks","X-ray detection: best model depends on the dataset","Generic detectors beat custom X-ray modules in new study","Six benchmarks, ten detectors: no clear champion","X-ray AI comparison finds no one-size-fits-all detector"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the per-dataset choices of optimizer, learning rate, weight decay, and epoch count are fair enough that any mAP differences reflect the detector architectures rather than tuning effort.","fun_headline_variants_meta":{"raw":{"variants":["No universal winner in X-ray object detection benchmarks","X-ray detection: best model depends on the dataset","Generic detectors beat custom X-ray modules in new study","Six benchmarks, ten detectors: no clear champion","X-ray AI comparison finds no one-size-fits-all detector"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2921,"prompt_tokens":1099,"completion_tokens":1822,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":715,"completion_tokens_details":{"reasoning_tokens":1746}},"tokens_in":715,"tokens_out":1822,"duration_ms":12817,"temperature":1.0,"reasoning_tokens":1746,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:45:53.654123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain each custom module (CHR, DOAM, LIM) inside its originally matched detector, such as SSD, YOLOv3, or YOLOv5, using the same data splits and hyperparameters, and also give the YOLOv8 variants a per-dataset hyperparameter search; if the modules recover their originally reported gains over the new generic baseline, the architectural-disharmony explanation would be contradicted.","supporting_citations":[],"review_version":1}