{"id":"1c7fd95f-5b01-4ed1-add4-6151e24812be","arxiv_id":"2507.22650","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Validation results for a YOLOv11n-based dual RGB/IR drone detection and payload classification system on the VIP Cup 2025 dataset, with reproducibility and consistency problems.","lead":"SpectraSentinel runs separate YOLOv11n detectors on thermal and visible video streams and fuses their outputs at a decision layer to detect drones, track them, and classify payloads as harmful or normal. The paper reports high accuracy on the VIP Cup 2025 validation set, but the evidence is weakened by validation-only metrics, missing code, and several internal inconsistencies in the tables and methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All reported accuracy figures come from an unspecified validation split used for tuning and model selection, so the central high-accuracy claim is unsupported without a held-out test evaluation; internal metric inconsistencies reinforce this concern.","rationale":"The reader's weakest assumption identifies the most load-bearing vulnerability: all quantitative claims rest on validation metrics from an unspecified split that was also used for tuning and model selection. If that split is not representative, or if tuning overfit it, the central claim of high detection accuracy collapses. This is exactly the kind of evaluation-protocol issue that cannot be repaired by more elaborate fusion or better augmentations; it requires a held-out test evaluation. My stress-test confirms this is the primary concern, and the internal inconsistencies (epoch counts, mAP ordering, tracker descriptions) strengthen it by suggesting the numbers were not produced or reported under a disciplined protocol. I found no independent evidence—no code, no data, no test-set results, no machine-checked proofs—that would mitigate this risk. Therefore the reader's REJECT verdict should stand unchanged.","tokens_in":8424,"tokens_out":4801,"duration_ms":54491,"concrete_test":"Require the authors to release the exact VIP Cup 2025 train/val/test split and rerun all reported YOLOv11n models (RGB and IR drone/bird detection, RGB and IR payload identification) on the held-out test split, reporting mAP@0.5:0.95 for each. If the test-set metrics are not released, or if they differ from Tables IV and VI by more than a few points (e.g., >3), the high-accuracy claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the accuracy numbers in Tables IV and VI. These numbers are computed on a validation split of the VIP Cup 2025 dataset that is never defined: split sizes, sequence IDs, and temporal separation are not given. The same split was used to tune augmentation probabilities (§II-A), training schedules (§IV-A-b), and to choose among fusion variants (§IV-A-e). Because model selection and hyperparameter tuning used the same data that produces the reported metrics, there is no evidence the numbers generalize to the competition test set. This is not a minor omission: the abstract promises 'high detection accuracy... within real-time constraints,' and every quantitative claim in the paper is validation-only. The concern is amplified by internal contradictions that make the validation numbers themselves hard to trust: §IV-A-b states 100–150 epochs while Table I reports epoch 293; Table V reports RGB mAP@0.5:0.95 (0.997) greater than mAP@0.5 (0.994), a COCO-ordering red flag; and the tracker is described once as IoU-based (§III-B) and once as DeepSORT (§V). Any one of these could be a typo; together they indicate the reported numbers have not been checked against a clear evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SpectraSentinel, a dual-stream system for drone detection, tracking, and payload identification using separate YOLOv11n detectors on RGB and infrared streams with decision-layer late fusion. It reports high validation accuracy (mAP@0.5:0.95 around 0.81 for drone/bird detection and 0.99 for payload detection) and real-time performance on RTX GPUs. It also compares several fusion strategies and describes a tracking and direction-estimation pipeline. The central claim is that the proposed lightweight models achieve high accuracy under real-time constraints.","tokens_in":8692,"tokens_out":2828,"duration_ms":33802,"significance":"If the reported accuracy were properly supported, the paper would offer a practical contribution to multimodal drone surveillance, particularly in its modality-specific augmentation strategy and its systematic comparison of mid-fusion versus late-fusion designs. The idea of placeholder injection to handle missing modalities is also worth reporting. However, the paper's current evidence is validation-only, the validation split is not defined, and the reported metrics contain internal inconsistencies (e.g., mAP@0.5:0.95 > mAP@0.5 in Table V). These issues prevent the reader from trusting the headline accuracy claims, so the practical significance is not yet established.","major_comments":[{"comment":"All reported accuracy numbers are computed on an unspecified validation split of the VIP Cup 2025 dataset. The split size, sequence identifiers, and temporal separation are not given. The same split was used to tune augmentation probabilities (§II-A), training schedules (§IV-A-b), and to select among fusion variants (§IV-A-e). Because model selection and hyperparameter tuning were performed on the same data that generate the reported metrics, there is no evidence that these numbers generalize to the competition test set or to unseen surveillance footage. The abstract's claim of 'high detection accuracy' therefore rests on an unsupported evaluation protocol. Please provide a held-out test evaluation, or explicitly document the split construction and demonstrate that no selection or tuning was performed on the reported split.","section":"§IV-A-b, §IV-A-e, Tables IV, VI"},{"comment":"Table V reports RGB Faster R-CNN payload detection with mAP@0.5:0.95 = 0.997 and mAP@0.5 = 0.994. Under the standard COCO metric ordering, mAP@0.5 must be greater than or equal to mAP@0.5:0.95, since the latter averages across IoU thresholds starting at 0.5. This violation signals a calculation or reporting error. Because Table V is used to support the payload identification claim, the numeric results must be recomputed and corrected before the payload accuracy can be assessed.","section":"Table V"},{"comment":"The training setup in §IV-A-b states that models were trained for 100–150 epochs, and Table III reports epochs 100 and 150. However, Table I reports 'Epoch 293' for the final RGB drone/bird model. These numbers are mutually inconsistent and suggest either that Table I comes from a different experiment or that the epoch count is misreported. The paper must state exactly which experiment produced each table and resolve the contradiction, as this directly affects reproducibility.","section":"§IV-A-b and Table I"},{"comment":"Section III-B describes tracking as a lightweight Intersection-over-Union (IoU)-based tracker that bridges gaps of 10–15 frames, while Section V states that tracking is achieved via DeepSORT, and reference [15] is the DeepSORT paper. These are different algorithms with different association logic and computational profiles. The paper must clarify which tracker was actually implemented and evaluated; the current text makes the tracking component ambiguous and unverifiable.","section":"§III-B and §V"}],"minor_comments":[{"comment":"The augmentation probabilities (mosaic 80%, copy-paste 20%, horizontal flip 30%, translation 5%, scale ±10%) are presented as design choices, but no sensitivity analysis or ablation is provided. Since these values were tuned on the same validation split used for final metrics, a short ablation or at least a statement about their mild influence would help the reader gauge robustness.","section":"§II-A"},{"comment":"The introduction contains a typographical artifact: 'UA Vs' appears at the start of the second paragraph. Please correct to 'UAVs'.","section":"§I"},{"comment":"Reference [10] cites 'YOLOv11' to arXiv:2401.00567, but that identifier does not correspond to the official YOLOv11 release. Please cite the correct source, such as the Ultralytics documentation or the actual YOLOv11 publication, to avoid misleading readers.","section":"References"},{"comment":"Captions for Figs. 5–7 say 'Dataset Statistics and Model Performance Overview for training RGB Drone Image' and similar phrases; these are grammatically awkward and should be rephrased to describe the content more clearly.","section":"Figure captions"},{"comment":"The analysis states that decision-layer fusion 'achieved a slightly lower' performance than transformer-based mid-fusion, but no quantitative comparison is given between Table III and Table IV/VI. Adding a direct comparison table would strengthen the discussion of fusion trade-offs.","section":"§IV-A-f"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competition report that lacks a proper evaluation protocol. The validation-only results, unspecified split, and internal metric inconsistencies are fixable in principle, but they require a substantial revision with a new held-out evaluation or a clear justification that the reported split was never used for tuning. I believe the authors should be given a chance to revise rather than an outright reject, but the revised manuscript must include a test-set or properly held-out validation evaluation before the central accuracy claim can be accepted. The tracker inconsistency and the mAP ordering error also need correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a competition engineering report, not a research paper. The dual-stream YOLOv11n setup with per-modality augmentation and late decision fusion is reasonable, and the placeholder-injection trick for handling missing modalities is a nice practical touch. But the accuracy numbers are not supported as reported, and without code or a defined test split there is no way to check them.\n\nWhat's actually here: the authors train separate YOLOv11n detectors on RGB and IR, tailor augmentations to each domain (no hue/saturation for IR, modest color jitter for RGB), and combine outputs with a heuristic decision layer. They also compare several fusion strategies, which is the right thing to do. That part is competent engineering.\n\nThe soft spots are not minor. Every reported metric is computed on an unspecified validation split of the VIP Cup 2025 dataset. The same split was used to tune augmentations, training schedules, and to choose among fusion variants, so the numbers are fitted to that split. There is no test-set evaluation. That alone would make me wary. Then there are internal contradictions that undermine even the validation numbers: Table V reports RGB mAP@0.5:0.95 = 0.997 while mAP@0.5 = 0.994, which violates the COCO ordering; Table I shows epoch 293 while Section IV-A-b says 100–150 epochs; the tracker is described as IoU-based in Section III-B but as DeepSORT in the conclusion. Any one could be a typo, but together they suggest the manuscript was not checked against a clear evaluation protocol.\n\nThere is also a mismatch between the abstract and the authors' own results: the abstract sells the dual-stream late-fusion system, while Section IV-A-f says transformer-based mid-fusion achieved higher mAP. The late-fusion system may still be attractive for flexibility, but the headline claim is not supported by their own comparison.\n\nNo code, data, or model weights are provided, so the numbers cannot be verified. The citations look fine, and the related work is acknowledged.\n\nWho this is for: a practitioner looking for a starting point on multimodal drone detection might skim it, but the lack of reproducibility makes that value small. This should not go to a serious referee in its current form. If the authors release code and data, specify the split, correct the tables, and evaluate on a held-out test set, it could become a useful extension. I would not bring it to the reading group and would not cite it. Desk reject.","headline":"A competent engineering report whose headline numbers are unverifiable and internally inconsistent; desk reject.","tokens_in":9267,"tokens_out":2917,"would_cite":false,"duration_ms":31463,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two independent YOLOv11n detectors on RGB and infrared streams spot drones, birds, and payloads in real time.","keywords":["drone detection","thermal infrared","RGB","YOLOv11","dual-stream","late fusion","payload identification","real-time object detection"],"falsifier":"Take the trained RGB and IR YOLOv11n models and the decision-layer fusion rule, run them on a held-out portion of the VIP Cup 2025 data or on another paired RGB-IR drone dataset, and compare mAP@0.5:0.95 with the reported values of about 0.81 for drone/bird detection and 0.99 for payloads; a large drop, or a frame rate below 30 FPS on RTX hardware, would refute the central claim.","tokens_in":8221,"feed_emoji":"🛸","tokens_out":5991,"duration_ms":63717,"temperature":0.7,"pith_summary":"This paper sets out to show that a lightweight surveillance system can detect drones, distinguish them from birds, and identify whether a drone carries a harmful or normal payload by running two independent YOLOv11n object detectors—one on RGB video and one on infrared video—and combining their outputs only at the final decision layer. The claimed payoff is modality complementarity without the cost of early fusion: RGB provides visual detail for payloads, IR provides thermal contrast in darkness or haze, and the system adapts when only one camera is available. On the VIP Cup 2025 validation data the paper reports mAP@0.5:0.95 of 0.807 (RGB) and 0.81 (IR) for drone/bird detection and payload mAP of 0.989/0.990, while running above 30 FPS on RTX GPUs. The paper also reports that a transformer-based mid-fusion variant reached a higher mAP of 0.84, positioning late fusion as a flexibility and speed trade rather than the accuracy ceiling.","feed_headline":"Dual YOLOv11n streams spot drones and payloads in real time","feed_subtitle":"RGB and infrared detectors, fused at the decision layer, report 0.99 payload mAP and over 30 FPS.","key_machinery":"The load-bearing mechanism is the decision-layer late-fusion pipeline: two independently trained YOLOv11n backbones produce separate detections, and a routing layer chooses the higher-confidence modality, applies non-maximum suppression, and, for payloads, flags harmful if either stream is confident. Placeholder-channel injection—a white image for a missing RGB stream, a grayscale image for a missing IR stream, and an IR triplet for a missing RGB stream—keeps the dual-input architecture uniform, so one system handles RGB-only, IR-only, and paired inputs without retraining.","core_discovery":"The central claim is that deliberately avoiding early fusion does not cost much accuracy while buying deployment flexibility. Two YOLOv11n models are trained separately, one per modality, with domain-specific augmentation—no hue or saturation changes for thermal images, small translations and scaling, plus mosaic and copy-paste to strengthen small-object detection. At inference, a decision layer combines the two streams using weighted confidence selection and non-maximum suppression; if one camera is missing, a white or grayscale placeholder keeps both backbones running. The authors' message is that this dual-stream, late-fused system achieves high validation accuracy on drone-versus-bird and payload classification while staying lightweight enough for real-time edge deployment, and that mid-level transformer fusion is the main alternative with a small accuracy edge.","pith_inferences":["The validation-only numbers mean the strongest test of the design is an unseen dataset; a reasonable extension would be to report generalization on a second paired RGB-IR drone dataset.","The accuracy gap between late fusion and transformer-based mid-fusion suggests a hybrid approach—late fusion for runtime with an optional attention-based fusion head for high-accuracy offline analysis—could capture both benefits.","The placeholder-injection trick could be reused for other sensor pairs, such as adding a third stream like radar or multispectral imagery by adding one backbone and one routing rule.","If the validation split has overfit the tuning choices, the reported payload mAP near 0.99 may shrink substantially; this is directly testable on a held-out split."],"forward_implications":["The system can switch between RGB-only, IR-only, and dual-mode inference without retraining, so a single model pair covers degraded sensor conditions.","Payload threat assessment uses a logical OR across modalities, so a harmful payload visible in only one spectrum is still flagged.","On RTX GPUs the reported frame rate exceeds 30 FPS at 320x256 resolution, indicating the design fits real-time surveillance budgets.","Because each backbone is trained separately, adding a new camera modality later requires only a new detector and a routing rule, not a fused retraining run.","Transformer-based mid-fusion reached the highest reported mAP of 0.84, so the paper positions late fusion as a flexibility and speed trade rather than the accuracy ceiling."],"supporting_citations":[{"why":"Supplies the rationale and prior evidence for combining thermal and RGB imagery for UAV detection.","marker":"[2]"},{"why":"Provides the original YOLO one-stage detection framework that the paper builds on.","marker":"[3]"},{"why":"Documents domain-specific challenges of thermal small-object detection that motivate the IR-tailored pipeline.","marker":"[4]"},{"why":"Provides a recent small-object UAV detection variant with multi-branch heads that the paper extends conceptually.","marker":"[5]"},{"why":"Supplies an infrared-specific modified YOLO baseline for small drone detection.","marker":"[6]"},{"why":"Introduces the mosaic and copy-paste augmentation strategy used to improve small-object detection.","marker":"[9]"},{"why":"Describes YOLOv11, the architecture family from which the nano detector is taken.","marker":"[10]"},{"why":"Provides the transformer-based infrared-RGB fusion approach used as the main alternative fusion baseline.","marker":"[12]"}],"fun_headline_variants":["Lightweight YOLOv11n pair spots drones, IDs payloads in real time","Two specialized YOLOv11n models track drones and classify payloads without early fusion","Late fusion of separate YOLOv11n streams for real-time drone and payload ID","Independent YOLOv11n per camera: thermal and RGB for drone payload ID","Real-time drone detection with dual YOLOv11n streams, no early fusion needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies come from a single validation split of the VIP Cup 2025 dataset, and the same split guided the choice of augmentation probabilities, fusion rules, and NMS behavior, so the central accuracy claim depends on that split being representative of unseen conditions.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight YOLOv11n pair spots drones, IDs payloads in real time","Two specialized YOLOv11n models track drones and classify payloads without early fusion","Late fusion of separate YOLOv11n streams for real-time drone and payload ID","Independent YOLOv11n per camera: thermal and RGB for drone payload ID","Real-time drone detection with dual YOLOv11n streams, no early fusion needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001082,"raw_usage":{"total_tokens":4511,"prompt_tokens":918,"completion_tokens":3593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":3483}},"tokens_in":534,"tokens_out":3593,"duration_ms":29744,"temperature":1.0,"reasoning_tokens":3483,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:25:46.198493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained RGB and IR YOLOv11n models and the decision-layer fusion rule, run them on a held-out portion of the VIP Cup 2025 data or on another paired RGB-IR drone dataset, and compare mAP@0.5:0.95 with the reported values of about 0.81 for drone/bird detection and 0.99 for payloads; a large drop, or a frame rate below 30 FPS on RTX hardware, would refute the central claim.","supporting_citations":[{"cited_title":"Activity distribution of comet 67P/Churyumov-Gerasimenko from combined measurements of non-gravitational forces and torques","cited_arxiv_id":"2301.04892","evidence_quote":"Supplies the rationale and prior evidence for combining thermal and RGB imagery for UAV detection."},{"cited_title":"Redmon et al., ”You Only Look Once: Unified, Real-Time Object Detection,” inProc","cited_arxiv_id":null,"evidence_quote":"Provides the original YOLO one-stage detection framework that the paper builds on."},{"cited_title":"Zhao et al., ”Thermal image-based small object detection with modified YOLO,”Sensors, vol","cited_arxiv_id":null,"evidence_quote":"Documents domain-specific challenges of thermal small-object detection that motivate the IR-tailored pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a recent small-object UAV detection variant with multi-branch heads that the paper extends conceptually."},{"cited_title":"Zhao and F","cited_arxiv_id":null,"evidence_quote":"Supplies an infrared-specific modified YOLO baseline for small drone detection."},{"cited_title":"YOLOv5 by Ultralytics,","cited_arxiv_id":null,"evidence_quote":"Introduces the mosaic and copy-paste augmentation strategy used to improve small-object detection."},{"cited_title":"Mean ergodic theorems in $L^r(\\mu)$ and $H^r(\\mathbb T)$, $0<r<1$","cited_arxiv_id":"2401.00567","evidence_quote":"Describes YOLOv11, the architecture family from which the nano detector is taken."},{"cited_title":"ICAFusion: Infrared and RGB Image Fusion for Robust Object Detection in Adverse Conditions,","cited_arxiv_id":null,"evidence_quote":"Provides the transformer-based infrared-RGB fusion approach used as the main alternative fusion baseline."}],"review_version":1}