{"id":"348fe3f3-c227-4186-a061-12a63f6abd57","arxiv_id":"2505.24489","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Deformable-DETR outperformed seven baselines on two remote sensing datasets, but the comparison protocol has methodological gaps.","lead":"This paper benchmarks Deformable-DETR, a transformer-based object detector, on optical and radar satellite images. It reports F1 scores of 95.12% on aircraft photos and 94.54% on ship radar images, beating several CNN and transformer baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol ambiguity and internal training-time inconsistency leave the central comparison unsupported.","rationale":"The reader correctly identified the evaluation protocol as the weakest point, and I agree that the ambiguity between the 80/10/10 split and the 10-fold cross-validation, plus the absence of error bars, undermines the reported averages. My stress-test sharpens this into three concrete, checkable issues. The first is the unresolved split-versus-cross-validation question, which affects every metric in Table 2. The second is a direct internal contradiction between the text (365 s and 3750 s training time for Deformable-DETR) and Table 2 (306.53 s and 3370.16 s), making the 'shortest training time' claim unreliable as reported. The third is the protocol shift in Table 3: different pretraining (ImageNet-1K vs COCO), different epoch count (200 vs 12), and different dataset version (original vs corrected annotations) mean that the comparison with specialized remote sensing detectors is not controlled. Since the specialized-model scores are drawn from their original publications rather than re-run under the same regime, the claim that Deformable-DETR outperforms these detectors is not supported by a fair benchmark. These issues do not require rejecting the paper, because they are potentially fixable by rerunning experiments under a unified protocol and reporting fold-level statistics; hence the reader's CONDITIONAL verdict remains appropriate. I mark agreement as partial because my concern, while aligned with the reader's, identifies additional internal inconsistencies and a specific protocol mismatch that go beyond the split ambiguity.","tokens_in":8477,"tokens_out":4907,"duration_ms":52554,"concrete_test":"Run all models in Tables 2 and 3 under one fixed protocol: same pretraining (COCO), same epoch count, same original SSDD annotations, and 10-fold cross-validation, reporting per-fold mean and standard deviation for F1 and training time. If Deformable-DETR's lead disappears or the training-time values change materially, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim rests on two comparisons: Table 2 (seven generic detectors) and Table 3 (three remote-sensing-specific detectors). The load-bearing assumption is that both comparisons are controlled and that the reported metrics are well-defined. This assumption is not met. First, Section 5 describes an 80/10/10 train/validation/test split and then a 'stratified 10-Fold validation' without reconciling them; Table 2 reports a single set of metrics, so the reader cannot tell whether these are fold-averaged or from a single split, and no variance is reported. Second, the training-time claim is internally inconsistent: the text says Deformable-DETR trained 'over 365 seconds for the optical dataset and 3750 seconds for the SSDD', but Table 2 lists 306.53 s and 3370.16 s; if the table values are per-fold, the total is about ten times larger, and if they are totals, the text is wrong. Third, Table 3 compares Deformable-DETR fine-tuned on the original SSDD with ImageNet-1K pretraining over 200 epochs, while Table 2 uses COCO pretraining, 12 epochs, and a corrected-annotation version of the dataset; the specialized models' scores appear to be taken from their original publications, so the comparison mixes training protocols, backbones, and data versions. Without a shared evaluation protocol, the conclusion that deformable attention mechanisms are a strong and efficient choice is not supported by the evidence as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates Deformable-DETR for object detection on two remote sensing datasets, one optical (Pleiades Aircraft) and one SAR (SSDD), and compares it with six other generic detectors (RetinaNet, Faster R-CNN, YOLOv11, DETR, DN-DETR, Conditional DETR, DAB-DETR) and three remote-sensing-specific transformer-based detectors (ASAFE, DAPN, CRTTransSar). The authors report that Deformable-DETR achieves the best F1 scores (95.12% on Pleiades, 94.54% on SSDD) and the shortest reported training time over 12 epochs, and conclude that deformable attention mechanisms are well suited to remote sensing object detection.","tokens_in":8810,"tokens_out":2238,"duration_ms":25879,"significance":"If the benchmark were properly controlled and reproducible, the paper would provide useful evidence that a generic deformable-attention detector can compete with or outperform both CNN-based detectors and remote-sensing-specific architectures while training quickly. The authors also release code and data, which is commendable for reproducibility. However, as presented, the central claim is weakened by protocol ambiguity, an internal inconsistency in the reported training times, and a mismatch between the main comparison and the comparison against specialized remote sensing detectors. The core idea is plausible and the work is potentially valuable, but the evidence does not yet support the strength of the stated conclusions.","major_comments":[{"comment":"The reported F1 score for DETR is inconsistent with the reported precision and recall. For precision 93.21 and recall 90.35, the harmonic mean is 2×93.21×90.35/(93.21+90.35) ≈ 91.76, not 91.19 as listed. This suggests an arithmetic or data-handling error in the metrics. Please verify all entries in Table 2 and re-check the F1 calculations for every model, since a single error in the table undermines confidence in the entire numerical comparison.","section":"Table 2, Pleiades Aircraft row for DETR"}],"minor_comments":[{"comment":"Several references have formatting issues, such as missing spaces after initials (e.g., 'V . V`eque') and missing page ranges; a careful proofread of the reference list would improve the manuscript.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is already accepted at KES 2025, but for a journal version the evaluation protocol must be clarified and the internal inconsistencies fixed. The most serious issue is the mismatch between the reported F1 and precision/recall for DETR in Table 2 and the ambiguous training-time statement. I would ask the authors to provide the code/data outputs that support the tables and to re-run or clearly re-label the Table 3 comparison. If the protocol cannot be clarified, the central benchmark claim would not withstand scrutiny."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one-line take: this is a straightforward application of Deformable-DETR to one optical and one SAR dataset, with a comparison against several generic and remote-sensing-specific detectors. The GitHub repo ships notebooks, which is genuinely useful. But the paper's central claim—that Deformable-DETR is both the most accurate and the fastest—rests on an evaluation protocol that is ambiguous in exactly the places where it matters.\n\nWhat is actually new: running this particular set of seven models on Pleiades Aircraft and SSDD under one roof, and reporting both optical and SAR results for Deformable-DETR. That is a modest extension over prior work, not a new method. The code and data links earn credit; they make the benchmark falsifiable, which is more than most papers of this type do.\n\nNow the soft spots, in proportion. First, the protocol: Section 5 first says 80/10/10 train/validation/test, then says a stratified 10-fold cross-validation was performed. These are not the same thing. Table 2 reports a single set of metrics, so the reader cannot tell whether those are fold-averaged or from one split. No variance or standard deviation is given. Without that, I cannot tell whether the differences between Deformable-DETR and, say, DAB-DETR are meaningful or noise. Second, the training-time numbers are internally inconsistent: the text says Deformable-DETR trained in 365 seconds on Pleiades and 3750 on SSDD, but Table 2 lists 306.53 s and 3370.16 s. If the table values are per fold, the totals are roughly ten times larger; if they are totals, the text is wrong. Either way, the speed claim is unsupported as written. Third, Table 3 compares Deformable-DETR fine-tuned on the original SSDD with ImageNet-1K pretraining over 200 epochs, while Table 2 uses COCO pretraining, 12 epochs, and a corrected-annotation version. Those are different protocols, so comparing across tables is apples to oranges. Fourth, a concrete arithmetic check: Table 2 gives DETR on Pleiades precision 93.21 and recall 90.35, which yields an F1 of about 91.76, not the reported 91.19. That kind of inconsistency makes me distrust the other numbers until the code is run to verify.\n\nWhere the paper does well: the related work is relevant and the comparison set is reasonable. The idea of manually correcting truncated-object annotations is sensible, though the correction process is not described enough to be reproducible. The paper is clear about what it does; it is not hiding a bad method behind jargon.\n\nWho is this for? Someone wanting a quick baseline for Deformable-DETR on these two datasets, and who is willing to re-run the notebooks. Not someone who needs a definitive ranking, because the protocol holes prevent that.\n\nMy recommendation: send it to peer review, but with the expectation of major revision. The code availability makes the numbers checkable, and the benchmark question is worth answering cleanly. Require the authors to state whether they averaged over folds, report error bars, reconcile the training times, and redo Table 3 under a shared protocol. If those fixes land, this becomes a solid empirical note. As is, the central claim is not fully supported.","headline":"A usable but under-specified benchmark claiming Deformable-DETR wins on two remote sensing datasets; the core claim needs protocol details and corrected numbers before it can be trusted.","tokens_in":9201,"tokens_out":1148,"would_cite":false,"duration_ms":14779,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that Deformable-DETR, with multi-scale deformable attention, achieves the best F1 score and shortest training time among seven detectors on both optical and SAR remote sensing datasets.","keywords":["Deformable-DETR","deformable attention","object detection","remote sensing","optical aircraft detection","SAR ship detection","transformer detectors","10-fold cross-validation"],"falsifier":"Run the published benchmark notebooks on SSDD and Pleiades Aircraft using exactly the same folds, 12 epochs, COCO-pretrained weights, and augmentation pipeline, and check two numbers: whether Deformable-DETR's average F1 across the ten folds reproduces 94.54% and 95.12%, and whether its training time is still the shortest when all models are measured with identical I/O and logging. A single model whose per-fold metrics are not reproduced, or whose training time was measured without the same data-loading overhead, would falsify the speed-and-accuracy claim as stated.","tokens_in":8338,"feed_emoji":"🛰️","tokens_out":9135,"duration_ms":96810,"temperature":0.7,"pith_summary":"The paper sets out to show that Deformable-DETR—a transformer detector whose attention mechanism samples a sparse set of points at multiple feature scales instead of attending to every location—is the best all-round choice for object detection on remote sensing images. Using one optical dataset (Pleiades aircraft) and one SAR dataset (ship detection), it compares seven detectors under a common protocol and reports that Deformable-DETR achieves the highest F1 score on both datasets (95.12% and 94.54%) while also requiring the least training time over 12 epochs. It additionally reports that Deformable-DETR beats three remote-sensing-specific transformer detectors on the original SAR ship dataset after 200 epochs of fine-tuning. If these results hold, deformable attention would be a strong, efficient default for detecting small and multi-scale objects in overhead imagery, in both passive optical and active SAR modes.","feed_headline":"Deformable-DETR beats seven rivals on remote sensing","feed_subtitle":"Best F1 scores and shortest training times on optical aircraft and SAR ship images.","key_machinery":"The load-bearing object is the deformable attention module introduced by the Deformable-DETR architecture. Instead of computing attention over the full feature map, each query samples $N$ points at each of $M$ feature-map scales, with learnable sampling offsets and attention weights; the output is a weighted sum over these sampled points, written in the paper as $\\sum_{m=1}^{M}\\sum_{n=1}^{N} A_{mn} W_{mn} f(x_{mn})$. This mechanism simultaneously reduces the computational cost of transformer attention and lets the model adapt to objects of different sizes, which is why the paper attributes both the accuracy gains and the training-time savings to it.","core_discovery":"On the paper's own terms, the discovery is that deformable attention, not a larger model or longer training schedule, is what accounts for the best accuracy-speed trade-off. Deformable-DETR's multi-scale deformable attention lets each query concentrate on the most informative points across feature-map levels, which the paper identifies as the mechanism behind its strong performance on small-scale objects and its faster convergence. With COCO-pretrained weights and 12 epochs, the model posts F1 95.12% and mean average precision over IoU thresholds 0.5 to 0.95 of 76.75% on Pleiades Aircraft, and F1 94.54% and 76.14% on SSDD, with the shortest training times in both settings. After 200 epochs of fine-tuning on the original SSDD, Deformable-DETR reaches 98.13% mAP at IoU 0.5 and F1 96.36%, surpassing the specialized remote sensing detectors against which it was compared.","pith_inferences":["A caveat the paper does not state: its headline ranking is a 12-epoch result, so the conclusion that deformable attention trains fastest may not survive longer schedules where other detectors catch up; testing at matching convergence would separate 'faster to converge' from 'better at convergence'.","Because the dataset annotations were manually corrected, especially the truncated-object classes, the reported F1 scores partly reflect annotation curation; a sensitivity check on the original versus corrected annotations would show how much of the advantage is due to the attention mechanism itself.","A natural testable extension is to feed the same deformable attention into a segmentation head, a direction the paper itself proposes as future work; the mechanism's scale-adaptivity should transfer beyond bounding boxes and can be checked directly with the released code."],"forward_implications":["If Deformable-DETR is genuinely the fastest and most accurate among the seven models at 12 epochs, the same compute budget can support more training epochs or more experiments, so the reported advantage compounds in practice.","The result suggests that one deformable-attention architecture can serve both optical and SAR remote sensing, reducing the need for sensor-specific detectors.","The strong result on small-scale aircraft and ship instances implies that deformable attention is a useful mechanism for small-object detection in high-resolution overhead imagery.","The comparison against remote-sensing-specific transformers indicates that a general-purpose detector with deformable attention can match or beat specialized designs on SAR ship detection.","The release of the benchmark code and notebooks would let other teams reproduce the ranking and extend it to new datasets."],"supporting_citations":[{"why":"Supplies the Deformable-DETR architecture and the multi-scale deformable attention mechanism that is the paper's subject.","marker":"[18]"},{"why":"Defines the DETR baseline whose slow convergence motivates Deformable-DETR and which is compared in the benchmark.","marker":"[19]"},{"why":"Provides the RetinaNet baseline with focal loss and FPN used as a CNN comparison.","marker":"[20]"},{"why":"Provides the Faster R-CNN two-stage baseline with RPN used as a CNN comparison.","marker":"[22]"},{"why":"Provides the YOLOv11 detector baseline used as the latest YOLO comparison.","marker":"[23]"},{"why":"Supplies the optical Pleiades Aircraft dataset on which the F1 95.12% result is reported.","marker":"[27]"},{"why":"Supplies the SAR Ship Detection Dataset (SSDD) on which the F1 94.54% and the specialized-model comparison are based.","marker":"[28]"},{"why":"Provides the COCO annotation standard and pretrained weights used to fine-tune all models.","marker":"[29]"},{"why":"One of the remote-sensing-specific SAR detectors that Deformable-DETR is claimed to outperform in Table 3.","marker":"[30]"}],"fun_headline_variants":["Deformable attention gives DETR top F1 in remote sensing","Deformable-DETR beats CNN and transformer rivals on aircraft and ships","F1 95%+ from deformable attention in optical and SAR detection","Deformable attention key to faster, accurate remote sensing detection","Deformable-DETR sets benchmarks on optical and SAR remote sensing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim stands on the assumption that every model was trained and validated under the same protocol, with the reported 10-fold stratified cross-validation actually run as described; if the 80/10/10 split and the 10-fold validation were not properly reconciled, or if models used different augmentation, pretraining, or hyperparameters, the ranking could change.","fun_headline_variants_meta":{"raw":{"variants":["Deformable attention gives DETR top F1 in remote sensing","Deformable-DETR beats CNN and transformer rivals on aircraft and ships","F1 95%+ from deformable attention in optical and SAR detection","Deformable attention key to faster, accurate remote sensing detection","Deformable-DETR sets benchmarks on optical and SAR remote sensing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1525,"prompt_tokens":958,"completion_tokens":567,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":471}},"tokens_in":574,"tokens_out":567,"duration_ms":6775,"temperature":1.0,"reasoning_tokens":471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:20:20.590338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published benchmark notebooks on SSDD and Pleiades Aircraft using exactly the same folds, 12 epochs, COCO-pretrained weights, and augmentation pipeline, and check two numbers: whether Deformable-DETR's average F1 across the ten folds reproduces 94.54% and 95.12%, and whether its training time is still the shortest when all models are measured with identical I/O and logging. A single model whose per-fold metrics are not reproduced, or whose training time was measured without the same data-loading overhead, would falsify the speed-and-accuracy claim as stated.","supporting_citations":[{"cited_title":"An adaptive sample assignment strategy based on feature enhancement for ship detection in sar images,","cited_arxiv_id":null,"evidence_quote":"One of the remote-sensing-specific SAR detectors that Deformable-DETR is claimed to outperform in Table 3."},{"cited_title":"End-to-end object detection with transform- ers,","cited_arxiv_id":null,"evidence_quote":"Defines the DETR baseline whose slow convergence motivates Deformable-DETR and which is compared in the benchmark."},{"cited_title":"Focal loss for dense object detection,","cited_arxiv_id":null,"evidence_quote":"Provides the RetinaNet baseline with focal loss and FPN used as a CNN comparison."},{"cited_title":"Airbus aircrafts detection sample dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the optical Pleiades Aircraft dataset on which the F1 95.12% result is reported."},{"cited_title":"Sar ship detection dataset (ssdd): Official release and comprehensive data analysis,","cited_arxiv_id":null,"evidence_quote":"Supplies the SAR Ship Detection Dataset (SSDD) on which the F1 94.54% and the specialized-model comparison are based."},{"cited_title":"Microsoft coco: Common objects in context,","cited_arxiv_id":null,"evidence_quote":"Provides the COCO annotation standard and pretrained weights used to fine-tune all models."}],"review_version":1}