{"id":"a59e8238-ae39-4764-8284-6a21693a596e","arxiv_id":"2412.10453","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A new 3000-image small-object detection benchmark assembled from existing datasets is evaluated with standard detectors, yielding baseline mAP scores, but the dataset is not released and tracking results are missing.","lead":"This paper introduces a 3000-image satellite dataset for tiny object detection and reports baseline mAP scores for eight standard detection models. It also claims to test video tracking with ByteTrack, but no tracking results are actually presented.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unreleased dataset and unspecified test split make the central benchmark comparisons in Table 1 irreproducible and unverifiable.","rationale":"The reader's weakest assumption correctly targets dataset validity: without the dataset or statistics, the benchmark comparisons cannot be trusted. My stress-test sharpens this into a concrete reproducibility failure: Section III omits the split, class counts, object-area distribution, and annotation checks, while Table 1's mAP_S/mAP_M/mAP_L columns are only interpretable through COCO area thresholds. The paper also omits training settings, so the reported rankings could be artifacts of configuration. The ByteTrack claim has no reported results, but it is secondary to the central benchmark. These issues fully support the reader's REJECT verdict, so no verdict change is needed.","tokens_in":12568,"tokens_out":3429,"duration_ms":37730,"concrete_test":"Require the authors to release the exact image IDs and train/validation/test split of the 3000-image subset, together with per-class object counts and object-area histograms before and after tiling. Then independently rerun one detector (e.g., Faster R-CNN ResNet50-FPN) with the same split and a stated training protocol; if the reproduced mAP deviates substantially from Table 1, or the area histogram shows that most objects fall outside COCO's 0-32 squared-pixel small-object range, the central benchmark claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the curated 3000-image subset supports reliable mAP comparisons for tiny-object detection in satellite imagery. That claim requires the dataset to be a valid, reproducible benchmark. Section III describes sampling roughly 1000 images per class from AiTODv2 vehicle/ship and tiled Airbus aircraft images, with Roboflow-based COCO annotation standardization, but never states the train/validation/test split, per-class instance counts, object-area distribution, or annotation-quality checks. The COCO mAP columns in Table 1 (mAP_S, mAP_M, mAP_L) depend on area thresholds; without an area histogram one cannot tell whether mAP_S is computed on genuinely tiny objects or on objects that became small only after tiling/resizing. The paper also omits training hyperparameters such as epochs, learning rate, batch size, and augmentations, so even with a valid dataset, detector differences could reflect configuration choices rather than model efficacy. Because the dataset and code are not released, no independent verification is possible, making the core empirical contribution of a 'dataset-centric' benchmark unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to curate a 3000-image satellite dataset (\"SkyFusion\") for tiny object detection, comprising cars, ships, and airplanes sampled from AiTODv2 and the Airbus Aircraft Detection dataset, and to evaluate eight object detectors (Faster R-CNN, RetinaNet, FCOS, VarifocalNet, FoveaBox, RTMDet, DDOD, DetectoRS) as well as ByteTrack video tracking on SAT-MTB. The only quantitative results are in Table 1, which reports COCO mAP metrics for the detectors. The introduction and literature review are followed by descriptions of each detector and a Results section that contains a figure reference but little experimental detail. The conclusion is a short literature review of SNIP/SNIPER rather than a synthesis of the reported experiments.","tokens_in":12816,"tokens_out":3477,"duration_ms":35192,"significance":"If the dataset and benchmark were released and the results were reproducible, the paper would provide a useful comparison of mainstream detectors on satellite tiny-object detection. The choice of three vehicle, ship, and airplane classes is practical, and the detector coverage is reasonably broad. However, the manuscript provides no dataset release, no code or checkpoints, no training/validation/test split, no training hyperparameters, no error bars, and no analysis of object-size distributions. The reported mAP values cannot therefore be verified or interpreted, so the potential significance is currently unrealized. The paper's strength is its survey-like coverage of detectors, but as an empirical benchmark it lacks the essential evidence needed to support its claims.","major_comments":[{"comment":"The central benchmark claim rests on an unreleased, underspecified dataset. Section III describes sampling roughly 1000 images per class from AiTODv2 and tiled Airbus images, but it does not report the train/validation/test split, per-class instance counts, object-area histogram, or any annotation-quality checks. The COCO mAP_S, mAP_M, and mAP_L values cannot be interpreted without knowing how many objects fall in each area bracket, especially since tiling and resizing can change effective object areas. In addition, the mAP_L column is uniformly -1.000, which is not a valid COCO metric and is never explained; this looks like a placeholder or undefined value and casts doubt on the integrity of the table.","section":"§III and Table 1"},{"comment":"Training configurations are absent. The paper does not state epochs, learning rate, batch size, optimizer, scheduler, augmentations, or the number of independent runs for any of the sixteen reported model variants. Without this information, the differences in Table 1 (e.g., DDOD ResNet50+FPN at mAP 0.339 versus Faster R-CNN at 0.103) could be due to configuration choices rather than model efficacy. No code, configuration files, or checkpoints are released, so the comparisons are not reproducible by any third party.","section":"§III and Table 1"},{"comment":"The dataset curation is described in only three sentences and does not justify that the resulting subset is a representative or valid benchmark for tiny object detection. It is unclear how the AiTODv2 vehicle and ship images are combined with the tiled Airbus aircraft images, whether class distributions are balanced, whether tiling to 640x640 truncates objects, and what fraction of objects genuinely fall below the 1024-pixel area threshold. The assertion that \"the average area occupied by objects in the SkyFusion dataset falls within this threshold\" is unsupported by any distribution statistics, and the definition of the threshold itself is ambiguous: \"32×32 or 1024 pixels\" is redundant because 32×32 equals 1024 square pixels.","section":"§II.A and §III"},{"comment":"The abstract and introduction promise a satellite video tracking evaluation using ByteTrack on the SAT-MTB dataset, but Section V contains no tracking results whatsoever. The text in Section V.A is generic commentary that even says \"the image likely compares the mAP performance...\"; it does not report any tracking metrics. Figure 2 is referenced as though it contains learning curves, but the caption refers to \"Mean Average Position\" and no actual figure content is described or plotted. The tracking claim is therefore unsupported and the Figure 2 reference appears to be placeholder text.","section":"§I, §V.A, and Figure 2"},{"comment":"The Conclusion does not conclude the paper's empirical study. It is a two-paragraph review of the SNIP and SNIPER methods and repeats a statement about ``a comprehensive review'' that is not the paper's actual contribution. There is no summary of the Table 1 findings, no discussion of which detectors performed best or why, and no concrete future-work agenda tied to the reported experiments. This structural gap prevents the reader from understanding what the authors take their own results to mean.","section":"§VI"}],"minor_comments":[{"comment":"The heading and text use \"RetineNet\" instead of \"RetinaNet\".","section":"§IV.B"},{"comment":"The model name is spelled \"Varifocal Net\" but the original is \"VarifocalNet\"; the spelling should be consistent.","section":"§IV.D"},{"comment":"Reference [56] cites the paper \"DDoD: Dual Denial of Decision Attacks on Human-AI Teams\" (IEEE Pervasive Computing, 2023), which is not the DDOD object detector discussed in §IV.G; the correct DDOD reference needs to be supplied.","section":"References"},{"comment":"The sentence defining tiny objects as \"those occupying areas less than or equal to 32 × 32 or 1024 pixels\" is circular because 32×32 = 1024; the intended thresholds should be clarified.","section":"§II.A"},{"comment":"The caption uses \"Mean Average Position\" instead of \"Mean Average Precision (mAP)\".","section":"Figure 2 caption"},{"comment":"The last sentence of the DetectoRS subsection refers to \"the HTC model\" without any prior definition or connection to DetectoRS; this is confusing and should be corrected or removed.","section":"§IV.H"},{"comment":"The phrase \"the image likely compares\" is speculative and reads as placeholder text; the actual comparison shown in Figure 2 must be described explicitly.","section":"§V.A"}],"recommendation":"reject","confidential_remarks":"I agree with the reader's assessment. The manuscript reads as an early draft: it contains placeholder text in Section V.A, an unexplained mAP_L column of -1.000, a mis-cited DDOD reference, and no dataset or code release. The core empirical claim cannot be verified or reproduced, and the tracking experiments promised in the abstract are absent. These are not presentation-level fixes; they require the authors to conduct and report a complete, statistically meaningful experimental study. For a serious journal this falls below the acceptance threshold."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a short empirical report that subsamples about 1,000 images each of cars, ships, and airplanes from AiTODv2 and tiled Airbus imagery, converts them to COCO format, and runs a set of MMDetection models. The two things you should know up front: the dataset is not released and the training protocol is never specified, so the mAP table is essentially unverifiable; and the ByteTrack tracking study promised in the abstract simply is not there—Section V-A describes what a figure \"likely compares\" instead of showing any tracking results.\n\nWhat the paper does well is modest but real. The idea of a class-balanced, tiny-object-focused subset is sensible, and the authors do evaluate a reasonable spread of modern detectors (Faster R-CNN, RetinaNet, FCOS, VFNet, FoveaBox, RTMDet, DDOD, DetectoRS) on the same setup. The equations for focal loss, VFL, and FoveaBox are reproduced correctly, and the literature review, though shallow, touches the right reference points like SODA-A and AI-TOD-v2.\n\nThe soft spots are large. There is no train/val/test split, no epochs, learning rate, batch size, or augmentation details. mAP_L is -1.000 across every row, which is never explained (presumably no large objects, but the paper should say so). The DDOD citation is wrong: reference [56] is a paper on human-AI teams, not the object detector. The ByteTrack section is a placeholder, and the conclusion introduces SNIP and SNIPER without connecting them to the experiments. These are not nits; without the dataset and a documented protocol, the reported mAP numbers are not interpretable as a benchmark.\n\nFor whom is this paper? A reader who wants a rough sense of how standard detectors compare on a small custom subset might glance at Table 1, but there is nothing to act on because the data is withheld. The central contribution is the dataset, and the dataset is not available, so the paper cannot be independently checked. It deserves to be sent back for major revision: release the dataset, specify the split and hyperparameters, explain the mAP_L values, correct the DDOD reference, and either add the tracking experiments or remove that claim. As it stands, I would not send it to peer review; the flaws are load-bearing, not cosmetic.","headline":"Unreleased data and missing training details make the benchmark claims unverifiable, and the advertised tracking study is absent.","tokens_in":13314,"tokens_out":2777,"would_cite":false,"duration_ms":29202,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DDOD tops tiny-object benchmark on 3,000 satellite images","keywords":["small object detection","satellite imagery","dataset curation","object detection benchmark","deep learning","mAP evaluation","ByteTrack","aerial imagery"],"falsifier":"Release the curated dataset with per-image object-area histograms and an annotation audit; then retrain the same eight detectors on the original full AiTODv2 and Airbus sources. If the mAP ranking among detectors changes materially, the subset is not a faithful tiny-object benchmark.","tokens_in":12402,"feed_emoji":"🛰️","tokens_out":7166,"duration_ms":64200,"temperature":0.7,"pith_summary":"The paper argues that a carefully assembled 3,000-image satellite dataset of cars, ships, and airplanes can serve as a fair test bed for tiny-object detection, and it reports baseline mean-average-precision (mAP) results for eight state-of-the-art detectors. It also applies ByteTrack to satellite video segments to show tracking performance on the same object classes. The value would be a compact, resource-friendly benchmark that lets researchers compare models on small objects without the huge images and large instances that dominate existing aerial datasets. The reported numbers are offered as reference points for future work on small-object detection in satellite imagery.","feed_headline":"DDOD tops tiny-object benchmark on 3,000 satellite images","feed_subtitle":"Curated cars-ships-airplanes dataset gives baseline mAPs and shows anchor-free detectors lead on small objects.","key_machinery":"The operative mechanism is the dataset itself: roughly 1,000 images per class sampled from the AiTODv2 vehicle and ship classes and from tiled Airbus aircraft imagery, re-annotated to a common COCO-style format. All detectors are standard open-source implementations trained on a single consumer GPU. The comparison metric is COCO-style mAP at IoU thresholds 0.5, 0.75, and averaged over 0.5:0.95, with a size-stratified mAP_S/mAP_M/mAP_L split that makes the tiny-object challenge explicit.","core_discovery":"On the paper's own terms, the central discovery is that dense, anchor-free, and one-stage detectors outperform the classic two-stage Faster R-CNN on tiny satellite objects: DDOD with a ResNet50 and FPN backbone reaches the highest mAP (0.339), followed by FCOS (0.300), RetinaNet (0.296), and FoveaBox (0.290), while DetectoRS and VarifocalNet score below 0.20. The size-stratified mAP values show that small-object scores (mAP_S between 0.09 and 0.30) lag medium-object scores for every model, quantifying the tiny-object difficulty. All models list mAP_L as -1.000, indicating the benchmark contains no evaluated large objects, so the comparison is effectively about small and medium instances. The paper also applies ByteTrack to SAT-MTB video segments to assess satellite video object tracking for the same classes.","pith_inferences":["Because the dataset is not released and no per-class instance counts or object-area histograms are reported, the mAP rankings may not generalize to other tiny-object distributions; a testable extension is to publish the dataset statistics.","The -1.000 mAP_L values suggest the curated subset contains essentially no large objects, so the benchmark measures small/medium performance only; future work could add large instances to probe scale robustness.","The Airbus component is tiled from larger images, which can alter object appearance and context; re-evaluating on the original full-resolution tiles would show whether tiling biases the airplane class results.","A direct comparison against the full AiTODv2 and Airbus datasets would reveal whether the 3,000-image subset preserves the original difficulty or introduces sampling bias."],"forward_implications":["DDOD, FCOS, and RetinaNet with ResNet50 backbones give the strongest baselines for tiny satellite objects, all exceeding 0.29 mAP.","Two-stage Faster R-CNN variants perform worst (mAP 0.103), suggesting region-proposal pipelines are at a disadvantage when objects are only a few pixels wide.","The size-stratified mAP shows small-object scores (mAP_S between 0.09 and 0.30) are consistently well below medium-object scores, quantifying the tiny-object difficulty.","ByteTrack on SAT-MTB video segments provides a tracking baseline for cars, ships, and airplanes in satellite video.","A 3,000-image dataset can be trained on a single Tesla T4 GPU, making small-object detection research accessible in resource-constrained settings."],"supporting_citations":[{"why":"Supplies the AI-TOD-v2 dataset, from which the vehicle and ship images are sampled.","marker":"[17]"},{"why":"Defines Faster R-CNN, one of the evaluated two-stage detectors.","marker":"[3]"},{"why":"Introduces RetinaNet and Focal Loss, evaluated as a one-stage baseline.","marker":"[4]"},{"why":"Introduces FCOS, the anchor-free detector that scores second-best.","marker":"[45]"},{"why":"Introduces VarifocalNet, one of the evaluated detectors.","marker":"[48]"},{"why":"Introduces FoveaBox, the anchor-free detector evaluated.","marker":"[50]"},{"why":"Introduces RTMDet, the real-time detector family evaluated in four sizes.","marker":"[51]"},{"why":"Introduces DetectoRS with recursive feature pyramid and switchable atrous convolution, evaluated in two variants.","marker":"[58]"},{"why":"Provides the SAT-MTB satellite video dataset used for the ByteTrack tracking evaluation.","marker":"[34]"}],"fun_headline_variants":["DDOD tops tiny-object satellite benchmark with curated dataset","Anchor-free detectors outclass two-stage on tiny satellite objects","Curated dataset shows anchor-free wins for tiny satellite objects","DDOD leads on tiny satellite objects: cars, ships, planes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests entirely on the assumption that the 3,000-image subset is a representative, correctly annotated sample of tiny objects; the paper describes the dataset in three sentences and reports no object-size distributions, class counts, or annotation-quality checks.","fun_headline_variants_meta":{"raw":{"variants":["DDOD tops tiny-object satellite benchmark with curated dataset","Anchor-free detectors outclass two-stage on tiny satellite objects","Curated dataset shows anchor-free wins for tiny satellite objects","DDOD leads on tiny satellite objects: cars, ships, planes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000668,"raw_usage":{"total_tokens":3046,"prompt_tokens":941,"completion_tokens":2105,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":2037}},"tokens_in":557,"tokens_out":2105,"duration_ms":13690,"temperature":1.0,"reasoning_tokens":2037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:58.082530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Release the curated dataset with per-image object-area histograms and an annotation audit; then retrain the same eight detectors on the original full AiTODv2 and Airbus sources. If the mAP ranking among detectors changes materially, the subset is not a faithful tiny-object benchmark.","supporting_citations":[{"cited_title":"Y., Feichtenhofer, C., Darrell, T., & Xie, S","cited_arxiv_id":null,"evidence_quote":"Supplies the AI-TOD-v2 dataset, from which the vehicle and ship images are sampled."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Faster R-CNN, one of the evaluated two-stage detectors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces FCOS, the anchor-free detector that scores second-best."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces VarifocalNet, one of the evaluated detectors."},{"cited_title":"Y., & Berg, A","cited_arxiv_id":null,"evidence_quote":"Introduces FoveaBox, the anchor-free detector evaluated."},{"cited_title":"Y., Cubuk, E","cited_arxiv_id":null,"evidence_quote":"Introduces DetectoRS with recursive feature pyramid and switchable atrous convolution, evaluated in two variants."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SAT-MTB satellite video dataset used for the ByteTrack tracking evaluation."}],"review_version":1}