{"id":"967f9648-7420-43a8-81fb-017824f8956f","arxiv_id":"1909.00133","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper combines a survey of deep learning object detection in remote sensing with a new large-scale benchmark dataset, DIOR, containing 23,463 images and 192,472 labeled instances across 20 classes.","lead":"This paper reviews deep learning methods for detecting objects in optical remote sensing images and introduces DIOR, a new benchmark dataset with 23,463 images and 192,472 labeled objects across 20 classes. The benchmark is meant to give the remote sensing community a larger, more diverse training and evaluation resource than earlier datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark claim undercut by unreleased test split and inconsistent Stadium counts in Table 2.","rationale":"The reader's verdict is CONDITIONAL, and our analysis does not change that: the paper should be accepted only after the dataset and benchmark protocol are verified. However, our most load-bearing concern differs from the reader's weakest assumption. The reader focused on annotation quality (no inter-annotator agreement, no error rate). That is a valid concern, but it is not the most immediate threat to the central claim. A benchmark is defined by a reproducible evaluation protocol; the paper never releases the train/test split, so even perfectly accurate annotations would not make DIOR usable as a benchmark. In addition, the internal arithmetic error in Table 2 for the Stadium class (train+val=581 but reported trainval=851) is a concrete, checkable flaw that indicates the dataset statistics may be unreliable. This is not an outside-consensus disagreement but a testable internal inconsistency. Therefore, the paper should remain CONDITIONAL, with the added explicit conditions that the split be released and the table be corrected. We partially agree with the reader because both concerns fall under dataset verification, but the specific locus is different.","tokens_in":35122,"tokens_out":6406,"duration_ms":128321,"concrete_test":"Download the DIOR release from the URL in footnote 2 and check whether it includes a fixed train/val/test split file or explicit image IDs for each subset. Then recompute the per-class image counts from the annotation files and verify the Stadium row: train (289) + val (292) should equal trainval (581), not 851. If the split file is absent, the benchmark protocol is not reproducible; if the count discrepancy persists, the dataset statistics contain an error that must be corrected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that DIOR is a large-scale, publicly available benchmark for object detection. For a dataset to function as a benchmark, the evaluation protocol must be fixed and reproducible. Section 5.1 states that 11,725 images were randomly selected as trainval and 11,738 as test, but the paper does not provide the actual split, a random seed, or any statement that the split is included in the release. Without the exact image-to-subset mapping, the results in Table 3 cannot be reproduced, and future methods cannot be compared under the same protocol. This directly undermines the 'benchmark' aspect of the central claim, regardless of annotation quality. Additionally, Table 2 contains a concrete arithmetic error: for Stadium, train=289, val=292, yet trainval=851; 289+292=581, so the reported trainval is 270 images too high. No other row in Table 2 has such a discrepancy. This inconsistency in the dataset's core statistics suggests the headline numbers (23,463 images, 192,472 instances) may not have been carefully verified, and it erodes confidence in the reliability of the benchmark's descriptive claims. Both issues are concrete, textually verifiable, and more immediately load-bearing than the reader's annotation-quality concern, because they block even the basic use of DIOR as a standardized benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper has two parts: a survey of deep-learning-based object detection methods and datasets in both the computer vision and earth observation communities, and the introduction of a new benchmark dataset called DIOR (DetectIon in Optical Remote sensing images). DIOR is claimed to contain 23,463 images, 192,472 instances, and 20 object classes, with images of 800x800 pixels collected from Google Earth at spatial resolutions from 0.5m to 30m. The authors describe the dataset's characteristics (large scale, object size variation, image variation, inter-class similarity and intra-class diversity), report per-class instance and image counts, and benchmark 12 representative detectors (R-CNN, RICNN, RICAOD, RIFD-CNN, Faster R-CNN, Faster R-CNN with FPN, Mask R-CNN with FPN, PANet, RetinaNet, YOLOv3, SSD, CornerNet) using a roughly 50/50 trainval/test split. The central claim is that DIOR is the largest publicly available object detection benchmark for optical remote sensing images on both image count and category count, and that it provides a challenging testbed for future research.","tokens_in":35375,"tokens_out":2156,"duration_ms":21124,"significance":"If the DIOR dataset is as described and is released with a fixed evaluation protocol, this paper would make a substantial contribution to the remote sensing object detection community: it would provide a significantly larger and more diverse benchmark than existing datasets such as DOTA, NWPU VHR-10, and UCAS-AOD, with more categories and images, and a systematic comparison of 12 widely used detectors. The survey portion is also useful as a broad (if occasionally shallow) literature review. The paper's quantitative claims are measurements on a new dataset rather than the result of a fitted model, so there is no circularity concern in the usual sense. However, the benchmark's utility depends on the dataset being publicly accessible with a reproducible split and trustworthy annotations, and the paper currently does not establish these prerequisites fully.","major_comments":[{"comment":"Table 2 contains a concrete arithmetic inconsistency in the Stadium row: Train = 289 and Val = 292, but Trainval is reported as 851, whereas 289 + 292 = 581. The reported Trainval value is 270 images larger than the sum of its components, and no other row in the table shows such a discrepancy (e.g., Airplane 344+338=682, Bridge 379+495=874, Vehicle 1556+1558=3114). This error in the core statistics of the dataset undermines confidence in the headline numbers (23,463 images and 192,472 instances) and must be corrected or explicitly explained, and the paper should state whether the corrected value changes any of the benchmark results.","section":"Table 2"},{"comment":"The paper describes the trainval/test split as a random selection of 11,725 trainval and 11,738 test images, but it does not provide the actual image-to-subset mapping, a random seed, or any statement that the exact split is included in the released dataset. Without this information, the results in Table 3 cannot be reproduced and future methods cannot be compared under the same evaluation protocol, which directly undermines the paper's central claim that DIOR functions as a benchmark. The authors should release the exact split (or a reproducible procedure that generates it) and clearly document it in the paper.","section":"Section 5.1"},{"comment":"The paper states that experts manually labeled every instance with LabelMe, but it reports no annotation quality metrics, such as inter-annotator agreement, a labeling error rate, or a quality control procedure. Since the benchmark's value depends on the accuracy and completeness of the 192,472 annotations, the absence of any quantitative evidence about label quality is a significant gap. The authors should add annotation statistics, a description of the labeling protocol (including the number of annotators and how disagreements were resolved), and ideally a small error-rate study on a random subset.","section":"Section 4.2"}],"minor_comments":[{"comment":"In the second paragraph of the Introduction, the phrase \"the difference between remote sensing images and natural scene images is significant\" is followed by a grammatically awkward sentence, \"as we have pointed out above, earth observation images significantly differ from natural scene images is significant\"; this should be rewritten for clarity.","section":"Section 1"},{"comment":"The text refers to \"data argumentation methods\" when the intended term is \"data augmentation methods\"; this typo appears in the SNIP/SNIPER paragraph.","section":"Section 2.2.1"},{"comment":"In the first paragraph of Section 4.2, the dataset is described as consisting of \"optimal remote sensing images\" but the correct word is \"optical\"; this typo should be corrected.","section":"Section 4.2"},{"comment":"In the DOTA row of Table 1, the year is listed as 2017 while the reference (Xia et al., 2018) is dated 2018; the authors should verify and align the year with the published version of the DOTA paper.","section":"Table 1"},{"comment":"The third observation in Section 5.2 states that YOLOv3 uses \"the backbone network of Darknet-53\" but earlier in the same paragraph it is written as \"Darketnet-53\"; the spelling should be made consistent.","section":"Section 5.2"},{"comment":"Figure 4 is said to show the object size distribution per class, but the text does not specify how object size is measured (e.g., box area, width, height); adding this definition would improve interpretability.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim, that DIOR is the largest publicly available object detection benchmark for optical remote sensing images, is plausible and potentially significant, but the manuscript currently has two load-bearing issues: an arithmetic inconsistency in Table 2 and an unreleased trainval/test split that prevents reproducibility of the benchmark results. Both are fixable within the scope of a revision, but they need to be addressed before the paper can be accepted. I would also encourage the editor to request that the authors confirm the dataset URL and the exact contents of the public release, since the paper does not state whether the split is included. The survey portion, while broad, is not the main contribution and should not be the focus of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. The DIOR dataset is real and genuinely useful: 20 classes, 23,463 images, 192,472 instances, collected from Google Earth with horizontal boxes. That is a real step up from DOTA in category count and image count. The survey half is competent but mostly well-trodden ground; the benchmark half is a standard application of off-the-shelf detectors to the new data. The new asset is the dataset itself.\n\nCredit where due: the class selection is thoughtful, the statistics mostly hang together, and the figures showing size variation and inter-class similarity make the case that DIOR is harder and richer than earlier sets. The authors clearly put in real annotation effort. If the dataset is released as described, it will be a useful training and evaluation resource for the remote sensing detection community.\n\nNow the soft spots, and I think the stress-test note has the right priorities. Section 5.1 says 11,725 images were randomly selected as trainval and 11,738 as test, but the paper never says the split is included in the release, nor gives a seed. Without the exact image-to-subset mapping, the numbers in Table 3 cannot be reproduced and no future method can be compared under the same protocol. That directly undercuts the 'publicly available benchmark' claim. This is fixable but essential.\n\nSecond, Table 2 has a concrete arithmetic error: Stadium train=289, val=292, but trainval=851. 289+292=581. Either trainval is wrong or one of the other numbers is. No other row has such a discrepancy. It is a small error, but it suggests the headline statistics were not carefully verified, and it makes you wonder about the 23,463 / 192,472 numbers. The reader's annotation-quality concern is real but secondary; no inter-annotator agreement or error rate is reported, which is common for remote sensing datasets but should be addressed.\n\nThe central idea holds: a large, diverse, publicly available detection benchmark for optical remote sensing is needed, and DIOR is a plausible candidate. But the paper in this form is not a reproducible benchmark until the split is released and the table fixed. For a serious read: this deserves peer review, not a desk reject, because the dataset itself is valuable. It needs major revision before acceptance. I would bring it to the reading group to discuss the split issue and what dataset papers owe the community.","headline":"DIOR is a genuinely useful remote sensing detection dataset, but an undisclosed train/test split and a concrete Table 2 error mean the benchmark claim needs major revision.","tokens_in":35834,"tokens_out":2931,"would_cite":true,"duration_ms":27138,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces DIOR, a 23,463-image, 192,472-instance benchmark covering 20 object classes, and claims it is the largest public optical remote-sensing detection dataset.","keywords":["object detection","optical remote sensing","benchmark dataset","DIOR","deep learning","aerial imagery","bounding box annotation","mean average precision"],"falsifier":"Have two independent annotators re-label a random 200-image sample of DIOR; if the average intersection-over-union between the original and re-annotated boxes falls below 0.5, or a nontrivial fraction of instances differs by more than one box, the ground-truth quality on which every benchmark number rests is not established.","tokens_in":34965,"feed_emoji":"🛰️","tokens_out":6897,"duration_ms":62761,"temperature":0.7,"pith_summary":"This paper argues that deep learning object detectors for optical remote sensing images have been held back by small, low-diversity datasets, and it responds with a new public benchmark, DIOR. The paper's central claim is that DIOR is the largest publicly available dataset of its kind, containing 23,463 images, 192,472 labeled instances, and 20 object classes. If that claim holds, DIOR gives the community a common training and evaluation ground with far more images than the previous large-scale option, plus deliberate variation in object size, weather, season, and image quality. The paper also reports a baseline evaluation of 12 detectors, with the best reaching 66.1% mean average precision, which shows the benchmark is challenging rather than saturated.","feed_headline":"23,463-image DIOR benchmark sets a larger stage for aerial detection","feed_subtitle":"192,472 labeled instances across 20 classes give detectors a harder, more diverse test.","key_machinery":"The central object is the DIOR dataset itself: 23,463 optical remote sensing images with 192,472 horizontal bounding boxes spanning 20 classes, split evenly into trainval and test sets. The construction protocol — selecting classes with high inter-class similarity, collecting images across weathers, seasons, and resolutions, and manually annotating every instance with a box — is the mechanism that makes the scale and diversity claims testable. The evaluation machinery is the standard average-precision metric computed at a 50% box-overlap threshold, applied to 12 detectors to produce a baseline.","core_discovery":"DIOR is a large-scale, publicly available benchmark for object detection in optical remote sensing images. It consists of 23,463 images of size 800x800 pixels, containing 192,472 manually annotated horizontal bounding boxes across 20 common geospatial object classes, with spatial resolutions ranging from 0.5m to 30m. The dataset was designed to have four properties: large scale in categories, images, and instances; a wide range of object size variations both within and across classes; rich image variations from different imaging conditions, weathers, seasons, and quality; and high inter-class similarity with high intra-class diversity. The paper benchmarks 12 representative deep learning detectors on a 50/50 trainval/test split, reporting the best mean average precision of 66.1% and identifying bridge, harbor, overpass, and vehicle as the hardest classes.","pith_inferences":["The \"largest\" claim is tied to the datasets compared in the paper; newer or differently annotated remote-sensing benchmarks released later could exceed DIOR, so the durable contribution is the fixed public test set rather than the size record.","Because the paper reports no inter-annotator agreement or label-error rate, the integrity of the ground truth is an open question; re-annotating a random sample and measuring box overlap would establish it.","The use of horizontal bounding boxes suits axis-aligned detectors but may disadvantage objects with strong orientation, such as ships; a rotated-box variant could produce different detector rankings.","The baseline runs keep each detector's original settings, so a unified hyperparameter search might reorder the ranking on DIOR."],"forward_implications":["Detectors can now be compared on a common 20-class test set with a standardized 50% overlap rule, making published remote-sensing detection results more directly reproducible.","The dataset's mix of resolutions, weathers, seasons, and similar-looking classes provides a harder stress test for scale invariance and fine-grained discrimination than earlier remote-sensing benchmarks.","The reported best mAP of 66.1% indicates substantial headroom, so DIOR can support future progress rather than saturating quickly.","The low per-class accuracy on bridge, harbor, overpass, and vehicle points to specific failure modes for future methods to attack.","Because the paper's own results show that feature pyramid networks and deeper backbones improve accuracy, multi-scale feature design is a promising direction for remote-sensing detectors."],"supporting_citations":[{"why":"Supplies DOTA, the closest large-scale prior benchmark that DIOR must surpass in image count and category number.","marker":"Xia et al., 2018"},{"why":"Provides NWPU VHR-10, a widely used 10-class dataset from which DIOR's common classes and annotation style are drawn.","marker":"Cheng et al., 2016a"},{"why":"Offers UCAS-AOD, a two-class aerial detection dataset used in the comparison table for scale.","marker":"Zhu et al., 2015a"},{"why":"Defines the PASCAL VOC average-precision protocol and horizontal bounding box standard adopted for DIOR evaluation.","marker":"Everingham et al., 2010"},{"why":"Motivates large-scale dataset construction through MSCOCO, cited as the natural-image counterpart DIOR emulates.","marker":"Lin et al., 2014"},{"why":"Faster R-CNN is one of the benchmarked detectors and a backbone-agnostic baseline on DIOR.","marker":"Ren et al., 2017"},{"why":"FPN is the feature pyramid component that the paper credits with large accuracy gains on DIOR.","marker":"Lin et al., 2017b"},{"why":"RetinaNet is one of the top-scoring benchmarked detectors, reaching the best mAP with ResNet-101.","marker":"Lin et al., 2017c"},{"why":"YOLOv3 is the benchmarked detector that performs best on small objects such as ships and vehicles on DIOR.","marker":"Redmon and Farhadi, 2018"},{"why":"LabelMe is the annotation tool used to manually draw the horizontal boxes that form DIOR's ground truth.","marker":"Russell et al., 2008"}],"fun_headline_variants":["DIOR benchmark: 23k images, 20 classes, 192k instances for aerial detection","Aerial detection's new stress test: DIOR's 20 diverse classes","DIOR: 192k instances, 20 classes, 0.5m-30m resolution range","Benchmarking detectors on DIOR: hardest classes are bridge, harbor, overpass","DIOR sets higher bar for aerial detection with 23k images, 20 classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's annotations are assumed to be accurate and complete, but the paper reports no inter-annotator agreement or label-error rate, so noisy or inconsistent labels would undermine every benchmark number built on DIOR.","fun_headline_variants_meta":{"raw":{"variants":["DIOR benchmark: 23k images, 20 classes, 192k instances for aerial detection","Aerial detection's new stress test: DIOR's 20 diverse classes","DIOR: 192k instances, 20 classes, 0.5m-30m resolution range","Benchmarking detectors on DIOR: hardest classes are bridge, harbor, overpass","DIOR sets higher bar for aerial detection with 23k images, 20 classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001368,"raw_usage":{"total_tokens":5575,"prompt_tokens":999,"completion_tokens":4576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":4459}},"tokens_in":615,"tokens_out":4576,"duration_ms":26790,"temperature":1.0,"reasoning_tokens":4459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T06:00:14.525647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent annotators re-label a random 200-image sample of DIOR; if the average intersection-over-union between the original and re-annotated boxes falls below 0.5, or a nontrivial fraction of instances differs by more than one box, the ground-truth quality on which every benchmark number rests is not established.","supporting_citations":[],"review_version":1}