REVIEW 3 major objections 6 minor 12 references
Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper introduces DIOR, a 23,463-image, 192,472-instance benchmark covering 20 object classes, and claims it is the largest public optical remote-sensing detection dataset.
desk verdict DIOR is a genuinely useful remote sensing detection dataset, but an undisclosed train/test split and a concrete Table 2 error mean the benchmark claim needs major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the DIOR dataset itself: 23,463 optical remote sensing images with 192,472 horizontal bounding boxes spanning 20 classes, split evenly into trainval and test sets. The construction protocol — selecting classes with high inter-class similarity, collecting images across weathers, seasons, and resolutions, and manually annotating every instance with a box — is the mechanism that makes the scale and diversity claims testable. The evaluation machinery is the standard average-precision metric computed at a 50% box-overlap threshold, applied to 12 detectors to produce a baseline.
What would settle it
Have two independent annotators re-label a random 200-image sample of DIOR; if the average intersection-over-union between the original and re-annotated boxes falls below 0.5, or a nontrivial fraction of instances differs by more than one box, the ground-truth quality on which every benchmark number rests is not established.
Extended reading notes
Core claim
DIOR is a large-scale, publicly available benchmark for object detection in optical remote sensing images. It consists of 23,463 images of size 800x800 pixels, containing 192,472 manually annotated horizontal bounding boxes across 20 common geospatial object classes, with spatial resolutions ranging from 0.5m to 30m. The dataset was designed to have four properties: large scale in categories, images, and instances; a wide range of object size variations both within and across classes; rich image variations from different imaging conditions, weathers, seasons, and quality; and high inter-class similarity with high intra-class diversity. The paper benchmarks 12 representative deep learning detectors on a 50/50 trainval/test split, reporting the best mean average precision of 66.1% and identifying bridge, harbor, overpass, and vehicle as the hardest classes.
Load-bearing premise
The dataset's annotations are assumed to be accurate and complete, but the paper reports no inter-annotator agreement or label-error rate, so noisy or inconsistent labels would undermine every benchmark number built on DIOR.
Editorial extensions
If this is right
- Detectors can now be compared on a common 20-class test set with a standardized 50% overlap rule, making published remote-sensing detection results more directly reproducible.
- The dataset's mix of resolutions, weathers, seasons, and similar-looking classes provides a harder stress test for scale invariance and fine-grained discrimination than earlier remote-sensing benchmarks.
- The reported best mAP of 66.1% indicates substantial headroom, so DIOR can support future progress rather than saturating quickly.
- The low per-class accuracy on bridge, harbor, overpass, and vehicle points to specific failure modes for future methods to attack.
- Because the paper's own results show that feature pyramid networks and deeper backbones improve accuracy, multi-scale feature design is a promising direction for remote-sensing detectors.
Reading between the lines
- The "largest" claim is tied to the datasets compared in the paper; newer or differently annotated remote-sensing benchmarks released later could exceed DIOR, so the durable contribution is the fixed public test set rather than the size record.
- Because the paper reports no inter-annotator agreement or label-error rate, the integrity of the ground truth is an open question; re-annotating a random sample and measuring box overlap would establish it.
- The use of horizontal bounding boxes suits axis-aligned detectors but may disadvantage objects with strong orientation, such as ships; a rotated-box variant could produce different detector rankings.
- The baseline runs keep each detector's original settings, so a unified hyperparameter search might reorder the ranking on DIOR.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper has two parts: a survey of deep-learning-based object detection methods and datasets in both the computer vision and earth observation communities, and the introduction of a new benchmark dataset called DIOR (DetectIon in Optical Remote sensing images). DIOR is claimed to contain 23,463 images, 192,472 instances, and 20 object classes, with images of 800x800 pixels collected from Google Earth at spatial resolutions from 0.5m to 30m. The authors describe the dataset's characteristics (large scale, object size variation, image variation, inter-class similarity and intra-class diversity), report per-class instance and image counts, and benchmark 12 representative detectors (R-CNN, RICNN, RICAOD, RIFD-CNN, Faster R-CNN, Faster R-CNN with FPN, Mask R-CNN with FPN, PANet, RetinaNet, YOLOv3, SSD, CornerNet) using a roughly 50/50 trainval/test split. The central claim is that DIOR is the largest publicly available object detection benchmark for optical remote sensing images on both image count and category count, and that it provides a challenging testbed for future research.
Significance. If the DIOR dataset is as described and is released with a fixed evaluation protocol, this paper would make a substantial contribution to the remote sensing object detection community: it would provide a significantly larger and more diverse benchmark than existing datasets such as DOTA, NWPU VHR-10, and UCAS-AOD, with more categories and images, and a systematic comparison of 12 widely used detectors. The survey portion is also useful as a broad (if occasionally shallow) literature review. The paper's quantitative claims are measurements on a new dataset rather than the result of a fitted model, so there is no circularity concern in the usual sense. However, the benchmark's utility depends on the dataset being publicly accessible with a reproducible split and trustworthy annotations, and the paper currently does not establish these prerequisites fully.
major comments (3)
- [Table 2] Table 2 contains a concrete arithmetic inconsistency in the Stadium row: Train = 289 and Val = 292, but Trainval is reported as 851, whereas 289 + 292 = 581. The reported Trainval value is 270 images larger than the sum of its components, and no other row in the table shows such a discrepancy (e.g., Airplane 344+338=682, Bridge 379+495=874, Vehicle 1556+1558=3114). This error in the core statistics of the dataset undermines confidence in the headline numbers (23,463 images and 192,472 instances) and must be corrected or explicitly explained, and the paper should state whether the corrected value changes any of the benchmark results.
- [Section 5.1] The paper describes the trainval/test split as a random selection of 11,725 trainval and 11,738 test images, but it does not provide the actual image-to-subset mapping, a random seed, or any statement that the exact split is included in the released dataset. Without this information, the results in Table 3 cannot be reproduced and future methods cannot be compared under the same evaluation protocol, which directly undermines the paper's central claim that DIOR functions as a benchmark. The authors should release the exact split (or a reproducible procedure that generates it) and clearly document it in the paper.
- [Section 4.2] The paper states that experts manually labeled every instance with LabelMe, but it reports no annotation quality metrics, such as inter-annotator agreement, a labeling error rate, or a quality control procedure. Since the benchmark's value depends on the accuracy and completeness of the 192,472 annotations, the absence of any quantitative evidence about label quality is a significant gap. The authors should add annotation statistics, a description of the labeling protocol (including the number of annotators and how disagreements were resolved), and ideally a small error-rate study on a random subset.
minor comments (6)
- [Section 1] In the second paragraph of the Introduction, the phrase "the difference between remote sensing images and natural scene images is significant" is followed by a grammatically awkward sentence, "as we have pointed out above, earth observation images significantly differ from natural scene images is significant"; this should be rewritten for clarity.
- [Section 2.2.1] The text refers to "data argumentation methods" when the intended term is "data augmentation methods"; this typo appears in the SNIP/SNIPER paragraph.
- [Section 4.2] In the first paragraph of Section 4.2, the dataset is described as consisting of "optimal remote sensing images" but the correct word is "optical"; this typo should be corrected.
- [Table 1] In the DOTA row of Table 1, the year is listed as 2017 while the reference (Xia et al., 2018) is dated 2018; the authors should verify and align the year with the published version of the DOTA paper.
- [Section 5.2] The third observation in Section 5.2 states that YOLOv3 uses "the backbone network of Darknet-53" but earlier in the same paragraph it is written as "Darketnet-53"; the spelling should be made consistent.
- [Section 4.2] Figure 4 is said to show the object size distribution per class, but the text does not specify how object size is measured (e.g., box area, width, height); adding this definition would improve interpretability.
Circularity Check
No circularity found: DIOR's benchmark claim rests on measured dataset statistics and external method evaluations, not on fitted parameters or self-citation chains.
full rationale
The paper's central claim is that DIOR is a large-scale, publicly available object detection benchmark with 23,463 images, 192,472 instances, and 20 categories. These figures are presented as direct measurements of the constructed dataset in Section 4.2 and Table 1, not as outputs of a derivation that presupposes the conclusion. The benchmark evaluation in Section 5 applies twelve existing detectors to the new dataset; three of those detectors (RICNN, RICAOD, RIFD-CNN) are prior works by the same authors, but they are used as baseline inputs alongside external methods such as Faster R-CNN, SSD, YOLOv3, and CornerNet, and the reported AP numbers are empirically measured. The authors' own detectors are not cited to justify the dataset's validity or size. The object-class selection process is explicitly based on existing datasets and semantic criteria, but this is dataset design, not circular reasoning. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from self-citations to force a conclusion. The reproducibility concern about the unreleased train/test split and the arithmetic inconsistency in Table 2's Stadium row (train=289, val=292, trainval=851, yet 289+292=581) are correctness and documentation issues, not circularity. Accordingly, the circularity burden is minimal and the score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Manual annotations are accurate and complete.
- ad hoc to paper The 20 selected object classes form a representative benchmark set.
- domain assumption The train/test split is leak-free.
Cite this review
Pith. "Pith review of Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark." pith.science (2026). https://pith.science/paper/KW32FGJN
@misc{pith2026190900133,
author = {Pith},
title = {Pith review of: Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark},
year = {2026},
howpublished = {\url{https://pith.science/paper/KW32FGJN}},
note = {Machine review of arXiv:1909.00133}
}
read the original abstract
Substantial efforts have been devoted more recently to presenting various methods for object detection in optical remote sensing images. However, the current survey of datasets and deep learning based methods for object detection in optical remote sensing images is not adequate. Moreover, most of the existing datasets have some shortcomings, for example, the numbers of images and object categories are small scale, and the image diversity and variations are insufficient. These limitations greatly affect the development of deep learning based object detection methods. In the paper, we provide a comprehensive review of the recent deep learning based object detection progress in both the computer vision and earth observation communities. Then, we propose a large-scale, publicly available benchmark for object DetectIon in Optical Remote sensing images, which we name as DIOR. The dataset contains 23463 images and 192472 instances, covering 20 object classes. The proposed DIOR dataset 1) is large-scale on the object categories, on the object instance number, and on the total image number; 2) has a large range of object size variations, not only in terms of spatial resolutions, but also in the aspect of inter- and intra-class size variability across objects; 3) holds big variations as the images are obtained with different imaging conditions, weathers, seasons, and image quality; and 4) has high inter-class similarity and intra-class diversity. The proposed benchmark can help the researchers to develop and validate their data-driven methods. Finally, we evaluate several state-of-the-art approaches on our DIOR dataset to establish a baseline for future research.
Reference graph
Works this paper leans on
-
[7]
arXiv preprint arXiv:1711.07264
Light‐head r‐cnn: In defense of two‐stage object detector. arXiv preprint arXiv:1711.07264. Lin, H., Shi, Z., Zou, Z., 2017a. Fully Convolutional Network With Task Partitioning for Inshore Ship Detection in Optical Remote Sensing Images. IEEE Geosci. Remote Sens. Lett. 14, 1665‐1669. Lin, T.‐Y., Dollár, P., Girshick, R.B., He, K., Hariharan, B., Belongie,...
-
[9]
arXiv preprint arXiv:1804.02767
Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767. Ren, S., He, K., Girshick, R., Sun, J.,
- [10]
-
[12]
Object Detection Based on Sparse Representation and Hough Voting for Optical Remote Sensing Imagery. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 8, 2053‐2062. Yu, Y., Guan, H., Ji, Z.,
-
[1312]
SCRDet: Towards More Robust Detection for Small, Cluttered and Rotated Objects
Yang, J., Zhu, Y., Jiang, B., Gao, L., Xiao, L., Zheng, Z., 2018a. Aircraft detection in remote sensing images based on a deep residual network and Super‐Vector coding. Remote Sensing Letters 9, 229‐237. Yang, X., Fu, K., Sun, H., Yang, J., Guo, Z., Yan, M., Zhan, T., Xian, S., 2018b. R2CNN++: Multi‐Dimensional Attention Based Rotation Invariant Detector ...
-
[2010]
Photogrammetrie ‐ Fernerkundung ‐ Geoinformation 2010, 73‐82
The DGPF‐test on digital airborne camera evaluation‐overview and test design. Photogrammetrie ‐ Fernerkundung ‐ Geoinformation 2010, 73‐82. Dai, J., Li, Y., He, K., Sun, J.,
work page 2010
-
[2013]
Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1915‐1929. Cramer, M.,
work page 1915
-
[2014]
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. IEEE 17 Trans. Pattern Anal. Mach. Intell. 37, 1904‐1916. He, K., Zhang, X., Ren, S., Sun, J.,
work page 1904
Show all 12 references
-
[2015]
IEEE Geosci
Fast Multiclass Vehicle Detection on Aerial Images. IEEE Geosci. Remote Sens. Lett. 12, 1938‐1942. Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., Pietikäinen, M., 2018a. Deep learning for generic object detection: A survey. arXiv preprint arXiv:1809.02165. Liu,...
1938 arXiv
-
[2017]
arXiv preprint arXiv:1701.06659
DSSD: Deconvolutional single shot detector. arXiv preprint arXiv:1701.06659. Gidaris, S., Komodakis, N.,
-
[2018]
arXiv preprint arXiv:1809.03193
Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks. arXiv preprint arXiv:1809.03193. Aksoy, S.,
-
[2019]
IEEE Trans
Learning Rotation‐Invariant and Fisher Discriminative Convolutional Neural Networks for Object Detection. IEEE Trans. Image Process. 28, 265‐278. Cheng, G., Yang, C., Yao, X., Guo, L., Han, J., 2018a. When deep learning meets metric learning: remote sensing image scene classif...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.