Pith. sign in

REVIEW 4 major objections 7 minor 3 cited by

A Simple Aerial Detection Baseline of Multimodal Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read After converting oriented boxes into text and fine-tuning, a multimodal language model performs aerial detection on par with conventional rotated detectors, measured without confidence scores.

desk verdict A useful, reproducible first baseline for turning general-purpose MLMs into oriented aerial detectors, but the headline parity claim rests on a nonstandard confidence-free metric and never reports standard mAP. read the letter →

arxiv 2501.09720 v3 pith:24IFQGZD submitted 2025-01-16 cs.CV cs.AI

Qingyun Li , Yushi Chen , Xinya Shu , Dong Chen , Xin He , Yi Yu , Xue Yang This is my paper · ORCID
classification cs.CVcs.AI
keywords multimodallanguagemodelaerialdetectionremotesensingorientedobjecttextnormalizationconfidence-freeevaluationfine-tuningrotatedboundingbox
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a generative multimodal language model can perform multi-class oriented aerial detection at a level comparable to conventional detectors, if the detection outputs are serialized as text and the comparison is made with confidence-free metrics. The authors fine-tune a general-purpose MLM (Florence-2) on five remote-sensing benchmarks by converting each oriented box into quantized integer coordinates and category names sorted alphabetically, then parsing the generated text with regex. They argue that confidence scores, which MLMs do not produce, are not essential for measuring detection quality, and they introduce mAPnc and mF1, which assign a constant confidence to every prediction and select the best threshold for conventional detectors on the evaluation set. Under these metrics the fine-tuned MLMs match or exceed rotated RetinaNet and rotated FCOS on most benchmarks. The paper is best read as a proof-of-concept baseline: MLMs can be instruction-tuned into exhaustive aerial detectors, not just grounders that localize a single mentioned object.

What carries the argument

The central mechanism is a normalization-and-parsing pipeline that turns the detection task into a text-generation task. Each oriented box is represented as a category phrase plus eight vertex coordinates quantized to integers in [0,1000], with objects ordered alphabetically by category and then by the position of the starting vertex; the MLM is fine-tuned to produce this fixed template in response to a detection instruction, and the outputs are recovered by regex extraction and fuzzy category matching using Levenshtein distance. This machinery carries the argument because it makes the autoregressive next-token objective directly applicable to exhaustive detection, and the proposed confidence-free metrics (mAPnc and mF1) carry the evaluation, since they remove the reliance on confidence scores that MLMs do not output.

What would settle it

Re-running Table I with standard mAP (which ranks predictions by confidence) on the held-out splits would show whether the claimed parity holds when confidence is required; if the MLM detectors fall far behind both rotated detectors on standard mAP, the comparability claim would be refuted under the conventional metric.

Watch

Extended reading notes

Core claim

The paper's central claim is that the autoregressive text-generation paradigm is not a barrier to dense multi-class detection. By normalizing each oriented object into the text tuple of a category name followed by eight integer vertex coordinates in the range 0 to 1000, sorting categories alphabetically and boxes by their starting vertex, and fine-tuning a pre-trained MLM with the standard next-token cross-entropy loss, the model learns to emit a complete, parsable list of all objects in an image. On DOTA-v1.0, DIOR-R, FAIR1M-v1.0, SRSDD and RSAR, the fine-tuned model achieves mAPnc and mF1 values comparable to rotated RetinaNet and rotated FCOS, and even surpasses them on several datasets under these metrics. The paper additionally claims that confidence scores are a byproduct of conventional detectors rather than an intrinsic part of the detection task, and therefore proposes confidence-free evaluation (mAPnc and mF1) as the fair basis for comparing MLM and conventional detectors.

Load-bearing premise

The comparison holds only if detection quality is measured without confidence scores, where each conventional detector's best confidence threshold is chosen using the evaluation set itself.

Editorial extensions

If this is right

  • A single fine-tuned MLM can act as both a visual grounder and an exhaustive multi-class detector, because both tasks share the same text interface.
  • The text-based normalization makes multi-dataset joint training straightforward: annotations from different datasets can be merged directly, and joint training improves performance on the smallest dataset (SRSDD).
  • Confidence-free metrics (mAPnc and mF1) provide a way to compare detection models that do not output confidence, and the authors recommend mAPnc for online benchmarks that currently require mAP.
  • MLM detectors need no non-maximum suppression, since the text generation naturally produces a set of objects without overlapping duplicates, simplifying the inference pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to measure the same fine-tuned MLM under standard mAP with confidence produced by auxiliary means (e.g., token likelihood); if the gap to conventional detectors narrows, the confidence-free metrics may have overstated parity.
  • The threshold selection protocol for conventional detectors (best threshold on the evaluation set) is optimistic; using a validation split to pick thresholds would likely reduce the reported mAPnc and mF1 of the baselines and change the comparison.
  • The serialization idea could transfer to other structured prediction tasks such as instance segmentation or keypoint detection, where outputs are also sets of coordinates that can be flattened into text.
  • Scaling the vision encoder or input resolution, together with more joint training data, is a testable route to closing any residual gap with conventional detectors under confidence-free metrics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes LMMRotate, a baseline that adapts open-source multimodal language models (MLMs) to multi-class oriented aerial detection by normalizing detection outputs into quantized textual responses and fine-tuning with standard next-token prediction. The authors introduce a confidence-free evaluation scheme (mAPnc and mF1) intended to make comparisons with conventional detectors fair, and they report results on DOTA-v1.0, DIOR-R, FAIR1M-v1.0, SRSDD, and RSAR using Florence-2-base and Florence-2-large. The central claim is that, after simple output normalization and fine-tuning, MLMs achieve detection performance comparable to conventional detectors, as summarized in the abstract and Table I.

Significance. If the comparability claim is upheld, the paper would be a useful practical baseline for applying MLMs to dense multi-class aerial detection, a task that existing RS MLMs have not directly addressed. The work provides an open-source implementation, explores joint training across multiple datasets, and demonstrates that the output-format gap between autoregressive text generation and detection can be bridged with a simple normalization. These are concrete contributions. However, the key evaluation claim is currently supported only by author-defined confidence-free metrics whose construction includes an evaluation-set oracle for conventional detectors, and the community-standard mAP is never reported. The significance of the result therefore hinges on the validity and acceptance of the proposed evaluation protocol, which the manuscript does not yet convincingly justify.

major comments (4)
  1. [II-C and III-B, Table I] The central claim of 'performance comparable to conventional detector' rests entirely on the author-defined metrics mAPnc and mF1. For conventional detectors, the confidence threshold is selected by sweeping a range of thresholds on the evaluation set and picking the best score (Section III-B: 'calculate the two types of scores under a range of confidence thresholds and then select the best scores'). Standard mAP is never reported in Table I. This means the comparison is not made under the metric the community actually uses for these benchmarks. The authors should report standard mAP on the official test servers for DOTA-v1.0 and FAIR1M-v1.0 (which are available), or provide a rigorous argument explaining why mAPnc is the appropriate metric and why the oracle threshold selection does not invalidate the comparison.
  2. [II-C, Figure 4] The definition of mAPnc is incomplete and internally problematic. The metric is described as replacing confidence with constant or random scores, yet the threshold filtering used in Figure 4 still depends on the original confidence scores of the conventional detectors. When all confidence scores are replaced by a constant value, the precision-recall curve is not well-defined because the ranking of detections becomes arbitrary unless ties are broken by an unspecified rule. The text also claims that 'the consideration of confidence is not necessary' for comparing MLMs and conventional detectors, but this is contradicted by the metric's own threshold-selection step, which uses confidence implicitly. Please provide a formal definition of mAPnc, specify tie-breaking, and clarify how the threshold sweep is compatible with the claim that confidence is irrelevant.
  3. [III-B] For DOTA-v1.0 and FAIR1M-v1.0, mF1 is computed on re-purposed validation splits because the official test servers only accept mAP. Specifically, the authors retrain DOTA models on the training set and evaluate on the validation set, and for FAIR1M they evaluate on the FAIR1M-v2.0 validation set. These entries are therefore not directly comparable to the test-set mF1 values reported for the other datasets, nor to published results. The table should clearly mark these entries as validation-set numbers, and the authors should consider reporting the standard mAP from the official servers as the primary comparison.
  4. [II-C and Figure 4] The robustness argument for mAPnc is insufficient. The claim that standard deviations are 'generally lower than 0.5%' is based on replacing confidence with ten random values and one constant value, but this does not address the main concern: the best confidence threshold for each conventional detector is selected using the evaluation set. Since the MLM detections are all assigned 100% confidence and are not subject to threshold selection, the comparison protocol gives conventional detectors an oracle advantage. The error-band analysis in Figure 4 also appears to be performed on conventional detectors only; the authors should explain how this demonstrates robustness of the metric for the MLM comparison.
minor comments (7)
  1. [Abstract] There is a grammar error: 'a evaluation method' should be 'an evaluation method'.
  2. [II-B] The word 'qualitize' should be 'quantize'.
  3. [II-A] In the sentence 'predicting tokens iteratively, .' there is a stray comma before the period; please remove it.
  4. [II-C and Table I] The notation is inconsistent: the text uses 'mF 1', 'mAP nc', and 'mAPnc' interchangeably. Please standardize the notation.
  5. [II-C] The definition of mAPnc is given only in the caption of Figure 4. A formal algorithm or equation should be included in Section II-C so the metric is unambiguous.
  6. [III-D] There are typos in the last paragraph: 'especally' should be 'especially' and 'benifits' should be 'benefits'.
  7. [III-D, Table I] The comparison includes only two conventional detectors, Rotated RetinaNet and Rotated FCOS. Adding a stronger contemporary oriented detector would strengthen the claim that performance is comparable to the current state of the art.

Circularity Check

1 steps flagged · score 6.0 of 10

The claimed parity with conventional detectors is operationalized by an author-defined metric in which each conventional detector's reported mAPnc/mF1 is the maximum over an evaluation-set confidence-threshold sweep, while standard mAP is never reported.

  1. fitted input called prediction [Section II-C and Section III-B; Table I]
    "We calculate the two types of scores under a range of confidence thresholds and then select the best scores."

    The conventional-detector scores in Table I are, by construction, the best mAPnc/mF1 obtained by sweeping a confidence threshold on the evaluation split (Section III-B), rather than a fixed prediction at a chosen operating point. Section II-C similarly states: 'we conduct threshold filtering to enhance the mAPnc and set the best result as the mAPnc of conventional detectors.' These threshold-fitted values are then used to support the paper's central claim that MLM detectors achieve performance 'comparable to conventional detector,' while no standard mAP is reported for either family.

full rationale

The paper has no analytical derivation chain; its core contribution is empirical fine-tuning of Florence-2 with textual detection outputs. However, the headline comparison is not evaluated under the field-standard mAP but under the paper's mAPnc/mF1 metrics. For every conventional detector, the reported score is the maximum over a confidence-threshold sweep performed on the evaluation data (Section III-B), and standard mAP is absent from Table I. This makes the 'comparable to conventional detector' claim depend, for the conventional side, on a threshold fitted to the evaluation set rather than on a fixed prediction. This is a partial circularity in the evaluation protocol: the chosen metric, combined with best-threshold selection, constructs the conventional baseline against which the MLM is compared. The fine-tuned MLM results themselves are external and reproducible, so the circularity is confined to the comparison claim rather than to the detection outputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper contributes no new physical entities and no derivation. Its central comparison rests on hand-chosen evaluation choices: a coordinate quantization scale, a constant confidence assignment for MLM outputs, and especially the selection of the best confidence threshold for conventional detectors on evaluation labels. The fairness claim further assumes that confidence-free metrics are valid substitutes for mAP.

free parameters (3)
  • Coordinate quantization scale = 1000
    Section II-B sets the quantization to 1000 bins per axis following Florence-2, asserting negligible precision loss without an ablation; the method depends on this scale.
  • MLM confidence score assignment = 1.0 (constant)
    In Section III-B all MLM predictions are assigned 100% confidence. This constant removes confidence-based ranking and is a necessary input to the proposed mAPnc evaluation.
  • Best confidence threshold for conventional detectors = not reported; selected per dataset
    In Section III-B the authors 'calculate the two types of scores under a range of confidence thresholds and then select the best scores,' using evaluation-set labels to pick the threshold. This is a test-set-fitted parameter in the comparison.
assumptions (3)
  • domain assumption Quantized integer coordinates cause negligible precision loss for oriented detection.
    Stated in Section II-B ('The precision loss introduced by the quantized integer coordinates... has a negligible impact on location accuracy') without quantitative evidence; the claim's validity depends on this.
  • domain assumption Confidence scores are not essential for evaluating detector quality, so mAPnc/mF1 can replace mAP.
    Section II-C argues confidence is a byproduct and not part of detection annotations. This premise is load-bearing for the fairness claim; standard benchmark protocols disagree.
  • domain assumption Levenshtein fuzzy matching can map predicted category names to meta categories without a tuned threshold.
    Section III-B uses fuzzy matching for cases like 'pool' vs 'swimming pool' but does not state the distance threshold or its effect on the results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Simple Aerial Detection Baseline of Multimodal Language Models." pith.science (2026). https://pith.science/paper/24IFQGZD

@misc{pith2026250109720,
  author       = {Pith},
  title        = {Pith review of: A Simple Aerial Detection Baseline of Multimodal Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24IFQGZD}},
  note         = {Machine review of arXiv:2501.09720}
}
read the original abstract

The multimodal language models (MLMs) based on generative pre-trained Transformer are considered powerful candidates for unifying various domains and tasks. MLMs developed for remote sensing (RS) have demonstrated outstanding performance in multiple tasks, such as visual question answering and visual grounding. In addition to visual grounding that detects specific objects corresponded to given instruction, aerial detection, which detects all objects of multiple categories, is also a valuable and challenging task for RS foundation models. However, aerial detection has not been explored by existing RS MLMs because the autoregressive prediction mechanism of MLMs differs significantly from the detection outputs. In this paper, we present a simple baseline for applying MLMs to aerial detection for the first time, named LMMRotate. Specifically, we first introduce a normalization method to transform detection outputs into textual outputs to be compatible with the MLM framework. Then, we propose a evaluation method, which ensures a fair comparison between MLMs and conventional object detection models. We construct the baseline by fine-tuning open-source general-purpose MLMs and achieve impressive detection performance comparable to conventional detector. We hope that this baseline will serve as a reference for future MLM development, enabling more comprehensive capabilities for understanding RS images. Code is available at https://github.com/Li-Qingyun/mllm-mmrotate.

Figures

Figures reproduced from arXiv: 2501.09720 by the authors.

Figure 1
Figure 1. Visualization of the objects detected by our MLM detector based [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. An example of a RS image and its response that contains category [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 2
Figure 2. The overall framework of the proposed MLM detector baseline. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The impact of confidence scores on mAP / mAP [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Point2RBox-v2: Rethinking Point-supervised Oriented Object Detection with Spatial Layout Among Instances

    cs.CV 2025-02 conditional novelty 7.0 of 10

    Point2RBox-v2 uses Gaussian overlap, Voronoi watershed, edge, and consistency losses to learn oriented boxes from point annotations, reaching 62.61 AP50 on DOTA-v1.0.

  2. Wholly-WOOD: Wholly Leveraging Diversified-quality Labels for Weakly-supervised Oriented Object Detection

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A unified weakly-supervised framework that, using only horizontal boxes or points, matches or approaches fully RBox-supervised oriented detectors, with point-supervised DOTA-v1.0 AP50 of 62.63, far above prior cited p...

  3. PointOBB-v3: Expanding Performance Boundaries of Single Point-Supervised Oriented Object Detection

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A single point-supervised oriented object detection framework combines scale consistency and symmetry-based angle learning to set new state-of-the-art results on seven aerial benchmarks.

Reference graph

Works this paper leans on

14 extracted references · 10 canonical work pages · cited by 3 Pith papers

  1. [1]

    Towards vision- language geo-foundation models: A survey,

    Y . Zhou, L. Feng, Y . Ke et al. , “Towards vision- language geo-foundation models: A survey,” arXiv preprint arXiv:2406.09385, 2024. 1

    arXiv 2024

  2. [2]

    Geochat: Grounded large vision-language model for remote sensing,

    K. Kuckreja, M. S. Danish, M. Naseer et al. , “Geochat: Grounded large vision-language model for remote sensing,” CVPR, 2024. 1

    work page 2024

  3. [3]

    Earthgpt: A universal multimodal large language model for multisensor image comprehension in remote sensing domain,

    W. Zhang, M. Cai, T. Zhang et al. , “Earthgpt: A universal multimodal large language model for multisensor image comprehension in remote sensing domain,” IEEE Trans. Geosci. Remote Sens. , vol. 62, pp. 1–20, 2024. 1

    work page 2024

  4. [4]

    Skysensegpt: A fine-grained instruction tun- ing dataset and model for remote sensing vision-language understanding,

    J. Luo et al. , “Skysensegpt: A fine-grained instruction tun- ing dataset and model for remote sensing vision-language understanding,” arXiv preprint arXiv:2406.10100 , 2024. 1

    arXiv 2024

  5. [5]

    Florence-2: Advancing a unified representation for a variety of vision tasks,

    B. Xiao, H. Wu, W. Xu et al. , “Florence-2: Advancing a unified representation for a variety of vision tasks,” in CVPR, June 2024, pp. 4818–4829. 1, 2, 3, 4

    work page 2024

  6. [6]

    Dota: A large-scale dataset for object detection in aerial images,

    G.-S. Xia, X. Bai, J. Ding et al., “Dota: A large-scale dataset for object detection in aerial images,” in CVPR, June 2018. 1, 3, 4

    work page 2018

  7. [7]

    Rsar: Restricted state angle resolver and rotated sar benchmark,

    X. Zhang, X. Yang, Y . Li et al. , “Rsar: Restricted state angle resolver and rotated sar benchmark,” arXiv preprint arXiv:2501.04440, 2025. 1, 3, 4

    arXiv 2025

  8. [8]

    Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling,

    Z. Chen, W. Wang, Y . Cao et al. , “Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling,” arXiv preprint arXiv:2412.05271, 2024. 2

    arXiv 2024

Show all 14 references
  1. [9]

    Fair1m: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery,

    X. Sun, P. Wang, Z. Yan et al. , “Fair1m: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery,” ISPRS J. Photogram. Remote Sens., vol. 184, pp. 116–130, 2022. 3, 4

    2022

  2. [10]

    Object detection in optical remote sensing im- ages: A survey and a new benchmark,

    K. Li et al., “Object detection in optical remote sensing im- ages: A survey and a new benchmark,” ISPRS J. Photogram. Remote Sens. , vol. 159, pp. 296–307, 2020. 3, 4

    2020

  3. [11]

    Srsdd-v1. 0: A high-resolution sar rotation ship detection dataset,

    S. Lei, D. Lu, X. Qiu, and C. Ding, “Srsdd-v1. 0: A high-resolution sar rotation ship detection dataset,” Remote Sensing, vol. 13, no. 24, p. 5104, 2021. 3, 4

    2021

  4. [12]

    Focal loss for dense object detection,

    T.-Y . Ross and G. Doll ´ar, “Focal loss for dense object detection,” in CVPR, 2017, pp. 2980–2988. 4

    2017

  5. [13]

    Fcos: A simple and strong anchor-free object detector,

    Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: A simple and strong anchor-free object detector,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 44, no. 4, pp. 1922–1933, 2020. 4

    1922

  6. [14]

    Mmrotate: A rotated object detection benchmark using pytorch,

    Y . Zhou, X. Yang, G. Zhang et al. , “Mmrotate: A rotated object detection benchmark using pytorch,” in ACM MM , 2022, pp. 7331–7334. 4

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.