{"id":"d392b172-20dd-4bc3-8ab9-8a409382b2bd","arxiv_id":"2501.09720","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned Florence-2 detects rotated aerial objects in text form and matches conventional detectors under the paper's confidence-free mAPnc and mF1 metrics.","lead":"A team fine-tuned Florence-2, a general-purpose multimodal language model, to perform aerial object detection in remote sensing images, using text-formatted box coordinates and categories. It reports detection quality close to conventional detectors such as Rotated FCOS and RetinaNet, and proposes confidence-free metrics for fairer comparison.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparability claim rests on a nonstandard metric: mAPnc/mF1 are computed after choosing each conventional detector's confidence threshold on the evaluation set, and standard mAP is never reported. If standard mAP is the reference, parity is unestablished.","rationale":"The reader's weakest assumption correctly identifies the evaluation protocol as the load-bearing point: comparability to conventional detectors is asserted through mAPnc/mF1, which are nonstandard and depend on selecting the best confidence threshold on the evaluation set, while standard mAP is absent. My read confirms this concern and sharpens it: the confidence-free framing is internally inconsistent because the threshold used to compute mAPnc/mF1 is derived from the detectors' confidence scores, so the metric is not confidence-free, and the selection of the best threshold on test/validation splits gives conventional detectors an oracle operating point. This is a correctness risk in the argument, not merely a disagreement with consensus; the claim would need to be re-evaluated under standard mAP or with thresholds fixed on a held-out split. The paper's code, clear normalization scheme, and reproducible training setup are genuine strengths, and the conditional verdict remains appropriate: the central claim should not be accepted as stated until standard metrics and non-oracle threshold selection are reported.","tokens_in":6016,"tokens_out":5239,"duration_ms":60534,"concrete_test":"Recompute Table I with standard oriented mAP (IoU=0.5) using MMRotate's official evaluator for all models, with no confidence-threshold selection on the evaluation set; for MLM detectors, feed the raw extracted boxes. If the gap versus Rotated FCOS widens by more than a few mAP points on DOTA-v1.0 or DIOR-R, the claimed parity is an artifact of the proposed nonstandard, threshold-tuned metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim of 'performance comparable to conventional detector' is operationalized only by the author-defined metrics mAPnc and mF1 (Sec. II-C, III-B). These are not the metrics used by the field's benchmarks, and their computation contains an evaluation-set oracle: for every conventional detector, a confidence threshold is swept and the best mAPnc/mF1 is selected (Sec. III-B: 'calculate the two types of scores under a range of confidence thresholds and then select the best scores'), while MLM detections are assigned 100% confidence with no threshold selection. On DOTA-v1.0 and FAIR1M-v1.0, mF1 is computed on re-purposed validation splits because the official test servers only accept mAP. Standard mAP is never reported for any model. The issue is not merely a missing table: mAP is a ranking-based quantity, and after confidence is replaced by a constant or random value, the PR curve is not well-defined unless arbitrary tie ordering is imposed; the threshold filtering still relies on the very confidence scores the metric claims to make unnecessary. Thus the comparison conflates confidence-based filtering with confidence-free evaluation, and the headline parity is not established under the evaluation protocol the community actually uses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LMMRotate, a baseline that adapts open-source multimodal language models (MLMs) to multi-class oriented aerial detection by normalizing detection outputs into quantized textual responses and fine-tuning with standard next-token prediction. The authors introduce a confidence-free evaluation scheme (mAPnc and mF1) intended to make comparisons with conventional detectors fair, and they report results on DOTA-v1.0, DIOR-R, FAIR1M-v1.0, SRSDD, and RSAR using Florence-2-base and Florence-2-large. The central claim is that, after simple output normalization and fine-tuning, MLMs achieve detection performance comparable to conventional detectors, as summarized in the abstract and Table I.","tokens_in":6261,"tokens_out":3864,"duration_ms":39090,"significance":"If the comparability claim is upheld, the paper would be a useful practical baseline for applying MLMs to dense multi-class aerial detection, a task that existing RS MLMs have not directly addressed. The work provides an open-source implementation, explores joint training across multiple datasets, and demonstrates that the output-format gap between autoregressive text generation and detection can be bridged with a simple normalization. These are concrete contributions. However, the key evaluation claim is currently supported only by author-defined confidence-free metrics whose construction includes an evaluation-set oracle for conventional detectors, and the community-standard mAP is never reported. The significance of the result therefore hinges on the validity and acceptance of the proposed evaluation protocol, which the manuscript does not yet convincingly justify.","major_comments":[{"comment":"The central claim of 'performance comparable to conventional detector' rests entirely on the author-defined metrics mAPnc and mF1. For conventional detectors, the confidence threshold is selected by sweeping a range of thresholds on the evaluation set and picking the best score (Section III-B: 'calculate the two types of scores under a range of confidence thresholds and then select the best scores'). Standard mAP is never reported in Table I. This means the comparison is not made under the metric the community actually uses for these benchmarks. The authors should report standard mAP on the official test servers for DOTA-v1.0 and FAIR1M-v1.0 (which are available), or provide a rigorous argument explaining why mAPnc is the appropriate metric and why the oracle threshold selection does not invalidate the comparison.","section":"II-C and III-B, Table I"},{"comment":"The definition of mAPnc is incomplete and internally problematic. The metric is described as replacing confidence with constant or random scores, yet the threshold filtering used in Figure 4 still depends on the original confidence scores of the conventional detectors. When all confidence scores are replaced by a constant value, the precision-recall curve is not well-defined because the ranking of detections becomes arbitrary unless ties are broken by an unspecified rule. The text also claims that 'the consideration of confidence is not necessary' for comparing MLMs and conventional detectors, but this is contradicted by the metric's own threshold-selection step, which uses confidence implicitly. Please provide a formal definition of mAPnc, specify tie-breaking, and clarify how the threshold sweep is compatible with the claim that confidence is irrelevant.","section":"II-C, Figure 4"},{"comment":"For DOTA-v1.0 and FAIR1M-v1.0, mF1 is computed on re-purposed validation splits because the official test servers only accept mAP. Specifically, the authors retrain DOTA models on the training set and evaluate on the validation set, and for FAIR1M they evaluate on the FAIR1M-v2.0 validation set. These entries are therefore not directly comparable to the test-set mF1 values reported for the other datasets, nor to published results. The table should clearly mark these entries as validation-set numbers, and the authors should consider reporting the standard mAP from the official servers as the primary comparison.","section":"III-B"},{"comment":"The robustness argument for mAPnc is insufficient. The claim that standard deviations are 'generally lower than 0.5%' is based on replacing confidence with ten random values and one constant value, but this does not address the main concern: the best confidence threshold for each conventional detector is selected using the evaluation set. Since the MLM detections are all assigned 100% confidence and are not subject to threshold selection, the comparison protocol gives conventional detectors an oracle advantage. The error-band analysis in Figure 4 also appears to be performed on conventional detectors only; the authors should explain how this demonstrates robustness of the metric for the MLM comparison.","section":"II-C and Figure 4"}],"minor_comments":[{"comment":"There is a grammar error: 'a evaluation method' should be 'an evaluation method'.","section":"Abstract"},{"comment":"The word 'qualitize' should be 'quantize'.","section":"II-B"},{"comment":"In the sentence 'predicting tokens iteratively, .' there is a stray comma before the period; please remove it.","section":"II-A"},{"comment":"The notation is inconsistent: the text uses 'mF 1', 'mAP nc', and 'mAPnc' interchangeably. Please standardize the notation.","section":"II-C and Table I"},{"comment":"The definition of mAPnc is given only in the caption of Figure 4. A formal algorithm or equation should be included in Section II-C so the metric is unambiguous.","section":"II-C"},{"comment":"There are typos in the last paragraph: 'especally' should be 'especially' and 'benifits' should be 'benefits'.","section":"III-D"},{"comment":"The comparison includes only two conventional detectors, Rotated RetinaNet and Rotated FCOS. Adding a stronger contemporary oriented detector would strengthen the claim that performance is comparable to the current state of the art.","section":"III-D, Table I"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's contribution is potentially useful as a reproducible baseline, and I do not question the authors' intent. However, the evaluation protocol is the crux of the paper's central claim, and as written it does not establish the claimed comparability under the community-standard metric. I believe the paper can be revised to address this by reporting standard mAP and by tightening the discussion of the proposed metrics. I would like the revision to be reviewed for whether the new results actually support the abstract's wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is the first to fine-tune general-purpose MLMs for multi-class oriented aerial detection, and it ships code. The baseline itself looks real: quantizing coordinates, sorting outputs, and fine-tuning Florence-2 variants produces plausible detections on DOTA, DIOR-R, FAIR1M, and two SAR datasets. The joint-training experiments and the robustness check on mAPnc are nice touches. Credit where due: this is a straightforward, reproducible starting point for anyone who wants MLMs to do dense rotated detection.\n\nThe soft spot is exactly where the reader and the stress-test put it: the evaluation protocol. The central claim of parity with conventional detectors is supported only by mAPnc and mF1, metrics defined by the authors. For conventional detectors, the confidence threshold is swept on the evaluation set and the best score is selected—an oracle procedure. For MLMs, every prediction is assigned 100% confidence and no threshold is applied. Standard mAP, the metric DOTA and FAIR1M actually use, is never reported. That means the headline \"comparable to conventional detector\" is not established under the metric the field uses. Also, the threshold filtering for conventional detectors still relies on confidence scores, so mAPnc is not truly confidence-free. The mF1 results on DOTA and FAIR1M are computed on re-purposed validation splits, which further changes the comparison.\n\nThat said, the flaw is in the evaluation, not in the method. The paper is honest about the confidence problem and proposes a metric rather than hiding it. The code and the empirical baseline are valuable regardless. With a revised evaluation—reporting standard mAP on the official servers or at least on validation with standard tools, and avoiding oracle threshold selection—the parity claim could become defensible. As it stands, the quantitative comparison should be read as conditional on accepting mAPnc/mF1.\n\nWorth a serious referee. The work is a legitimate baseline contribution, the weakness is fixable, and the discussion of evaluation metrics for MLM detectors is timely. I'd bring it to a reading group as a case study in how metric choices carry the weight of a claim.","headline":"A useful, reproducible first baseline for turning general-purpose MLMs into oriented aerial detectors, but the headline parity claim rests on a nonstandard confidence-free metric and never reports standard mAP.","tokens_in":6783,"tokens_out":1921,"would_cite":true,"duration_ms":20433,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"After converting oriented boxes into text and fine-tuning, a multimodal language model performs aerial detection on par with conventional rotated detectors, measured without confidence scores.","keywords":["multimodal language model","aerial detection","remote sensing","oriented object detection","text normalization","confidence-free evaluation","fine-tuning","rotated bounding box"],"falsifier":"Re-running Table I with standard mAP (which ranks predictions by confidence) on the held-out splits would show whether the claimed parity holds when confidence is required; if the MLM detectors fall far behind both rotated detectors on standard mAP, the comparability claim would be refuted under the conventional metric.","tokens_in":5830,"feed_emoji":"🛰️","tokens_out":6598,"duration_ms":58134,"temperature":0.7,"pith_summary":"This paper claims that a generative multimodal language model can perform multi-class oriented aerial detection at a level comparable to conventional detectors, if the detection outputs are serialized as text and the comparison is made with confidence-free metrics. The authors fine-tune a general-purpose MLM (Florence-2) on five remote-sensing benchmarks by converting each oriented box into quantized integer coordinates and category names sorted alphabetically, then parsing the generated text with regex. They argue that confidence scores, which MLMs do not produce, are not essential for measuring detection quality, and they introduce mAPnc and mF1, which assign a constant confidence to every prediction and select the best threshold for conventional detectors on the evaluation set. Under these metrics the fine-tuned MLMs match or exceed rotated RetinaNet and rotated FCOS on most benchmarks. The paper is best read as a proof-of-concept baseline: MLMs can be instruction-tuned into exhaustive aerial detectors, not just grounders that localize a single mentioned object.","feed_headline":"Fine-tuned language model matches classic aerial detectors","feed_subtitle":"Serializing oriented boxes as text plus confidence-free metrics lets an MLM rival rotated RetinaNet and FCOS on five benchmarks.","key_machinery":"The central mechanism is a normalization-and-parsing pipeline that turns the detection task into a text-generation task. Each oriented box is represented as a category phrase plus eight vertex coordinates quantized to integers in [0,1000], with objects ordered alphabetically by category and then by the position of the starting vertex; the MLM is fine-tuned to produce this fixed template in response to a detection instruction, and the outputs are recovered by regex extraction and fuzzy category matching using Levenshtein distance. This machinery carries the argument because it makes the autoregressive next-token objective directly applicable to exhaustive detection, and the proposed confidence-free metrics (mAPnc and mF1) carry the evaluation, since they remove the reliance on confidence scores that MLMs do not output.","core_discovery":"The paper's central claim is that the autoregressive text-generation paradigm is not a barrier to dense multi-class detection. By normalizing each oriented object into the text tuple of a category name followed by eight integer vertex coordinates in the range 0 to 1000, sorting categories alphabetically and boxes by their starting vertex, and fine-tuning a pre-trained MLM with the standard next-token cross-entropy loss, the model learns to emit a complete, parsable list of all objects in an image. On DOTA-v1.0, DIOR-R, FAIR1M-v1.0, SRSDD and RSAR, the fine-tuned model achieves mAPnc and mF1 values comparable to rotated RetinaNet and rotated FCOS, and even surpasses them on several datasets under these metrics. The paper additionally claims that confidence scores are a byproduct of conventional detectors rather than an intrinsic part of the detection task, and therefore proposes confidence-free evaluation (mAPnc and mF1) as the fair basis for comparing MLM and conventional detectors.","pith_inferences":["A natural next test is to measure the same fine-tuned MLM under standard mAP with confidence produced by auxiliary means (e.g., token likelihood); if the gap to conventional detectors narrows, the confidence-free metrics may have overstated parity.","The threshold selection protocol for conventional detectors (best threshold on the evaluation set) is optimistic; using a validation split to pick thresholds would likely reduce the reported mAPnc and mF1 of the baselines and change the comparison.","The serialization idea could transfer to other structured prediction tasks such as instance segmentation or keypoint detection, where outputs are also sets of coordinates that can be flattened into text.","Scaling the vision encoder or input resolution, together with more joint training data, is a testable route to closing any residual gap with conventional detectors under confidence-free metrics."],"forward_implications":["A single fine-tuned MLM can act as both a visual grounder and an exhaustive multi-class detector, because both tasks share the same text interface.","The text-based normalization makes multi-dataset joint training straightforward: annotations from different datasets can be merged directly, and joint training improves performance on the smallest dataset (SRSDD).","Confidence-free metrics (mAPnc and mF1) provide a way to compare detection models that do not output confidence, and the authors recommend mAPnc for online benchmarks that currently require mAP.","MLM detectors need no non-maximum suppression, since the text generation naturally produces a set of objects without overlapping duplicates, simplifying the inference pipeline."],"supporting_citations":[{"why":"Supplies the pre-trained MLM architecture and weights (Florence-2) that are fine-tuned for detection and whose localization ability is inherited.","marker":"[5]"},{"why":"Provides the coordinate quantization scheme (normalize each axis to 0-1000 and round to integers) that the paper follows for turning boxes into text.","marker":"[8]"},{"why":"Primary optical benchmark and training source for oriented detection; its online test server motivates the recommendation to adopt mAPnc.","marker":"[6]"},{"why":"Fine-grained optical benchmark used for training and evaluation; evaluation is performed on the FAIR1M-v2.0 validation set.","marker":"[9]"},{"why":"Optical benchmark (DIOR-R) used for training and testing; one of the datasets where the MLM surpasses conventional detectors under confidence-free metrics.","marker":"[10]"},{"why":"Rotated SAR benchmark providing one of the two SAR evaluation datasets (RSAR) used in the comparison.","marker":"[7]"},{"why":"High-resolution SAR ship detection dataset (SRSDD) used as the small-data benchmark that gains the most from joint training.","marker":"[11]"},{"why":"Conventional rotated detector (Rotated RetinaNet) used as a baseline in Table I, with its mAPnc and mF1 computed at a selected confidence threshold.","marker":"[12]"},{"why":"Anchor-free conventional detector (Rotated FCOS) used as the strongest baseline; the MLM is compared against it on all five benchmarks.","marker":"[13]"}],"fun_headline_variants":["MLM detects aerial objects via text output","Aerial detection without confidence scores","Text-based detection rivals rotated RetinaNet","Simple baseline: MLM for aerial detection","Language model achieves detection parity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison holds only if detection quality is measured without confidence scores, where each conventional detector's best confidence threshold is chosen using the evaluation set itself.","fun_headline_variants_meta":{"raw":{"variants":["MLM detects aerial objects via text output","Aerial detection without confidence scores","Text-based detection rivals rotated RetinaNet","Simple baseline: MLM for aerial detection","Language model achieves detection parity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1374,"prompt_tokens":985,"completion_tokens":389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":328}},"tokens_in":601,"tokens_out":389,"duration_ms":5019,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:43:04.331563+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running Table I with standard mAP (which ranks predictions by confidence) on the held-out splits would show whether the claimed parity holds when confidence is required; if the MLM detectors fall far behind both rotated detectors on standard mAP, the comparability claim would be refuted under the conventional metric.","supporting_citations":[{"cited_title":"Florence-2: Advancing a unified representation for a variety of vision tasks,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained MLM architecture and weights (Florence-2) that are fine-tuned for detection and whose localization ability is inherited."},{"cited_title":"Dota: A large-scale dataset for object detection in aerial images,","cited_arxiv_id":null,"evidence_quote":"Primary optical benchmark and training source for oriented detection; its online test server motivates the recommendation to adopt mAPnc."},{"cited_title":"Fair1m: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery,","cited_arxiv_id":null,"evidence_quote":"Fine-grained optical benchmark used for training and evaluation; evaluation is performed on the FAIR1M-v2.0 validation set."},{"cited_title":"Object detection in optical remote sensing im- ages: A survey and a new benchmark,","cited_arxiv_id":null,"evidence_quote":"Optical benchmark (DIOR-R) used for training and testing; one of the datasets where the MLM surpasses conventional detectors under confidence-free metrics."},{"cited_title":"Srsdd-v1. 0: A high-resolution sar rotation ship detection dataset,","cited_arxiv_id":null,"evidence_quote":"High-resolution SAR ship detection dataset (SRSDD) used as the small-data benchmark that gains the most from joint training."},{"cited_title":"Focal loss for dense object detection,","cited_arxiv_id":null,"evidence_quote":"Conventional rotated detector (Rotated RetinaNet) used as a baseline in Table I, with its mAPnc and mF1 computed at a selected confidence threshold."},{"cited_title":"Fcos: A simple and strong anchor-free object detector,","cited_arxiv_id":null,"evidence_quote":"Anchor-free conventional detector (Rotated FCOS) used as the strongest baseline; the MLM is compared against it on all five benchmarks."}],"review_version":1}