REVIEW 3 major objections 6 minor 20 references
The paper claims that current vision-language models lack reliable mathematical reasoning on UAV imagery, showing that the best model, GPT-4o, scores only 34.6% on its new 3,773-question AVI-Math benchmark.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 18:12 UTC pith:T72SHENB
load-bearing objection A solid, useful benchmark for VLM math reasoning in UAV imagery; the flat-ground label uncertainty is real but doesn't sink the main result. the 3 major comments →
Multimodal Mathematical Reasoning Embedded in Aerial Vehicle Imagery: Benchmarking, Analysis, and Exploration
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central discovery is that multimodal mathematical reasoning in aerial vehicle imagery is an unsolved problem for current vision-language models: across 14 models, overall accuracy tops out at 34.6% (GPT-4o), and the best open-source model reaches only 27.9%. The authors trace the failures to a missing domain-specific knowledge (the largest share of GPT-4o's errors), limited perception of small objects in high-resolution UAV images, and visual encoders that cannot preserve detail when 4K inputs are downsampled. They also find that a correct answer is often accompanied by an incorrect rationale (29% of GPT-4o's correct answers), showing that accuracy scores overstate genuine reason
What carries the argument
The load-bearing component is the dataset's labeling pipeline, which converts pixel coordinates to 3D camera coordinates using the pinhole camera model. With a flat-ground assumption encoded in the plane equation −cosθ·Y_C − sinθ·Z_C + H = 0, each vehicle's pixel center is projected to a camera-coordinate point, so distances, areas, and flight times in geometry and algebra questions have computable ground truth. The evaluation also uses a two-stage generation-and-extraction protocol with format constraints per answer type to score free-form responses.
Load-bearing premise
The geometry and algebra ground-truth answers assume the ground is a perfect plane and that the drone's recorded altitude, pitch, and focal length are exactly correct, so a sloped or uneven surface—or any sensor inaccuracy—would systematically corrupt those labels.
What would settle it
Survey several AVI-Math capture sites with survey-grade GPS or a total station, re-compute the vehicle positions on the actual terrain, and check whether the distances and coordinates computed from Eq. (7) match; if systematic errors on the sloped sites change the ground-truth answer for more than a handful of geometry or algebra questions, the benchmark's conclusions on those subjects would not be reproducible.
If this is right
- AVI-Math provides a test bed where accuracy across reasoning steps (2 to 6) falls sharply with step count, so it can differentiate models by multi-step reasoning ability rather than by memorized Q&A.
- Models using CLIP-style encoders with short token sequences underperform on AVI-Math, pointing to visual token capacity as a concrete bottleneck for remote-sensing reasoning.
- LoRA fine-tuning on the 215k instruction set improves all three tested open models by at least 37%, with DeepSeek-VL rising 68%, showing that domain-specific instruction data is a viable route to better UAV reasoning.
- Chain-of-thought prompting helps InternVL2-40B and GPT-4o but hurts LLaVA-v1.6-34B, so the benefit of CoT depends on the model's baseline reasoning skill.
- The 45-degree pitch angle is hardest for every model, indicating that oblique views are underrepresented in VLM training and should be a focus for domain adaptation.
Where Pith is reading between the lines
- If the flat-ground labeling is accurate, the same pinhole projection could be used to generate synthetic 'non-flat' terrain labels from a digital elevation model, creating a diagnostic split to test whether models actually estimate metric geometry or merely pattern-match image features.
- The 0.85 Pearson correlation between response length and accuracy hints that a training reward that incentivizes longer, verifiable reasoning traces (as in reasoning-RL) might transfer better to this domain than standard SFT; this is a hypothesis the paper does not test.
- The benchmark's reliance on vehicle price and brand knowledge suggests a less obvious failure mode: cross-view knowledge transfer from natural images to aerial views is itself an ability worth benchmarking separately, and AVI-Math could be split into a 'perception-only' and a 'knowledge-plus-perception' set to quantify it.
- Because the benchmark includes true/false and multiple-choice questions, shortcut-avoidance matters; the paper argues free-form questions prevent shortcuts, so a future version could measure the 'shortcut gap' by comparing model performance on matched free-form vs multiple-choice versions of the same question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces AVI-Math, a benchmark of 3,773 UAV-based aerial-vehicle mathematical reasoning questions spanning six subjects (geometry, logic, statistics, arithmetic, counting, algebra) and 20 topics. Ground-truth coordinates for geometry and algebra questions are derived from manually annotated pixel coordinates using a pinhole camera model, with the ground plane assumed flat and sensor metadata (pitch angle, AGL, focal length) taken as exact. The authors benchmark 14 VLMs, reporting that even the best model (GPT-4o) achieves only about 33.5% average accuracy (Table 7), analyze error patterns, and explore LoRA fine-tuning on a 215k-sample instruction set plus several Chain-of-Thought prompting variants. The paper claims to be the first rigorous multimodal mathematical reasoning benchmark for UAV imagery.
Significance. If the benchmark's ground truth is trustworthy, AVI-Math fills a genuine gap: it moves beyond simple counting and perception questions in remote sensing VQA to multi-step mathematical reasoning in a realistic UAV setting. The paper has clear strengths: the coordinate transformation is given explicitly as standard pinhole optics; the evaluation covers 14 models with controlled splits by AGL and pitch angle; GPT-4o output analysis is a useful first step; and the code/data are promised for release. The benchmark labels are not circular in the sense of fitted model parameters, but their correctness depends on unquantified physical assumptions. Because the central claims about VLM limitations rest on the accuracy scores for GEO and ALG, the lack of quantitative validation of flat-ground and sensor-metadata assumptions is the key risk.
major comments (3)
- [§3.3, Eqs. (4)–(7); §3.4.1] The geometry and algebra ground truth is load-bearing for the benchmark's central claim, but the physical assumptions are not quantitatively validated. Eq. (4) assumes an exactly flat ground plane and exactly known pitch angle θ and AGL H; Eqs. (6)–(7) then convert pixel coordinates into 3D camera-coordinate positions. The only validation in §3.4.1 is a qualitative statement about placing normal vectors on poles. The paper provides no sensitivity analysis, no error bars, and no comparison against surveyed ground control points. A pitch error of 1° at H=40 m can shift the ground intersection by several meters, and an AGL error of 1 m propagates almost proportionally through every distance. Since GEO and ALG together are roughly 35% of the benchmark, systematic errors of this kind would make the reported subject-level accuracies uninterpretable and would confound the conclusion that VLMs l
- [§3.6 and §4.5.1] The fine-tuning exploration uses AVI-Math-215K, which is generated from the same image collection, same template pipeline, and same annotation conventions as the AVI-Math benchmark. The LoRA gains (37–68% relative improvement reported in Table 9) are therefore at least partly attributable to template and format overlap between the instruction set and the test set, rather than to improved domain knowledge. The paper itself acknowledges in §4.4 that current VLGFMs are 'fundamentally still engaged in data fitting,' yet §4.5.1 presents the gains as evidence that a large instruction set is effective. Without an out-of-distribution evaluation (e.g., other UAV/math benchmarks or different question templates), the fine-tuning contribution is overstated. This does not affect the primary benchmark results, but it is load-bearing for the paper's third contribution.
- [§4.1 and Table 7] The claim in §4.3 that 'open source VLMs still lag behind GPT-4o by at least 25%' is not supported by Table 7. The best open-source model, Qwen2.5-VL-7B, scores 27.9% versus GPT-4o's 33.5%, a gap of 5.6 percentage points (about 20% relative, and far less if computed on error rates). The statement appears to be a miscalculation or an undefined metric. Although this is not a fatal flaw, it is a concrete inconsistency in a headline analysis that should be corrected.
minor comments (6)
- [Abstract / Table 7 / Table 9] The abstract says the highest overall accuracy is 34.6%, but Table 7 lists the best average as 33.5% (GPT-4o). The 34.6% figure appears only in Table 9 for GPT-4o with Plan-and-Solve. Please harmonize the numbers and clarify which configuration is being cited.
- [§3.3, Eq. (4)] The pitch angle θ is not formally defined, and the sign convention in the ground-plane equation is not explained. Readers cannot check Eq. (4) without inferring the axis orientation. Please add an explicit definition of θ and the camera coordinate frame.
- [§3.2] The inter-annotator consistency score of 0.9768 is reported as a single Cohen's kappa on 10% of vehicle samples. Please report per-attribute kappa values (brand, model, type, color, bounding box) and state whether the re-annotation was done by the same annotators or independent ones.
- [§4.2] The evaluation setup does not report decoding parameters (temperature, top-p, max tokens) for the 14 models. Since free-form answer extraction is sensitive to generation length and sampling, these details are needed for reproducibility.
- [Fig. 5, Fig. 6, §4.4] The manual analysis of GPT-4o outputs reports percentages such as 54.3% and 21.9%, but no sample size or inter-rater reliability is given. Please state how many outputs were manually judged and by how many annotators.
- [§2, §3.6] The nomenclature is inconsistent: 'UAV-MATH-215K' appears in §2 and Table 1, while 'AVI-MATH-215K' is used elsewhere. Please unify to the AVI-Math naming convention.
Circularity Check
No significant circularity: benchmark labels are computed from measured sensor metadata and manually annotated pixel locations; model performance is evaluated externally.
full rationale
AVI-Math's ground-truth labels for geometry and algebra questions are produced by a deterministic camera-model pipeline. Equation (4) is the standard flat-ground plane equation using measured AGL H and pitch angle θ, and Equation (7) is the corresponding pinhole projection inverse mapping from pixel coordinates to camera coordinates. The inputs are sensor metadata and manually annotated pixel/box locations; no model output, fitted parameter, or benchmark-derived quantity enters the label construction. The benchmark's central empirical claim—that 14 VLMs score poorly, with GPT-4o at 34.6%—is an external evaluation against this fixed label set, not a reduction to the paper's own assumptions. The flat-ground and exact-metadata assumptions are validity risks (a correctness concern), not circularity, because they do not make the evaluation tautological. The fine-tuning experiments use an instruction set generated with the same template pipeline as the benchmark, and this overlap is explicitly disclosed in Section 3.6; it is a distributional similarity that weakens the generality of the fine-tuning improvement, but it does not feed back into the benchmark labels or into the measured zero-shot scores. No load-bearing self-citation was found: the only author-overlapping reference (Yang et al., 2022) is used for rotated bounding-box annotation, a standard tool that does not justify the benchmark's mathematical-reasoning claim. Therefore no circular step is present.
Axiom & Free-Parameter Ledger
axioms (5)
- domain assumption Flat ground plane assumption for UAV scenes (Eq. 4)
- standard math Pinhole camera model with negligible lens distortion (Eqs. 1-3, 5-7)
- domain assumption Sensor metadata (AGL, pitch angle, focal length, pixel size) are accurate
- domain assumption Vehicle attribute database and manual annotation are complete and correct
- domain assumption Template-generated questions and rationales are correct after manual review
read the original abstract
Mathematical reasoning is critical for tasks such as precise distance and area computations, trajectory estimations, and spatial analysis in unmanned aerial vehicle (UAV) based remote sensing, yet current vision-language models (VLMs) have not been adequately tested in this domain. To address this gap, we introduce AVI-Math, the first benchmark to rigorously evaluate multimodal mathematical reasoning in aerial vehicle imagery, moving beyond simple counting tasks to include domain-specific knowledge in areas such as geometry, logic, and algebra. The dataset comprises 3,773 high-quality vehicle-related questions captured from UAV views, covering 6 mathematical subjects and 20 topics. The data, collected at varying altitudes and from multiple UAV angles, reflects real-world UAV scenarios, ensuring the diversity and complexity of the constructed mathematical problems. In this paper, we benchmark 14 prominent VLMs through a comprehensive evaluation and demonstrate that, despite their success on previous multimodal benchmarks, these models struggle with the reasoning tasks in AVI-Math. Our detailed analysis highlights significant limitations in the mathematical reasoning capabilities of current VLMs and suggests avenues for future research. Furthermore, we explore the use of Chain-of-Thought prompting and fine-tuning techniques, which show promise in addressing the reasoning challenges in AVI-Math. Our findings not only expose the limitations of VLMs in mathematical reasoning but also offer valuable insights for advancing UAV-based trustworthy VLMs in real-world applications. The code, and datasets will be released at https://github.com/VisionXLab/avi-math
Figures
Reference graph
Works this paper leans on
-
[5]
arXiv preprint arXiv:2404.06395
Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395 . Hu, W., Tong, M.,
-
[6]
arXiv preprint arXiv:2303.01894
Trr360d: A dataset for 360 degree rotated rectangular box table detection. arXiv preprint arXiv:2303.01894 . Hu,Y.,Yuan,J.,Wen,C.,Lu,X.,Li,X.,2023. Rsgpt:Aremotesensingvi- sion language model and benchmark. arXiv preprint arXiv:2307.15266 . Kuckreja,K.,Danish,M.S.,Naseer,M.,Das,A.,Khan,S.,Khan,F.S.,2024. Geochat: Grounded large vision-language model for r...
Pith/arXiv arXiv 2023
-
[7]
ISPRS journal of photogrammetry and remote sensing 159, 296–307
Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS journal of photogrammetry and remote sensing 159, 296–307. Li,X.,Ding,J.,Elhoseiny,M.,2024. Vrsbench:Aversatilevision-language benchmark dataset for remote sensing image understanding. arXiv preprint arXiv:2406.12384 . Liu, H., Li, C., Li, Y., Lee, Y.J., 2024a. Improved...
Pith/arXiv arXiv 2024
-
[8]
arXiv preprint arXiv:2304.08485
Visual instruction tuning. arXiv preprint arXiv:2304.08485 . Yue Zhou: Preprint submitted to Elsevier Page 16 of 17 UAV-Math Liu, W., Pan, Q., Zhang, Y., Liu, Z., Wu, J., Zhou, J., Zhou, A., Chen, Q., Jiang,B.,He,L.,2024b.Cmm-math:Achinesemultimodalmathdataset to evaluate and enhance the mathematics reasoning of large multimodal models. arXiv preprint arX...
-
[10]
IEEE Transactions on Geoscience and Remote Sensing 58, 8555–8566
Rsvqa: Visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing 58, 8555–8566. Loshchilov,I.,2017.Decoupledweightdecayregularization.arXivpreprint arXiv:1711.05101 . Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Sun, Y., et al., 2024a. Deepseek-vl: towards real-world vision-...
Pith/arXiv arXiv 2017
-
[12]
Learning transferable visual models from natural language supervision, in: Inter- national conference on machine learning, PmLR. pp. 8748–8763. Rahnemoonfar, M., Chowdhury, T., Sarkar, A., Varshney, D., Yari, M., Murphy,R.R.,2021. Floodnet:Ahighresolutionaerialimagerydataset for post flood scene understanding. IEEE Access 9, 89644–89654. Sandmann, S., Heg...
2021
-
[14]
arXiv preprint arXiv:2504.07491
Kimi-vl technical report. arXiv preprint arXiv:2504.07491 . Team,Q.,2025. Qwen2.5-vl. URL: https://qwenlm.github.io/blog/qwen2. 5-vl/. Wang, C., Ning, R., Pan, B., Wu, T., Guo, Q., Deng, C., Bao, G., Hu, X., Zhang, Z., Wang, Q., et al., 2024a. Novelqa: Benchmarking ques- tion answering on documents exceeding 200k tokens. arXiv preprint arXiv:2403.12766 . ...
Pith/arXiv arXiv 2025
-
[15]
Plan-and-solveprompting:Improvingzero-shotchain-of-thought reasoningbylargelanguagemodels. arXivpreprintarXiv:2305.04091. Wang, S., Han, W., Huang, X., Zhang, X., Wang, L., Li, J., 2024d. Trustworthy remote sensing interpretation: Concepts, technologies, and applications. ISPRS Journal of Photogrammetry and Remote Sensing 209, 150–172. Wang, S., Yu, L., G...
-
[16]
arXiv preprint arXiv:2506.01939
Beyond the 80/20 rule: High-entropy minoritytokensdriveeffectivereinforcementlearningforllmreasoning. arXiv preprint arXiv:2506.01939 . Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.,
-
[18]
A survey onmultimodallargelanguagemodels. arXivpreprintarXiv:2306.13549 . Yuan, Z., Mou, L., Xiong, Z., Zhu, X.X.,
-
[19]
Long-clip: Unlocking the long-text capability of clip, in: European conference on computer vision, Springer. pp. 310–325. Zhang, M., Chen, F., Li, B., 2023a. Multistep question-driven visual ques- tion answering for remote sensing. IEEE Transactions on Geoscience and Remote Sensing 61, 1–12. Zhang, P., Wang, X.D.B., Cao, Y., Xu, C., Ouyang, L., Zhao, Z., ...
Pith/arXiv arXiv 2021
-
[20]
URL: https://arxiv.org/abs/2504.10479, arXiv:2504.10479
Internvl3: Exploring advanced trainingandtest-timerecipesforopen-sourcemultimodalmodels. URL: https://arxiv.org/abs/2504.10479, arXiv:2504.10479. Zhu,P.,Wen,L.,Du,D.,Bian,X.,Fan,H.,Hu,Q.,Ling,H.,2021.Detection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 7380–7399. Yue Zhou: Preprint submitted to E...
Pith/arXiv arXiv 2021
-
[2016]
A large contextual dataset for classification, detection and counting of cars with deep learning, in: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part III 14, Springer. pp. 785–800. Novak, L.,
2016
-
[2018]
Dota: A large-scale dataset for object detection in aerial images, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3974–3983. Xu,L.,Zhao,L.,Guo,W.,Li,Q.,Long,K.,Zou,K.,Wang,Y.,Li,H.,2024. Rs-gpt4v:Aunifiedmultimodalinstruction-followingdatasetforremote sensing image understanding. arXiv preprint arXiv:2406.12479 . Ya...
Pith/arXiv arXiv 2024
-
[2020]
In-time uav flight-trajectory estimation andtrackingusingbayesianfilters,in:2020IEEEaerospaceconference, IEEE. pp. 1–9. Chen, Q., Qin, L., Zhang, J., Chen, Z., Xu, X., Che, W., 2024a. M3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of- thought. arXiv preprint arXiv:2405.16473 . Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S.,...
Pith/arXiv arXiv 2023
-
[2021]
Rsvqa meets bigearthnet: A new, large-scale, visual question answering dataset for remote sensing, in: 2021 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), IEEE. pp. 1218–1221. Lobry, S., Marcos, D., Murray, J., Tuia, D.,
2021
-
[2022]
International Journal of Remote Sensing 43, 6809–6823
Open-ended remote sensing visual question answering with transformers. International Journal of Remote Sensing 43, 6809–6823. Alvarez-Vanhard,E.,Corpetti,T.,Houet,T.,2021.Uav&satellitesynergies for optical remote sensing applications: A literature review. Science of remote sensing 3, 100019. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J....
2021
-
[2023]
arXiv preprint arXiv:2308.12966
Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 . Banerjee, P., Corbetta, M.,
-
[2024]
arXiv preprint arXiv:2402.13791
Opening the black-box: A sys- tematic review on explainable ai in remote sensing. arXiv preprint arXiv:2402.13791 . Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.,
-
[2025]
arXiv preprint arXiv:2504.07615
Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615 . Team,K.,Du,A.,Yin,B.,Xing,B.,Qu,B.,Wang,B.,Chen,C.,Zhang,C., Du, C., Wei, C., et al.,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.