{"id":"0f581d63-a73e-4247-9ed1-fd529d0dc43e","arxiv_id":"2506.04998","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RAG improved one LLM's exact-match accuracy on a small UAV math benchmark, but most reported gains compare different models rather than the same model with and without retrieval.","lead":"This paper combines retrieval-augmented generation with UAV engineering documents to help large language models solve arithmetic problems about drones. On a 20-question benchmark the best augmented model answered 75 percent exactly, but the study lacks same-model no-retrieval comparisons for nearly all models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline RAG gains compare o1-with-RAG to ChatGPT-without-RAG; only a weak within-model baseline exists, so the central causal claim is not yet controlled.","rationale":"The reader's weakest assumption correctly identifies the central confound: the headline improvement compares different models, so model capability and RAG are entangled. I partially agree because the manuscript does contain one within-model pair—ChatGPT/GPT-4 Turbo without RAG versus Turbo with RAG—which the reader's statement overlooks; this pair shows a large accuracy gain (30% to 60%) and a large miscalculation drop, providing some genuine support for a RAG effect. However, that paired result is not the comparison highlighted in the abstract, and no significance test is reported for a 20-question benchmark. The o1-specific claims remain unsupported because o1 was never run without RAG. The numerical inconsistency in the o1 row (Miscalc 15% vs. the 20% implied by Correct Form and MisC) further weakens confidence in the reported metrics. The limitation section acknowledges the small benchmark but does not address the missing paired baselines or the benchmark's overlap with the retrieval corpus. Because the proposed method is plausible and a properly paired evaluation could confirm the direction of the effect, the existing CONDITIONAL verdict is appropriate; I do not move it.","tokens_in":11060,"tokens_out":8310,"duration_ms":89701,"concrete_test":"Run every model in Table 2 on all 20 benchmark questions under two matched conditions: (a) the original question with no retrieved context, and (b) the same question with the paper's RAG prompt and top-k retrieved chunks, fixing sampling parameters and using identical evaluation scripts. For each model, compute paired exact-match accuracy, formulation-selection error, and MSE, then apply McNemar's exact test or a permutation test on the per-question correctness vectors. The headline claim is settled only if o1-without-RAG is substantially below o1-with-RAG (e.g., an accuracy gap of at least two questions with p < 0.05) and the same direction holds for at least two other models. Also report the retrieval hit rate per question and recheck whether the o1 miscalculation metric is 15% or 20%.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing problem is that the headline causal claim is built on confounded comparisons. Table 2 shows ChatGPT without RAG (30% correctness, 75% correct formulation) versus o1 with RAG (75%, 95%), and the abstract converts exactly this pair into '25% without RAG to 5% with RAG' and into the accuracy and MSE gains. Because o1 is a substantially more capable model, model identity and the RAG treatment are fully entangled in the main result. The paper does contain one within-model pair—ChatGPT/GPT-4 Turbo without RAG (30% accuracy, 60% MisC) versus Turbo with RAG (60%, 26.67% MisC)—which is genuine supporting evidence, but it is not the comparison advertised, and with only 20 questions the paired difference needs a significance test. No other model, including o1, was run without RAG. This confound directly undermines the abstract's 'substantially increases exact answer accuracy' and 'reduces numerical errors by orders of magnitude' claims. A secondary issue is that the benchmark is drawn from the same five papers used to build the retrieval corpus, so retrieval hit rates should be reported; as it stands, the paper demonstrates formula lookup in a closed corpus, not yet a robust engineering-reasoning gain. The o1 row of Table 2 also has an internal inconsistency: Correct Form = 95% and MisC = 21.05% imply an overall miscalculation rate of about 20%, not the reported 15%.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAG-UAV, a retrieval-augmented generation framework that supplies retrieved passages from five UAV-related research papers to several LLMs (GPT o1, o1-preview, GPT-4 Turbo, Llama-3.2/3.3, Mistral, DeepSeek R1) to improve their performance on UAV-specific mathematical word problems. The authors introduce UAV-Math-Bench, a 20-question benchmark spanning four difficulty levels, and evaluate models on exact-answer accuracy, formulation-selection error, and MSE. The central claim is that RAG substantially improves exact accuracy, reduces incorrect formulation selection (from 25% without RAG to 5% with RAG), and lowers MSE by orders of magnitude, based chiefly on comparing non-RAG ChatGPT with RAG-augmented models. The paper also includes a within-model comparison of ChatGPT/GPT-4 Turbo with and without RAG, attention-pattern visualizations for DeepSeek R1, and a public code/data repository.","tokens_in":11304,"tokens_out":4070,"duration_ms":43124,"significance":"If the central claim were substantiated, the paper would provide a useful pilot demonstration that RAG can make LLMs more reliable for engineering calculations in a specialized domain, with a public benchmark and clearly described prompting pipeline. The release of the benchmark, the detailed prompt template, the attempt to use Mathpix to preserve equations, and the explicit discussion of limitations are all strengths. However, the headline result currently rests on a confounded comparison, and the benchmark's tight coupling to the retrieval corpus makes the evaluation closer to closed-book formula lookup than to open-ended engineering reasoning. The within-model Turbo comparison is genuine supporting evidence, but it is not the comparison advertised. With additional controlled experiments and statistical grounding, the contribution could be a solid pilot study; as it stands, the evidence is not yet sufficient for the strength of the claims made in the abstract and conclusion.","major_comments":[{"comment":"The statistical basis is thin for the strength of the claims. With only 20 questions (5 per difficulty level), each 20% cell in Table 3 corresponds to a single question, so level-by-level comparisons such as 'o1 achieves 80% accuracy on Expert' are not statistically meaningful. Similarly, the 'orders of magnitude' MSE claim needs the actual per-model MSE values and a paired, per-question comparison; reporting log10(MSE) for one model (214.74) without the other models' values does not establish the claimed reductions, especially given the confounded design.","section":"§4.2, Table 3 and Figure 7"}],"minor_comments":[{"comment":"There is a typo: 'ol performed the best' should be 'o1 performed the best'.","section":"§4.2, first paragraph"},{"comment":"The sentence 'may produce outputs that violate physical constraints or Our work aligns with this goal...' is an incomplete sentence and should be restructured.","section":"§2.2, final sentence of second paragraph"},{"comment":"The text contains the orphan fragment 'where [6]' immediately after 'Chroma vector database.' It should be removed or completed.","section":"§3.2"},{"comment":"The notation is inconsistent: the table uses 'Chat' and 'Turbo' while the text uses 'ChatGPT Turbo' and 'ChatGPT-4-Turbo'. Clarify that Chat = ChatGPT Turbo and Turbo = GPT-4 Turbo, and that the within-model comparison uses the same model with and without RAG.","section":"§4.2, model naming"},{"comment":"The citation [12] is a RAG survey and does not support the attention-pattern analysis; please cite a primary source on attention mechanisms.","section":"§4.4"},{"comment":"Figure 7 is referenced in §4.2 but appears after Figures 5 and 6; ensure the figure ordering and referencing are consistent.","section":"Figure 7"},{"comment":"Define 'Miscalc.' precisely: does it count miscalculations among all attempts or only among attempts with correct formulation? The table has both 'Miscalc.' and 'MisC†' with a footnote, but the text should define both in the metrics subsection.","section":"§4.1, metrics definitions"},{"comment":"The prompt template contains a formatting artifact: 'F ormulations' should be 'Formulations'.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper gives us a small, reusable UAV-math benchmark and a clean RAG pipeline, but the headline claim that RAG cuts formulation errors from 25% to 5% and MSE by orders of magnitude is built on a confounded comparison. Only ChatGPT was run without RAG; every other model, including the best-performing o1, only appears with RAG. So the abstract's numbers compare o1-with-RAG to ChatGPT-without-RAG, entangling model capability with the treatment.\n\nWhat's genuinely new: UAV-Math-Bench, 20 categorized problems across four difficulty levels, with a public GitHub repo and human-verified answers. That's a real, if modest, contribution for a domain that lacks standardized evaluation. The RAG pipeline itself is standard vector retrieval, but the paper is open about its components and prompts, so the recipe is reproducible.\n\nWhere it's soft: first, the missing paired baselines. The one within-model comparison that exists—Turbo without RAG (30% correct, 60% miscalc among correct forms) vs Turbo with RAG (60% correct, 26.67%)—supports the direction, but on 20 questions that difference needs a significance test. Second, the questions appear to be constructed from the same five papers that form the retrieval corpus, so the task is partly formula lookup in a closed corpus; reporting retrieval hit rates would help. Third, the paper's limitation section notes the small benchmark but doesn't acknowledge the confound. Finally, the o1 row of Table 2 has an internal inconsistency: 95% correct formulation and 21.05% miscalc among correct forms should give an overall miscalculation rate of 20%, not the reported 15%.\n\nOn balance, the paper is a solid pilot, honestly presented in most respects, but the central empirical claim goes beyond what the data can support. A serious referee would ask for paired baselines across all models and a larger benchmark. I'd send it to review—the benchmark and the recipe deserve to be on record, and the revision path is clear.\n\nRegards.","headline":"Useful small UAV math benchmark and a plausible RAG recipe, but the headline RAG gains come from comparing different models, not a controlled ablation.","tokens_in":11881,"tokens_out":2310,"would_cite":false,"duration_ms":26145,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval-augmented generation lifts exact-answer accuracy on UAV arithmetic to 75 percent.","keywords":["retrieval-augmented generation","large language models","mathematical reasoning","UAV-Math-Bench","arithmetic reasoning","formulation selection","mean squared error","engineering analysis"],"falsifier":"Run each evaluated model on UAV-Math-Bench both with and without retrieval; if a no-retrieval version already reaches roughly the same exact-answer accuracy, the reported gain is not caused by retrieval. A second check: keep the same prompt template but remove the retrieved documents, so correct formulas would indicate the prompt style, not the retrieval, is doing the work.","tokens_in":10819,"feed_emoji":"🚁","tokens_out":7258,"duration_ms":80143,"temperature":0.7,"pith_summary":"This paper argues that retrieval-augmented generation (RAG) can make general-purpose large language models reliably solve UAV-specific arithmetic problems by feeding them the right equations from domain literature at query time. To test this, the authors build RAG-UAV, a pipeline that embeds five UAV modeling papers into a vector store, retrieves relevant passages for each question, and injects them into the prompt. They introduce UAV-Math-Bench, 20 problems in four difficulty tiers, and report that RAG raises exact-answer accuracy to 75 percent for the best model, cuts wrong-formula selections from 25 percent to 5 percent, and lowers mean squared error by orders of magnitude. If the results hold, engineers could use RAG-augmented LLMs as calculation assistants for trajectory planning and power management without hardcoding every equation.","feed_headline":"Retrieval boosts drone-math answers to 75 percent","feed_subtitle":"Giving LLMs the right UAV equations cuts wrong-formula picks from 25% to 5% and shrinks numerical error.","key_machinery":"The load-bearing mechanism is a retrieval-augmented prompt construction loop. A user query is embedded and matched against 162 page-level LaTeX chunks taken from five UAV modeling papers; the top retrieved segments are placed into a prompt that tells the model to use only those documents for formulation selection while allowing it to do its own arithmetic. The prompt template itself is the central object: it isolates the effect of retrieval on formula choice from the model's internal calculation engine, which is exactly what lets the authors claim the improvement comes from grounding rather than from better computation.","core_discovery":"The paper's central discovery is that retrieval, rather than larger models alone, is the lever that fixes the two failure modes that stop LLMs from doing UAV math: choosing the wrong physical formula and making numerical slips inside the right formula. On a 20-question benchmark, the retrieval-augmented o1 model answers 75 percent exactly and selects the correct formulation 95 percent of the time, while the non-augmented ChatGPT baseline gets 30 percent exact and 75 percent correct formulations. The paper attributes the gain to grounding: when the retrieved chunk supplies the blade-power or path-loss equation, the model stops declaring inputs insufficient and instead follows the formula through multi-step arithmetic, reducing miscalculation among correctly formulated answers and cutting MSE dramatically.","pith_inferences":["A natural next step is to run every evaluated model both with and without retrieval on the same benchmark, which would make the retrieval effect directly measurable across model families.","Because the corpus is only five papers, the framework's ceiling depends on coverage; scaling the knowledge base to standards and experimental datasets is the direct route to broader engineering use.","The difficulty tiers suggest an immediate stress test: add problems with more nested equations and variables, since the reported gains are strongest on exactly those multi-step cases."],"forward_implications":["RAG-augmented LLMs could serve as calculation co-pilots for UAV engineering tasks such as trajectory planning, power budgeting, and link-budget analysis, catching wrong formulas before they reach the autopilot.","The same retrieval-plus-prompt recipe should transfer to other engineering domains whose math is formula-driven, since the benchmark's difficulty tiers show gains concentrate on multi-step nested problems.","Because the framework keeps arithmetic inside the model, its accuracy is bounded by the model's numerical precision; pairing retrieval with an external calculator would directly address the residual miscalculation rate.","The 20-question UAV-Math-Bench gives the community a small, public target for measuring whether future models or retrieval strategies improve UAV math before deployment."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation recipe the framework adapts, grounding generation in retrieved documents.","marker":"[1]"},{"why":"Provides the embedding model used to index the corpus for OpenAI-based models.","marker":"[3]"},{"why":"Provides the embedding model used to index the corpus for open-source models.","marker":"[5]"},{"why":"One of the five corpus papers contributing UAV-specific formulas to the vector knowledge base.","marker":"[6]"},{"why":"One of the five corpus papers whose mathematical content is embedded and retrieved at query time.","marker":"[8]"},{"why":"One of the five corpus papers supplying formulations for the benchmark problems.","marker":"[9]"},{"why":"One of the five corpus papers supplying formulations used in the benchmark.","marker":"[10]"},{"why":"One of the five corpus papers whose formulas are turned into LaTeX chunks for retrieval.","marker":"[11]"},{"why":"Describes the open-weight reasoning model whose output and attention patterns are analyzed in the evaluation.","marker":"[13]"}],"fun_headline_variants":["RAG gives drones' LLMs a math boost to 75%","Retrieval lifts UAV math accuracy to 75% exact","Grounding LLMs in equations cuts UAV math errors","For drones, RAG beats bigger brains on math","Retrieval fixes LLM formula picks, slashes errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the observed gains in accuracy and formulation selection come from retrieval itself, rather than from the fact that the retrieval-augmented models are larger or stronger than the single no-retrieval baseline.","fun_headline_variants_meta":{"raw":{"variants":["RAG gives drones' LLMs a math boost to 75%","Retrieval lifts UAV math accuracy to 75% exact","Grounding LLMs in equations cuts UAV math errors","For drones, RAG beats bigger brains on math","Retrieval fixes LLM formula picks, slashes errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3043,"prompt_tokens":952,"completion_tokens":2091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":2017}},"tokens_in":568,"tokens_out":2091,"duration_ms":17242,"temperature":1.0,"reasoning_tokens":2017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:28:35.421137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each evaluated model on UAV-Math-Bench both with and without retrieval; if a no-retrieval version already reaches roughly the same exact-answer accuracy, the reported gain is not caused by retrieval. A second check: keep the same prompt template but remove the retrieved documents, so correct formulas would indicate the prompt style, not the retrieval, is doing the work.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the retrieval-augmented generation recipe the framework adapts, grounding generation in retrieved documents."},{"cited_title":"text-embedding-ada-002: AI Model for Text Embeddings","cited_arxiv_id":null,"evidence_quote":"Provides the embedding model used to index the corpus for OpenAI-based models."},{"cited_title":"all-mpnet-base-v2: Pretrained Sentence Transformer Model","cited_arxiv_id":null,"evidence_quote":"Provides the embedding model used to index the corpus for open-source models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the five corpus papers contributing UAV-specific formulas to the vector knowledge base."},{"cited_title":"Mollel, Oluwakayode Onireti, Metin Ozturk, Iftikhar Ahmad, Syed Muhammad Asad, Yusuf Sambo, Ahmed Zoha, Sajjad Hussain, Muhammad Ali Imran","cited_arxiv_id":null,"evidence_quote":"One of the five corpus papers whose mathematical content is embedded and retrieved at query time."},{"cited_title":"Backhaul-aware drone base station placement and resource management for FSO-based drone-assisted mobile networks","cited_arxiv_id":null,"evidence_quote":"One of the five corpus papers supplying formulations for the benchmark problems."},{"cited_title":"Energy-efficiency computation offloading strategy in UA V aided V2X network with integrated sensing and communication","cited_arxiv_id":null,"evidence_quote":"One of the five corpus papers supplying formulations used in the benchmark."},{"cited_title":"Energy efficient deployment of aerial base stations for mobile users in multi-hop UA V networks.Ad Hoc Networks, 157:103463, 2024","cited_arxiv_id":null,"evidence_quote":"One of the five corpus papers whose formulas are turned into LaTeX chunks for retrieval."}],"review_version":1}