Pith. sign in

REVIEW 5 major objections 6 minor 23 references

Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Graph RAG answers airport flight queries more accurately than SQL RAG or traditional RAG, and structured-query RAG methods hallucinate less.

desk verdict Practical airport RAG comparison with plausible qualitative takeaways, but the headline accuracy numbers are internally inconsistent and rest on an undisclosed evaluation split. read the letter →

arxiv 2505.13006 v1 pith:Y35JHETZ submitted 2025-05-19 cs.CL

Yuyang Li , Philip J.M. Kerbusch , Raimon H.R. Pruim , Tobias Käfer This is my paper · ORCID
classification cs.CL
keywords Retrieval-AugmentedGenerationText-to-SQLKnowledgeGraphConversationalAIAirportoperationsHallucinationreductionFlightinformationsystemsQuestionclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Airports are high-risk, jargon-heavy environments where a conversational assistant for ground staff must answer from flight data without inventing facts. This paper builds three such assistants — traditional retrieval-augmented generation (RAG), text-to-SQL RAG, and knowledge-graph RAG — and evaluates them on flight-information questions. It claims that Graph RAG reaches 91.49% accuracy on straightforward queries and 68.75% on reasoning queries, outperforming SQL RAG (80.85% and 6.25%) and traditional RAG (84.84% and 9.38%). The central argument is that translating a question into a structured database query, instead of feeding the model a slab of retrieved text, sharply reduces hallucinations. If correct, the finding gives airport operators a concrete reason to prefer graph- or SQL-backed assistants over plain document retrieval.

What carries the argument

The mechanism that carries the result is structured-query conversion with schema-aware prompting. For SQL RAG, the Code Representation Prompt embeds full CREATE TABLE statements with column types and keys, so the LLM generates SQL and the database returns only the exact rows requested. For Graph RAG, the same idea works over a property graph: the LLM is given the graph schema (extracted with the APOC `apoc.meta.schema()` call) and writes Cypher queries that traverse relationships — connecting flight, same ramp, gate — rather than scanning a text index. This is what lets a question like "What is the expected on-ramp time of DL0123's connecting flight?" be answered by following a link in the graph. The comparison of exact-match against execution-match scores shows the trade-off: SQL queries are syntactically rigid (high EM, moderate EX), while Cypher queries are syntactically flexible but executionally accurate (low EM, high EX).

What would settle it

Re-run the evaluation on a held-out question set that removes every example used in the prompts, score the answers by two independent annotators who do not know which RAG method produced them, and compare Graph RAG's execution-match accuracy to SQL RAG's on the straightforward set. If Graph RAG no longer exceeds SQL RAG (91.49% vs 80.85%), the headline ordering was an artifact of prompt overlap or scoring bias; if the gap persists, the central claim survives.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a ranking of RAG architectures for a specific high-stakes domain. Using a flight-information table from an airport operation and a manually built question set, the authors report that Graph RAG, implemented as an LLM writing Cypher queries against a Neo4j knowledge graph, achieves an execution match of 91.49% on straightforward questions and 68.75% on a 30-question reasoning set. SQL RAG, where the LLM writes SQL using a detailed CREATE TABLE schema prompt, scores 80.85% execution match on straightforward questions but only 6.25% on reasoning questions. Traditional RAG with BM25 + GPT-4 reaches 84.84% on straightforward questions and 9.38% on reasoning, and it is the pipeline in which the authors observed the model producing destinations not present in the dataset. The paper interprets the low exact-match but high execution-match scores of Graph RAG as evidence that graph query languages let the model express the same correct answer in different syntactic forms, and it recommends SQL RAG and Graph RAG over traditional RAG for airport environments because structured queries feed the LLM only verified rows.

Load-bearing premise

The reported accuracy ranking collapses if the few-shot example questions embedded in the prompts (47 for SQL RAG, 60 for question classification, 20 for hallucination reduction) overlap with the questions used for evaluation, or if the manual True/False scoring was inconsistent, because the paper does not describe this split.

Editorial extensions

If this is right

  • A graph-backed conversational assistant could be deployed at airport operations desks to answer standard flight queries at roughly 91% execution-match accuracy, reducing the need for staff to write database queries.
  • For airports that already have relational databases, SQL RAG offers a safer alternative to document-style RAG on straightforward questions, but its 6.25% accuracy on reasoning questions means it should not handle multi-step time or connection queries alone.
  • The hallucination argument implies that any high-safety deployment of an LLM assistant should prefer generation over database query results over generation over retrieved passages, whenever the data is tabular or relational.
  • The reasoning-query results suggest that representing flight relationships as graph edges, rather than as columns in a table, is what enables multi-hop temporal questions such as connecting-flight and same-ramp queries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper tests on a static snapshot, the gap between Graph RAG and SQL RAG on reasoning questions may grow when live data is introduced: the relationships the authors hand-built (connecting flight number, same ramp) are exactly the edges a real-time feed would update, and multi-hop queries would then need temporal reasoning over changing times.
  • The 30-question reasoning set is small enough that the 68.75% vs 6.25% gap could shift with a different sample; a larger, time-varying benchmark would tell whether Graph RAG's reasoning advantage is structural or dataset-specific.
  • A held-out evaluation that excludes the 47 SQL, 60 classification, and 20 hallucination few-shot examples, with blind double scoring, would be the direct test of whether the reported ordering survives clean methodology.
  • The same structured-query design could transfer to other operations domains with dense jargon and tabular data (warehouses, hospitals, control rooms), where "look it up in the database" is safer than "read what the model remembers."
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents a case study comparing three RAG pipelines (traditional RAG with BM25+GPT-4, SQL RAG, and Graph RAG) for a conversational AI system assisting airport staff at Schiphol. It reports that Graph RAG achieves the highest accuracy on straightforward questions (91.49% execution match) and on a reasoning-question subset (68.75%), with fewer hallucinations than traditional RAG, and recommends SQL RAG and Graph RAG for airport use. The authors also describe a question-classification pipeline for ambiguous and jargon-heavy queries. The evaluation is based on self-created datasets and manual True/False scoring, with the data not publicly released.

Significance. If the evaluation were trustworthy, the comparison would be a useful applied data point for practitioners choosing RAG architectures in domain-specific, high-safety settings. The paper's strengths include its industrial context, the detailed prompt templates in the appendix, and a candid limitations section acknowledging the prototype nature and single-airport scope. However, the central accuracy claims are not currently verifiable because the evaluation protocol is under-specified and internally inconsistent.

major comments (5)
  1. [Section 3, Tables 3-4] Section 3 states that 100-200 question-answer pairs were randomly selected for the straightforward dataset, and Appendix A.1.2 states 150-200, but the SQL and Graph RAG percentages in Tables 3 and 4 are exact fractions with denominator 47 (e.g., 80.85% = 38/47, 91.49% = 43/47). The paper never states that exactly 47 questions were used for these evaluations, so the sample size and the basis of the central comparison are unknowable from the manuscript.
  2. [Section 5.1.1, Section 4] The SQL RAG evaluation uses 47 manually created few-shot examples (Section 5.1.1), and the denominator implied by Table 3 is also 47; the paper does not disclose whether the few-shot demonstrations are disjoint from the evaluation questions. If they overlap or coincide, the reported EM/EX scores reflect memorization rather than generalization, and the comparison between RAG methods collapses. The same omission applies to the 60 question-classification examples and the 20 hallucination-reduction examples.
  3. [Section 5.1.3, Table 5] The text states that a dataset of 30 reasoning questions was created, but the percentages in Table 5 (68.75%, 6.25%, 9.38%) are not possible with a denominator of 30; the smallest denominator that yields these percentages is 32 (22/32, 2/32, 3/32). This internal inconsistency makes the size of the reasoning dataset and the headline Graph RAG advantage on reasoning questions unverifiable.
  4. [Abstract, Section 5.1.1] The abstract reports 84.84% accuracy for traditional RAG (BM25 + GPT-4), while Section 5.1.1 reports 84.40% for the same configuration. Even a small numerical discrepancy in the headline result indicates a lack of care in reporting and reduces confidence in the other metrics.
  5. [Section 5.1.1, Appendix A.1.2] All accuracy numbers are based on manual True/False scoring by the authors, with no reported inter-annotator agreement, no error bars, and no significance tests; differences such as 86.54% vs. 85.78% in Table 1 may be within noise, and the claimed superiority of Graph RAG over SQL RAG on the straightforward set rests on a 10-point gap that is not tested for statistical significance.
minor comments (6)
  1. [Introduction] The phrase "Which fights are at ramp D07?" appears to be a typo for "Which flights are at ramp D07?".
  2. [Section 2.2] The phrase "natural language automatically questions(NLQs)" is a typographical error; it should read "natural language questions (NLQs)".
  3. [Section 5.1.2, Appendix A.2.1] The acronym "TW AQ" is inconsistently spaced, and the category names differ between the main text (e.g., "Time With Ambiguous Flight Number Questions") and the appendix (e.g., "Time with Aircraft Questions"); these should be harmonized.
  4. [Appendix A.1.2] The sentence "To enrich our questions, we used language models to generate more variations." is repeated verbatim in consecutive sentences.
  5. [Section 3, Table 1] The dataset is described as containing "thousands of flights," but Table 1 reports "Total Rows 1350" for the retrieval experiments; the relationship between these numbers should be clarified.
  6. [Section 5.1.1, Appendix A.1.2] The statement "we randomly selected 100 questions for the experiment" conflicts with the 150-200 pairs mentioned elsewhere; the exact number of questions used in each experiment should be specified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical comparison, and the methodological concerns about dataset construction and few-shot overlap affect reproducibility rather than derivational circularity.

full rationale

This paper is an empirical evaluation of three RAG pipelines on a self-constructed airport-domain QA dataset. It does not present a formal derivation in which a predicted quantity is equivalent to an input by construction. The reported accuracies in Tables 1-5 are measurements from experiments, not values derived from fitted parameters. The suspiciously matching denominator of 47 in Tables 3-4 and the number of few-shot examples, as well as the arithmetic inconsistency in Table 5, are serious reproducibility and validity concerns, but the paper never states that the few-shot examples are the evaluation set, so the specific reduction required for a circularity claim cannot be exhibited from the text. The paper also does not rely on load-bearing self-citations, imported uniqueness theorems, or ansatze smuggled in via citations. Its recommendations follow from its own experimental observations. Therefore, per the hard rule that circularity must be exhibited by quotation and specific reduction, the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper rests on a private dataset, manually generated ground truth, and hand-tuned prompts; no external benchmark or independent labels are used.

free parameters (5)
  • Hybrid search weight ratio (BM25:FAISS) = 9:1
    Chosen after comparing 9:1, 5:5, and 1:9 on the same test set; the 9:1 ratio gave the highest retrieval accuracy and is selected on test data.
  • Few-shot example count for SQL RAG = 47
    Manually created few-shot examples for prompting; count chosen by hand, no ablation.
  • Few-shot example count for question classification = 60
    Selected to fit in the context window; not ablated or justified independently.
  • Few-shot example count for hallucination reduction = 20
    Chosen by hand to reduce traditional RAG hallucinations; no ablation.
  • Time offset in ambiguous question generation = 1 hour before scheduled block time
    Used to construct time-based ambiguous questions; chosen by hand, not justified.
assumptions (5)
  • domain assumption The Schiphol flight dataset is representative of airport operations and the manually generated question templates cover real operator queries.
    The paper builds its ground truth from this dataset and generalizes from it to airport environments.
  • domain assumption Manual True/False labeling by the authors is reliable and consistent.
    Accuracy numbers are computed from manual labels; no inter-annotator agreement is reported.
  • ad hoc to paper Few-shot examples used in prompts are disjoint from the test questions.
    The paper never states this split; if examples overlap with the test set, the reported accuracies are inflated.
  • ad hoc to paper Prompt engineering effort is balanced across the three RAG methods.
    Each method receives custom hand-written prompts; no evidence that the effort or quality is comparable.
  • domain assumption LLM outputs are stable enough to evaluate a single generation pass.
    Accuracy is measured on one answer per question without repeated sampling or robustness checks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain." pith.science (2026). https://pith.science/paper/Y35JHETZ

@misc{pith2026250513006,
  author       = {Pith},
  title        = {Pith review of: Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y35JHETZ}},
  note         = {Machine review of arXiv:2505.13006}
}
read the original abstract

Airports from the top 20 in terms of annual passengers are highly dynamic environments with thousands of flights daily, and they aim to increase the degree of automation. To contribute to this, we implemented a Conversational AI system that enables staff in an airport to communicate with flight information systems. This system not only answers standard airport queries but also resolves airport terminology, jargon, abbreviations, and dynamic questions involving reasoning. In this paper, we built three different Retrieval-Augmented Generation (RAG) methods, including traditional RAG, SQL RAG, and Knowledge Graph-based RAG (Graph RAG). Experiments showed that traditional RAG achieved 84.84% accuracy using BM25 + GPT-4 but occasionally produced hallucinations, which is risky to airport safety. In contrast, SQL RAG and Graph RAG achieved 80.85% and 91.49% accuracy respectively, with significantly fewer hallucinations. Moreover, Graph RAG was especially effective for questions that involved reasoning. Based on our observations, we thus recommend SQL RAG and Graph RAG are better for airport environments, due to fewer hallucinations and the ability to handle dynamic questions.

Figures

Figures reproduced from arXiv: 2505.13006 by the authors.

Figure 1
Figure 1. Traditional RAG Method [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. SQL RAG Method [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Graph RAG Method [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Method on Ambiguous question dataset 5 Results In this section, we present the experimental results, structured using our research questions. 5.1 RQ1: How to handle flight data for different questions? 5.1.1 Straightforward questions [PITH_FULL_IMAGE:figures/full_fig_…
Figure 5
Figure 5. Figure 5: Confusion Matrix of Question Classifications [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Question Classifications To evaluate how effectively different retrieval methods performed, several tests were run. Two ground truth datasets were created: a straight￾forward dataset and a complicated and ambigu￾ous dataset. The straightforward dataset con￾tains questi…
Figure 7
Figure 7. Figure 7: Dataset generation examples of straightfor [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Dataset generation examples of compli￾cated/ambiguous questions for example, "Which gate is assigned to the 0164 flight?", "At what gate is the 0164?". These flight numbers might be incomplete, possibly consisting only of numbers, or may include letters but do not dire…
Figure 9
Figure 9. Figure 9: Prompts of Question Classifications types or constraints. The ODP style emphasizes straightforward task instructions, such as “Include only valid SQL syntax, without additional format￾ting or explanation” guiding the model to generate SQL queries directly without unnec…
Figure 10
Figure 10. Figure 10: ODP Prompt for SQL RAG [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: CRP Prompt for SQL RAG treat connecting flights as merely a column in the table, limiting access to further relational informa￾tion. A.2.4 Hallucination Analyses This section provides additional information about hallucinations in different RAG methods. The hallucinat…
Figure 12
Figure 12. Figure 12: Hallucination case for traditional RAG [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: The same case for SQL RAG [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: The same case for Graph RAG [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Graph RAG Prompt [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Graph RAG in Reasoning Questions [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 6 canonical work pages

  1. [1]

    Barrasa, J

    J. Barrasa, J. Webber, and J. Webber. 2023. https://books.google.de/books?id=Ztb5zgEACAAJ Building Knowledge Graphs: A Practitioner's Guide . O'Reilly

    work page 2023

  2. [2]

    Xiaoyin Chen and Sam Wiseman. 2023. https://arxiv.org/abs/2305.14087 Bm25 query augmentation learned end-to-end . Preprint, arXiv:2305.14087

    work page Pith review arXiv 2023

  3. [3]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. https://arxiv.org/abs/2404.16130 From local to global: A graph rag approach to query-focused summarization . Preprint, arXiv:2404.16130

    arXiv 2024

  4. [4]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. https://arxiv.org/abs/2308.15363 Text-to-sql empowered by large language models: A benchmark evaluation . Preprint, arXiv:2308.15363

    arXiv 2023

  5. [5]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. https://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey . Preprint, arXiv:2312.10997

    arXiv 2024

  6. [6]

    Godwin George and Rajeev Rajan. 2022. https://doi.org/10.1109/INDICON56171.2022.10039758 A faiss-based search for story generation . In 2022 IEEE 19th India Council International Conference (INDICON), pages 1--6

    arXiv 2022

  7. [7]

    Chunxi Guo, Zhiliang Tian, Jintao Tang, Shasha Li, Zhihua Wen, Kaixuan Wang, and Ting Wang. 2023. https://arxiv.org/abs/2307.05074 Retrieval-augmented gpt-3.5-based text-to-sql framework with sample-aware prompting and dynamic revision chain . Preprint, arXiv:2307.05074

    work page Pith review arXiv 2023

  8. [8]

    Sven Jacobs and Steffen Jaschke. 2024. https://arxiv.org/abs/2405.06681 Leveraging lecture content for improved feedback: Explorations with gpt-4 and retrieval augmented generation . Preprint, arXiv:2405.06681

    arXiv 2024

Show all 23 references
  1. [9]

    Omid Jafari, Preeti Maurya, Parth Nagarkar, Khandker Mushfiqul Islam, and Chidambaram Crushev. 2021. https://arxiv.org/abs/2102.08942 A survey on locality sensitive hashing algorithms and their applications . Preprint, arXiv:2102.08942

    2021 arXiv

  2. [10]

    Hervé Jegou, Matthijs Douze, and Jeff Johnson. 2017. https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/ Faiss: A library for efficient similarity search . Facebook AI Research, Data Infrastructure, ML Applications. Access...

    2017

  3. [11]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. https://arxiv.org/abs/2005.11401 Retrieval-augmented generation for knowledge-int...

    2021 arXiv

  4. [12]

    Cai-zhi Liu, Yan-xiu Sheng, Zhi-qiang Wei, and Yong-Quan Yang. 2018. https://doi.org/10.1109/IRCE.2018.8492945 Research of text classification based on improved tf-idf algorithm . In 2018 IEEE International Conference of Intelligent Robotic and Control Engineering (IRCE), page...

    2018

  5. [13]

    Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2023. https://doi.org/10.48550/arXiv.2309.17050 Interpretable long-form legal question answering with retrieval-augmented large language models . In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligen...

    2023 doi arXiv

  6. [14]

    Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and Jiawei Han. 2020. https://arxiv.org/abs/2010.00117 Multi-document summarization with maximal marginal relevance-guided reinforcement learning . Preprint, arXiv:2010.00117

    2020 arXiv

  7. [15]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. https://arxiv.org/abs/1301.3781 Efficient estimation of word representations in vector space . Preprint, arXiv:1301.3781

    2013 arXiv

  8. [16]

    Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. https://arxiv.org/abs/2204.00498 Evaluating the text-to-sql capabilities of large language models . Preprint, arXiv:2204.00498

    2022 arXiv

  9. [17]

    Heidi Steen Robert Lee. 2024. Hybrid search using vectors and full text in azure ai search. https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview

    2024

  10. [18]

    Stephen Robertson. 2004. https://doi.org/10.1108/00220410410560582 Understanding inverse document frequency: On theoretical arguments for idf . Journal of Documentation - J DOC, 60:503--520

    2004 doi

  11. [19]

    Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: Bm25 and beyond . Foundations and Trends in Information Retrieval, 3:333--389

    2009 doi

  12. [20]

    Bhaskarjit Sarmah, Benika Hall, Rohan Rao, Sunil Patel, Stefano Pasquali, and Dhagash Mehta. 2024. https://arxiv.org/abs/2408.04948 Hybridrag: Integrating knowledge graphs and vector retrieval augmented generation for efficient information extraction . Preprint, arXiv:2408.04948

    2024 arXiv

  13. [21]

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. https://arxiv.org/abs/2401.11817 Hallucination is inevitable: An innate limitation of large language models . Preprint, arXiv:2401.11817

    2024 arXiv

  14. [22]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  15. [23]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.