{"id":"3283b6d8-b70f-4690-9503-ce56bed2192b","arxiv_id":"2505.13006","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"On a Schiphol flight-information test set, knowledge-graph RAG (91.49%) beat SQL RAG (80.85%) and traditional RAG (84.84%) on accuracy and did far better on reasoning questions (68.75% vs 6.25% and 9.38%).","lead":"This paper compares three ways of building a chatbot for airport staff: pulling flight info from text, from an SQL database, or from a knowledge graph. It reports that the knowledge-graph version answered more questions correctly, especially ones that require reasoning, and produced fewer made-up answers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 and Table 4 report accuracies with denominator 47 — exactly the number of few-shot examples — and Table 5's percentages are impossible for the stated 30-question reasoning set; the central RAG comparison is unverifiable without a disclosed train/test split.","rationale":"The reader's weakest assumption identified the lack of train/test split and potential few-shot leakage as the central vulnerability. My independent reading of the tables confirms this is not a mere assumption: the denominators in Tables 3 and 4 are numerically identical to the stated few-shot count (47), which strongly suggests the evaluation was performed on the demonstration examples themselves. The paper provides no alternative explanation, and the absence of code/data makes this unresolvable from the manuscript. This directly undermines the quantitative foundation of the central claim. In addition, Table 5's percentages are mathematically incompatible with the stated '30 reasoning questions' in Section 5.1.3, corroborating that the numerical reporting is unreliable. These issues are more severe than a minor methodological weakness because the paper's recommendation (Graph RAG and SQL RAG are safer for airport environments) is based entirely on these numbers. The hallucination-reduction claim, while unquantified, is secondary. I therefore agree with the reader's REJECT verdict and see no reason to change it.","tokens_in":12206,"tokens_out":3908,"duration_ms":37212,"concrete_test":"Request from the authors (or reconstruct from the released prompts if code/data become available) the exact list of 47 few-shot demonstration questions and the exact list of evaluation questions used for Tables 3 and 4, and compute their overlap. If any evaluation question appears among the few-shot demonstrations, the reported accuracies are inflated and the Graph-RAG superiority claim is unsupported. Separately, re-derive Table 5's percentages from the raw per-question results and check whether the reasoning dataset contained 30 or 32 questions; if the denominator is 30, recompute the percentages and report the corrected values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Graph RAG outperforms SQL RAG and traditional RAG (91.49% vs 80.85% on straightforward questions, 68.75% vs 6.25% on reasoning questions) rests on accuracy numbers whose evaluation setup is never specified. In Section 5.1.1, the text states: 'Few-shot learning was applied using 47 manually created examples, including questions, SQL queries, and corresponding answers. With CRP, GPT-4 achieves the highest performance (EM: 78.72%, EX: 80.85%).' These percentages are exactly 37/47 and 38/47, respectively. Table 4 uses the same denominator: 14.89% EM = 7/47 and 91.49% EX = 43/47. The only stated sample sizes elsewhere are 'around 100 to 200 pairs' (Section 3) and 'randomly selected 150-200 question-answer pairs' (Appendix A.1.2). The paper never discloses whether the 47 few-shot demonstrations are disjoint from the 47 evaluation questions. If they are the same, the high accuracies reflect memorization, not generalization, and the comparison collapses. Additionally, Section 5.1.3 claims 'We created a dataset of 30 reasoning questions,' yet Table 5 reports 68.75%, 6.25%, and 9.38%, which are not achievable with a denominator of 30 (they require at least a denominator of 32, e.g., 22/32, 2/32, 3/32). This internal inconsistency makes the exact sample size and the reasoning-specific result unknowable from the paper. The hallucination-reduction recommendation (Section 5.2) is also unquantified, but the primary load-bearing issue is the untestable evaluation protocol for the headline accuracy figures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a case study comparing three RAG pipelines (traditional RAG with BM25+GPT-4, SQL RAG, and Graph RAG) for a conversational AI system assisting airport staff at Schiphol. It reports that Graph RAG achieves the highest accuracy on straightforward questions (91.49% execution match) and on a reasoning-question subset (68.75%), with fewer hallucinations than traditional RAG, and recommends SQL RAG and Graph RAG for airport use. The authors also describe a question-classification pipeline for ambiguous and jargon-heavy queries. The evaluation is based on self-created datasets and manual True/False scoring, with the data not publicly released.","tokens_in":12658,"tokens_out":4599,"duration_ms":38848,"significance":"If the evaluation were trustworthy, the comparison would be a useful applied data point for practitioners choosing RAG architectures in domain-specific, high-safety settings. The paper's strengths include its industrial context, the detailed prompt templates in the appendix, and a candid limitations section acknowledging the prototype nature and single-airport scope. However, the central accuracy claims are not currently verifiable because the evaluation protocol is under-specified and internally inconsistent.","major_comments":[{"comment":"Section 3 states that 100-200 question-answer pairs were randomly selected for the straightforward dataset, and Appendix A.1.2 states 150-200, but the SQL and Graph RAG percentages in Tables 3 and 4 are exact fractions with denominator 47 (e.g., 80.85% = 38/47, 91.49% = 43/47). The paper never states that exactly 47 questions were used for these evaluations, so the sample size and the basis of the central comparison are unknowable from the manuscript.","section":"Section 3, Tables 3-4"},{"comment":"The SQL RAG evaluation uses 47 manually created few-shot examples (Section 5.1.1), and the denominator implied by Table 3 is also 47; the paper does not disclose whether the few-shot demonstrations are disjoint from the evaluation questions. If they overlap or coincide, the reported EM/EX scores reflect memorization rather than generalization, and the comparison between RAG methods collapses. The same omission applies to the 60 question-classification examples and the 20 hallucination-reduction examples.","section":"Section 5.1.1, Section 4"},{"comment":"The text states that a dataset of 30 reasoning questions was created, but the percentages in Table 5 (68.75%, 6.25%, 9.38%) are not possible with a denominator of 30; the smallest denominator that yields these percentages is 32 (22/32, 2/32, 3/32). This internal inconsistency makes the size of the reasoning dataset and the headline Graph RAG advantage on reasoning questions unverifiable.","section":"Section 5.1.3, Table 5"},{"comment":"The abstract reports 84.84% accuracy for traditional RAG (BM25 + GPT-4), while Section 5.1.1 reports 84.40% for the same configuration. Even a small numerical discrepancy in the headline result indicates a lack of care in reporting and reduces confidence in the other metrics.","section":"Abstract, Section 5.1.1"},{"comment":"All accuracy numbers are based on manual True/False scoring by the authors, with no reported inter-annotator agreement, no error bars, and no significance tests; differences such as 86.54% vs. 85.78% in Table 1 may be within noise, and the claimed superiority of Graph RAG over SQL RAG on the straightforward set rests on a 10-point gap that is not tested for statistical significance.","section":"Section 5.1.1, Appendix A.1.2"}],"minor_comments":[{"comment":"The phrase \"Which fights are at ramp D07?\" appears to be a typo for \"Which flights are at ramp D07?\".","section":"Introduction"},{"comment":"The phrase \"natural language automatically questions(NLQs)\" is a typographical error; it should read \"natural language questions (NLQs)\".","section":"Section 2.2"},{"comment":"The acronym \"TW AQ\" is inconsistently spaced, and the category names differ between the main text (e.g., \"Time With Ambiguous Flight Number Questions\") and the appendix (e.g., \"Time with Aircraft Questions\"); these should be harmonized.","section":"Section 5.1.2, Appendix A.2.1"},{"comment":"The sentence \"To enrich our questions, we used language models to generate more variations.\" is repeated verbatim in consecutive sentences.","section":"Appendix A.1.2"},{"comment":"The dataset is described as containing \"thousands of flights,\" but Table 1 reports \"Total Rows 1350\" for the retrieval experiments; the relationship between these numbers should be clarified.","section":"Section 3, Table 1"},{"comment":"The statement \"we randomly selected 100 questions for the experiment\" conflicts with the 150-200 pairs mentioned elsewhere; the exact number of questions used in each experiment should be specified.","section":"Section 5.1.1, Appendix A.1.2"}],"recommendation":"reject","confidential_remarks":"The manuscript is closer to an industry engineering report than a research contribution in its present form. Even if the reporting inconsistencies were corrected, the evaluation would need to be re-run on a properly disclosed train/test split with statistical support before the central comparison could be assessed. I would not encourage a major revision unless the authors can provide the missing evaluation metadata and corrected numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's most useful contribution is the structured comparison of three RAG pipelines on a real airport flight dataset, with a sensible taxonomy of questions: straightforward, ambiguous/jargon, and dynamic reasoning. The observation that BM25 outperforms semantic retrieval in this domain because of specialized terminology is plausible, and the qualitative claim that Graph RAG handles multi-hop reasoning better than a flat table or SQL feels intuitive. Those are legitimate takeaways, but the quantitative support is shaky.\n\nThe numbers do not add up. The abstract reports 84.84% accuracy for traditional RAG; the body says 84.40%. Tables 3 and 4 report accuracies that correspond to a denominator of 47, which is exactly the number of few-shot examples used for SQL RAG. The paper never states how many questions were in the evaluation set, nor whether the few-shot examples were disjoint from it. If the same 47 questions were used as demonstrations and test cases, the high scores reflect memorization, not generalization. The reasoning dataset is described as 30 questions, but Table 5 lists 68.75%, 6.25%, and 9.38%, which are only possible with a denominator of 32. That is a concrete mathematical inconsistency, not a rounding error. The hallucination reduction claim is also unquantified: the only number given is roughly 10% for traditional RAG, with no comparable data for SQL or Graph RAG. No code or data is released, so nothing can be independently checked.\n\nThese issues affect the central recommendation to choose Graph RAG over SQL RAG. The qualitative argument may still hold, but the reported accuracy values do not give the reader a reliable basis to accept it. The paper is honest in its limitations section and does not overstate the maturity of the system, which makes me think the errors are sloppy reporting rather than intentional deception. Still, the missing split disclosure is load-bearing.\n\nFor a reader, this is a useful cautionary example of how evaluation reporting can go wrong. I would not cite the numerical results, but I might bring it to a reading group to discuss what a proper benchmark should include. If the authors fix the denominator inconsistencies and clearly separate few-shot examples from test questions, this could become a passable applied case study. As is, it needs major revision and a skeptical referee. I would send it out for review only if the venue is willing to push for that transparency; otherwise, a desk reject with detailed feedback is defensible.","headline":"Practical airport RAG comparison with plausible qualitative takeaways, but the headline accuracy numbers are internally inconsistent and rest on an undisclosed evaluation split.","tokens_in":724,"tokens_out":1469,"would_cite":false,"duration_ms":50399,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph RAG answers airport flight queries more accurately than SQL RAG or traditional RAG, and structured-query RAG methods hallucinate less.","keywords":["Retrieval-Augmented Generation","Text-to-SQL","Knowledge Graph","Conversational AI","Airport operations","Hallucination reduction","Flight information systems","Question classification"],"falsifier":"Re-run the evaluation on a held-out question set that removes every example used in the prompts, score the answers by two independent annotators who do not know which RAG method produced them, and compare Graph RAG's execution-match accuracy to SQL RAG's on the straightforward set. If Graph RAG no longer exceeds SQL RAG (91.49% vs 80.85%), the headline ordering was an artifact of prompt overlap or scoring bias; if the gap persists, the central claim survives.","tokens_in":12035,"feed_emoji":"✈️","tokens_out":10756,"duration_ms":101139,"temperature":0.7,"pith_summary":"Airports are high-risk, jargon-heavy environments where a conversational assistant for ground staff must answer from flight data without inventing facts. This paper builds three such assistants — traditional retrieval-augmented generation (RAG), text-to-SQL RAG, and knowledge-graph RAG — and evaluates them on flight-information questions. It claims that Graph RAG reaches 91.49% accuracy on straightforward queries and 68.75% on reasoning queries, outperforming SQL RAG (80.85% and 6.25%) and traditional RAG (84.84% and 9.38%). The central argument is that translating a question into a structured database query, instead of feeding the model a slab of retrieved text, sharply reduces hallucinations. If correct, the finding gives airport operators a concrete reason to prefer graph- or SQL-backed assistants over plain document retrieval.","feed_headline":"Graph RAG wins airport flight-QA accuracy, with fewer hallucinations","feed_subtitle":"Converting questions to SQL or Cypher keeps LLM answers grounded in real flight data.","key_machinery":"The mechanism that carries the result is structured-query conversion with schema-aware prompting. For SQL RAG, the Code Representation Prompt embeds full CREATE TABLE statements with column types and keys, so the LLM generates SQL and the database returns only the exact rows requested. For Graph RAG, the same idea works over a property graph: the LLM is given the graph schema (extracted with the APOC `apoc.meta.schema()` call) and writes Cypher queries that traverse relationships — connecting flight, same ramp, gate — rather than scanning a text index. This is what lets a question like \"What is the expected on-ramp time of DL0123's connecting flight?\" be answered by following a link in the graph. The comparison of exact-match against execution-match scores shows the trade-off: SQL queries are syntactically rigid (high EM, moderate EX), while Cypher queries are syntactically flexible but executionally accurate (low EM, high EX).","core_discovery":"On the paper's own terms, the discovery is a ranking of RAG architectures for a specific high-stakes domain. Using a flight-information table from an airport operation and a manually built question set, the authors report that Graph RAG, implemented as an LLM writing Cypher queries against a Neo4j knowledge graph, achieves an execution match of 91.49% on straightforward questions and 68.75% on a 30-question reasoning set. SQL RAG, where the LLM writes SQL using a detailed CREATE TABLE schema prompt, scores 80.85% execution match on straightforward questions but only 6.25% on reasoning questions. Traditional RAG with BM25 + GPT-4 reaches 84.84% on straightforward questions and 9.38% on reasoning, and it is the pipeline in which the authors observed the model producing destinations not present in the dataset. The paper interprets the low exact-match but high execution-match scores of Graph RAG as evidence that graph query languages let the model express the same correct answer in different syntactic forms, and it recommends SQL RAG and Graph RAG over traditional RAG for airport environments because structured queries feed the LLM only verified rows.","pith_inferences":["Because the paper tests on a static snapshot, the gap between Graph RAG and SQL RAG on reasoning questions may grow when live data is introduced: the relationships the authors hand-built (connecting flight number, same ramp) are exactly the edges a real-time feed would update, and multi-hop queries would then need temporal reasoning over changing times.","The 30-question reasoning set is small enough that the 68.75% vs 6.25% gap could shift with a different sample; a larger, time-varying benchmark would tell whether Graph RAG's reasoning advantage is structural or dataset-specific.","A held-out evaluation that excludes the 47 SQL, 60 classification, and 20 hallucination few-shot examples, with blind double scoring, would be the direct test of whether the reported ordering survives clean methodology.","The same structured-query design could transfer to other operations domains with dense jargon and tabular data (warehouses, hospitals, control rooms), where \"look it up in the database\" is safer than \"read what the model remembers.\""],"forward_implications":["A graph-backed conversational assistant could be deployed at airport operations desks to answer standard flight queries at roughly 91% execution-match accuracy, reducing the need for staff to write database queries.","For airports that already have relational databases, SQL RAG offers a safer alternative to document-style RAG on straightforward questions, but its 6.25% accuracy on reasoning questions means it should not handle multi-step time or connection queries alone.","The hallucination argument implies that any high-safety deployment of an LLM assistant should prefer generation over database query results over generation over retrieved passages, whenever the data is tabular or relational.","The reasoning-query results suggest that representing flight relationships as graph edges, rather than as columns in a table, is what enables multi-hop temporal questions such as connecting-flight and same-ramp queries."],"supporting_citations":[{"why":"Defines the traditional RAG setup that the paper's baseline pipeline builds on.","marker":"(Lewis et al., 2021)"},{"why":"Supplies the text-to-SQL generation approach used in the SQL RAG pipeline.","marker":"(Guo et al., 2023)"},{"why":"Introduces Graph RAG, the method whose query-focused reasoning the paper adapts to flight data.","marker":"(Edge et al., 2024)"},{"why":"Provides the Code Representation Prompt and OpenAI Demonstration Prompt styles the SQL RAG experiments compare.","marker":"(Gao et al., 2023)"},{"why":"Formalizes BM25, the retrieval function that gives the best traditional-RAG accuracy in the paper's tests.","marker":"(Robertson and Zaragoza, 2009)"},{"why":"Underpins the paper's motivation that hallucinations are an inherent LLM limitation that structured retrieval can reduce.","marker":"(Xu et al., 2024)"},{"why":"Justifies the choice of Neo4j property graphs over RDF for the Graph RAG implementation.","marker":"(Barrasa et al., 2023)"}],"fun_headline_variants":["Graph RAG tops airport QA with fewer hallucinations","Airport flight queries: Graph RAG beats traditional and SQL","Graph RAG: 91% accuracy, less hallucination in airport AI","Airport chatbot: Graph RAG is most accurate and grounded"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy ranking collapses if the few-shot example questions embedded in the prompts (47 for SQL RAG, 60 for question classification, 20 for hallucination reduction) overlap with the questions used for evaluation, or if the manual True/False scoring was inconsistent, because the paper does not describe this split.","fun_headline_variants_meta":{"raw":{"variants":["Graph RAG tops airport QA with fewer hallucinations","Airport flight queries: Graph RAG beats traditional and SQL","Graph RAG: 91% accuracy, less hallucination in airport AI","Airport chatbot: Graph RAG is most accurate and grounded"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1326,"prompt_tokens":984,"completion_tokens":342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":283}},"tokens_in":600,"tokens_out":342,"duration_ms":3401,"temperature":1.0,"reasoning_tokens":283,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:21:36.644580+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on a held-out question set that removes every example used in the prompts, score the answers by two independent annotators who do not know which RAG method produced them, and compare Graph RAG's execution-match accuracy to SQL RAG's on the straightforward set. If Graph RAG no longer exceeds SQL RAG (91.49% vs 80.85%), the headline ordering was an artifact of prompt overlap or scoring bias; if the gap persists, the central claim survives.","supporting_citations":[{"cited_title":"Retrieval-augmented GPT-3.5-based Text-to-SQL Framework with Sample-aware Prompting and Dynamic Revision Chain","cited_arxiv_id":"2307.05074","evidence_quote":"Supplies the text-to-SQL generation approach used in the SQL RAG pipeline."}],"review_version":1}