Design and testing of an agent chatbot supporting decision making with public transport data

Bruno Zamengo; Daniele Irto; Francesco Silvestri; Luca Fantin; Marco Antonelli; Margherita Cesetti

REVIEW 3 major objections 5 minor 14 references

Design and testing of an agent chatbot supporting decision making with public transport data

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An agent chatbot with SQL tools answers 53% of transport questions correctly.

desk verdict A useful applied text-to-SQL paper for GTFS data whose headline accuracy figure is misleadingly scoped and rests on a single gold-query interpretation that the authors themselves know is ambiguous. read the letter →

arxiv 2505.22698 v1 pith:U6SIV2L3 submitted 2025-05-28 cs.AI

Luca Fantin , Marco Antonelli , Margherita Cesetti , Daniele Irto , Bruno Zamengo , Francesco Silvestri This is my paper

classification cs.AI

keywords agentchatbottext-to-SQLpublictransportdataGTFSlargelanguagemodelLLMevaluationmethodologyquerycorrectnessdecisionsupport

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports on a chatbot that lets non-technical users ask questions about public transport schedules in plain language and receive answers as text, plots, or maps. The chatbot is built as an LLM-based agent: a core language model plans and calls tools that generate, check, and execute SQL queries against a database assembled from public GTFS transport data (the standard format for schedules). To measure whether the system works, the authors create 146 test questions from three templates, write a reference gold SQL query for each, and compare the chatbot's results against those gold queries. On the two simpler templates the chatbot returns the exact correct result set 59 times out of 112, about 53%; on the more complex aggregate template it succeeds in 6 of 34 cases. The authors interpret this as evidence that the agent is good at selecting relevant example queries and adapting them, while queries that require larger rewrites remain the weak point.

What carries the argument

The load-bearing component is the agent architecture: a core LLM surrounded by an orchestration layer and by tools that generate a SQL query, check it for syntactic and logical errors, execute it, and build plots or maps from the retrieved data. A structured XML-like prompt carries the full database schema, query rules, and a small set of retrieved example queries that the model is told to adapt. The evaluation mechanism is a question-template workflow: base questions are completed with real data and varied wording, a gold SQL query is manually written for each, and the generated result set or value is compared with the gold one.

What would settle it

Have a second annotator independently write gold SQL queries for the same 146 test questions and measure how often two independent gold queries return identical results; if they often differ, the reported accuracy figures are mostly measuring interpretation ambiguity rather than chatbot correctness.

Watch

Extended reading notes

Core claim

The paper's central claim is that a natural-language chatbot for public transport data can be built from a single LLM agent that delegates query generation, error checking, execution, plotting, and map drawing to separate tools, and that the same system can be evaluated quantitatively by a template-based test pipeline rather than by subjective impressions. Its measured result is that, for the two simpler question types, the chatbot returns exactly the gold-query result set in 59 of 112 cases, returns a superset in 2 cases, a subset in 10, and a disjoint set in 14; for the aggregate question type it returns the correct value in only 6 of 34 cases, all from two repeated questions. The paper takes this as showing that the chatbot recognizes relevant example queries and modifies them well, but fails when the required query differs substantially from the examples in its prompt.

Load-bearing premise

The load-bearing premise is that each test question has exactly one correct interpretation, so the manually written gold SQL query is the single standard of truth; if the question admits other reasonable readings, a query that follows one of them is counted as an error.

Editorial extensions

If this is right

At the measured 53% exact-match rate on simple lookup questions, the chatbot can already serve as a query assistant for questions such as which routes serve a given municipality, provided a human checks the results.
The stored test questions, generated queries, and retrieved data form a reusable regression benchmark for future versions of the chatbot and for comparing prompt or tool changes.
The error breakdown points to concrete fixes, such as correcting the type mismatch that makes the model treat a textual direction column as an integer, and improving the model's ability to rewrite an example query when the new question adds conditions.
The authors' stated next step is a multi-agent graph architecture in which separate agents handle sub-tasks; they expect this to improve answer accuracy and plot and map creation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the gold query is one specific interpretation, a generated query that computes a different but defensible reading of the question is scored as wrong; the paper leaves implicit that the 53% and 18% figures therefore understate how often the user receives a useful answer.
The same template pipeline could be extended to measure answer-level acceptability, asking annotators whether the returned result set answers the question, which would separate query-construction failures from interpretation ambiguity.
A direct next experiment would be to run the same 146 questions with different underlying language models and prompt variants; the released template dataset makes this comparison cheap and would show whether the bottleneck is the model, the prompt, or the tool workflow.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a solid applied systems paper with a flawed headline number. The genuinely new content is the application of a fairly standard LangChain/GPT-4-Turbo text-to-SQL agent to GTFS public transport data, plus a template-based evaluation pipeline that generates test questions from real data and compares generated SQL against hand-written gold queries. That pipeline is the most reusable part and is described clearly. The paper is honest about the system's weaknesses and doesn't oversell the architecture.

The main problem is the 'around 53%' claim. It is computed over templates 1 and 2 only (59/112 exact matches). Template 3, which requires aggregation, succeeds in 6/34 cases. The text in Section VI-B calls it 'overall' without saying it excludes the harder template; a reader could easily walk away with the wrong impression of end-to-end accuracy. The authors should report the aggregate or at least label the scope.

The deeper issue is the gold-query assumption. Section V itself gives a concrete example where a question admits multiple defensible aggregation choices, and the chatbot has to pick one. Yet Section VI-A says one gold query was manually written for each question, with no inter-annotator check or alternative-answer analysis. For template 3, a different but legitimate interpretation is scored as wrong. That makes the 18% figure, and to a lesser extent the 53%, not well-defined. This is a real weakness, not a manufactured one. The fix is straightforward: document interpretation rules, have more than one annotator write gold queries for the ambiguous cases, or at least qualitatively examine false negatives for alternative interpretations.

Other soft spots are minor in comparison: no baselines (e.g., zero-shot GPT-4-Turbo without the agent tools, or the prior chatbot from [2]), no error bars, and no released test set or code, so the numbers can't be reproduced. Still, the paper is candid about its limitations and the evaluation methodology is a genuine contribution to a practical problem. The citation pattern looks fine; [2] is prior work by the same group and is properly cited.

Who should read this: practitioners building natural-language interfaces to public transport or GTFS data, and people working on evaluation of LLM agents in constrained domains. It deserves a serious referee, but the referee should push for a corrected aggregate accuracy, an ambiguity analysis, and ideally artifact release.

Referee Report

3 major / 5 minor

Summary. This paper presents an agent-based chatbot for interrogating public transport (GTFS) data with natural language. The system uses GPT-4-Turbo as a central decision maker, augmented with tools for SQL generation, syntax checking, query execution, and map creation. The authors also propose an evaluation methodology: three question templates are instantiated with real data (some with added conditions or deliberate errors), a gold SQL query is manually written for each question, and the chatbot's generated query result sets are compared with the gold results. For templates 1–2, 59 of 112 questions produced exact matches, reported as 'around 53%'; for template 3, 6 of 34 questions produced correct results. The paper concludes that the chatbot is good at recognizing and adapting relevant examples but struggles with queries requiring larger modifications.

Significance. If taken at face value, the paper demonstrates a working natural-language interface to GTFS data that integrates SQL querying and geospatial visualization, and it proposes a template-based evaluation pipeline that compares retrieved result sets rather than only surface-level text. The main strengths are the concrete system description and the explicit reporting of failure modes (syntax errors, incorrect joins, partial result sets). The evaluation is a useful first step, but its current form does not yet provide a reliable quantitative estimate of system performance, because the gold-query convention is not uniquely defined and the headline accuracy figure is computed on a subset of the test set. The contribution is likely of interest to practitioners in public transport data analysis and to developers of text-to-SQL systems.

major comments (3)

[VI-B] The sentence 'Overall, the chatbot provided the correct answers for around 53% of the questions' is based on the 59/112 exact matches from templates 1–2 only. When the 6/34 correct results from template 3 (Section VI-C) are included, the overall correctness is 65/146 = 44.5%. This is a scope error in the report of the main result; please correct the aggregate statement and any conclusions that depend on it.
[VI-A and V] The evaluation scores a generated query as correct only if its result set matches a single manually written gold query. However, Section V explicitly acknowledges that the average-trips question admits multiple defensible aggregation choices, such as considering a specific date range, a working day, or all working days, and whether to compute a sum or an average. The paper provides no inter-annotator check, no alternative-answer analysis, and no stated procedure for resolving this ambiguity. As a result, the reported rates (59/112 and 6/34) are not well-defined; a different but equally reasonable gold-query convention could materially change the counts. This is load-bearing because the paper's central claim is the measured accuracy. Please either (a) define a single explicit interpretation rule applied to both gold and generated queries, (b) allow multiple gold interpretations and count an answer as correct if it matches any, or (c) provide a sensitivity analysis of ambiguous cases.
[VI-C and VII] The paper evaluates query correctness, not answer correctness, but repeatedly uses the phrase 'correct answers'. In Section VI-C, the authors note that for invalid route-stop pairs the chatbot's answer 'there are no trips' is correct from the user's point of view even though the query that led to it was wrong. Similarly, in Section VI-B, 10 syntax errors and 17 incorrect queries may still yield answers that are not clearly wrong to a user. The headline result should therefore be phrased as 'query correctness under a single gold-query interpretation' and the abstract and conclusion should be updated to avoid overstating user-facing answer quality.

minor comments (5)

[VI-A] The paper states that 42 questions were generated and that additional conditions were appended to a subset, but it does not report the number of unique questions; the 146 total questions include repeated runs. Please clarify the distinction between unique tests and repeated runs, and discuss the potential dependence in the error counts.
[VI-B] The false positive rates 83% and 75% are reported without a definition of the denominator; please state whether these are the fraction of the generated result set not in the gold set, the fraction of the gold set that is missing, or another quantity.
[VI-C] The statement that the 6 correct cases 'all asked the same two questions' is unclear; please explain what 'the same two questions' means in the context of the 34 template-3 test cases.
[IV-B] The paper describes GPT-4-Turbo as 'currently considered state-of-the-art'; this is difficult to verify and should be supported by a citation or removed.
[References] Reference [5] is given as an arXiv preprint without a version number; please update with the published version if available.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central evaluation is an empirical measurement against manually written gold SQL queries, not a derivation from fitted inputs.

full rationale

No circular step is present. The paper's core result is the measured accuracy of a chatbot on a fixed set of 146 questions. The reference standard is described in Section VI-A: 'a gold query was manually written and executed for each question in the test set to obtain reference data to compare the chatbot's responses to.' This gold query is authored independently of the chatbot's output, so the comparison is not self-justifying. The 53% figure is an observed frequency, not a derived quantity fitted to the data. Section V does note that some questions admit multiple valid interpretations ('whether to consider the trips done in a specific date range, on a specific working day or just consider all working days'), which is a genuine threat to the validity of a single gold query, but it is a measurement-design limitation, not circularity: the chatbot's answer could be better than or worse than the gold on a different defensible reading, and nothing in the comparison is equivalent by construction. The use of GPT-4o to generate test questions and author-written prompt examples around the same templates is self-referential in design, and reference [2] is by overlapping authors, but neither is load-bearing: correctness is adjudicated by externally written gold SQL, not by the model's own outputs or by the cited paper. Hence score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central performance claims rest on assumptions about the evaluation data and the correctness of the gold queries rather than on mathematical axioms. The system itself uses standard software libraries and public data. No fitted constants or invented entities are involved.

assumptions (3)

domain assumption The manually written gold query is the unique correct interpretation of each test question.
Section VI-A: 'a gold query was manually written and executed for each question'. The paper does not consider multiple plausible interpretations (e.g., average over all working days vs. a date range), so a query that matches a different valid interpretation would be scored as incorrect.
domain assumption The GPT-4o-generated question completions from three templates adequately simulate random human user behavior.
Section VI-A: questions were generated by 'a version of ChatGPT based on the GPT-4o LLM ... while simulating a random human-user behavior'. This is an unvalidated proxy for real users.
domain assumption GTFS datasets from TPER (Bologna and Ferrara) and ATM (Milan) are representative of public transport data for the claimed scope.
Section III: only these two agencies' data are used; generalization to other agencies and regions is assumed without evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Design and testing of an agent chatbot supporting decision making with public transport data." pith.science (2026). https://pith.science/paper/U6SIV2L3

@misc{pith2026250522698,
  author       = {Pith},
  title        = {Pith review of: Design and testing of an agent chatbot supporting decision making with public transport data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U6SIV2L3}},
  note         = {Machine review of arXiv:2505.22698}
}

read the original abstract

Assessing the quality of public transportation services requires the analysis of large quantities of data on the scheduled and actual trips and documents listing the quality constraints each service needs to meet. Interrogating such datasets with SQL queries, organizing and visualizing the data can be quite complex for most users. This paper presents a chatbot offering a user-friendly tool to interact with these datasets and support decision making. It is based on an agent architecture, which expands the capabilities of the core Large Language Model (LLM) by allowing it to interact with a series of tools that can execute several tasks, like performing SQL queries, plotting data and creating maps from the coordinates of a trip and its stops. This paper also tackles one of the main open problems of such Generative AI projects: collecting data to measure the system's performance. Our chatbot has been extensively tested with a workflow that asks several questions and stores the generated query, the retrieved data and the natural language response for each of them. Such questions are drawn from a set of base examples which are then completed with actual data from the database. This procedure yields a dataset for the evaluation of the chatbot's performance, especially the consistency of its answers and the correctness of the generated queries.

Figures

Figures reproduced from arXiv: 2505.22698 by the authors.

**Figure 2.** High-level representation of the architecture imple [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Simplified version of the textual prompt given to the agent’s core LLM. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Example of the maps built by the dedicated tool. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Example of a test question, with its original template, [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

[2]

Mobility chatbot: supporting decision making in mobility data with chatbots,

L. Padoan, M. Cesetti, L. Brunello, M. Antonelli, B. Zamengo, and F. Silvestri, “Mobility chatbot: supporting decision making in mobility data with chatbots,” in2024 25th IEEE International Conference on Mobile Data Management (MDM), 2024, pp. 295–300

work page 2024
[1]

Large language model enhanced text- to-sql generation: A survey,

X. Zhu, Q. Li, L. Cui, and Y . Liu, “Large language model enhanced text- to-sql generation: A survey,” https://arxiv.org/abs/2410.06011, 2024

arXiv 2024
[3]

Artificial in- telligence for improving public transport: a mapping study,

A. Jevinger, C. Zhao, J. Persson, and P. Davidsson, “Artificial in- telligence for improving public transport: a mapping study,”Public Transport, vol. 16, pp. 1–60, 11 2023

work page 2023
[4]

Ai in public trans- port,

UITP Union Internationale des Transports Publics, “Ai in public trans- port,” https://www.uitp.org/publications/ai-public-transport/, Mar. 2025

work page 2025
[5]

Mac-sql: A multi-agent collaborative framework for text-to-sql,

B. Wang, C. Ren, J. Yang, X. Liang, J. Bai, L. Chai, Z. Yan, Q.-W. Zhang, D. Yin, X. Sun, and Z. Li, “Mac-sql: A multi-agent collaborative framework for text-to-sql,” https://arxiv.org/abs/2312.11242, 2025

arXiv 2025
[6]

Answerability: A custom metric for evaluating chatbot performance,

P. Gupta, A. A. Rajasekar, A. Patel, M. Kulkarni, A. Sunell, K. Kim, K. Ganapathy, and A. Trivedi, “Answerability: A custom metric for evaluating chatbot performance,” inProceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM). Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics, Dec. 20...

work page 2022
[7]

Open data,

Tper, “Open data,” https://solweb.tper.it/web/tools/open-data/open-data. aspx

work page
[8]

Pubblicazione orari del trasporto pubblico lo- cale in formato gtfs,

ATM, “Pubblicazione orari del trasporto pubblico lo- cale in formato gtfs,” https://www.amat-mi.it/it/servizi/ pubblicazione-orari-trasporto-pubblico-locale-formato-gtfs/

work page

Show all 14 references

[9]

General transit feed specification,

Mobility Data, “General transit feed specification,” https://gtfs.org/
[10]

Confini delle unit `a amministra- tive a fini statistici,

ISTAT, “Confini delle unit `a amministra- tive a fini statistici,” https://www.istat.it/notizia/ confini-delle-unita-amministrative-a-fini-statistici-al-1-gennaio-2018-2/

2018
[11]

Wiesinger, P

J. Wiesinger, P. Marlow, and V . Vuskovic, “Agents,” https://www.kaggle. com/whitepaper-agents, February 2025

2025
[12]

Langchain,

“Langchain,” https://python.langchain.com/docs/introduction/
[13]

New and improved embedding model,

OpenAI, “New and improved embedding model,” https://openai.com/ index/new-and-improved-embedding-model/, Dec. 2022

2022
[14]

How the choice of llm and prompt engineering affects chatbot effectiveness,

L. Pawlik, “How the choice of llm and prompt engineering affects chatbot effectiveness,”Electronics, vol. 14, p. 888, 02 2025

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2]

Mobility chatbot: supporting decision making in mobility data with chatbots,

L. Padoan, M. Cesetti, L. Brunello, M. Antonelli, B. Zamengo, and F. Silvestri, “Mobility chatbot: supporting decision making in mobility data with chatbots,” in2024 25th IEEE International Conference on Mobile Data Management (MDM), 2024, pp. 295–300

work page 2024

[2] [1]

Large language model enhanced text- to-sql generation: A survey,

X. Zhu, Q. Li, L. Cui, and Y . Liu, “Large language model enhanced text- to-sql generation: A survey,” https://arxiv.org/abs/2410.06011, 2024

arXiv 2024

[3] [3]

Artificial in- telligence for improving public transport: a mapping study,

A. Jevinger, C. Zhao, J. Persson, and P. Davidsson, “Artificial in- telligence for improving public transport: a mapping study,”Public Transport, vol. 16, pp. 1–60, 11 2023

work page 2023

[4] [4]

Ai in public trans- port,

UITP Union Internationale des Transports Publics, “Ai in public trans- port,” https://www.uitp.org/publications/ai-public-transport/, Mar. 2025

work page 2025

[5] [5]

Mac-sql: A multi-agent collaborative framework for text-to-sql,

B. Wang, C. Ren, J. Yang, X. Liang, J. Bai, L. Chai, Z. Yan, Q.-W. Zhang, D. Yin, X. Sun, and Z. Li, “Mac-sql: A multi-agent collaborative framework for text-to-sql,” https://arxiv.org/abs/2312.11242, 2025

arXiv 2025

[6] [6]

Answerability: A custom metric for evaluating chatbot performance,

P. Gupta, A. A. Rajasekar, A. Patel, M. Kulkarni, A. Sunell, K. Kim, K. Ganapathy, and A. Trivedi, “Answerability: A custom metric for evaluating chatbot performance,” inProceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM). Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics, Dec. 20...

work page 2022

[7] [7]

Open data,

Tper, “Open data,” https://solweb.tper.it/web/tools/open-data/open-data. aspx

work page

[8] [8]

Pubblicazione orari del trasporto pubblico lo- cale in formato gtfs,

ATM, “Pubblicazione orari del trasporto pubblico lo- cale in formato gtfs,” https://www.amat-mi.it/it/servizi/ pubblicazione-orari-trasporto-pubblico-locale-formato-gtfs/

work page