{"id":"396d3b1a-fee1-48da-a87a-329e6e4f9301","arxiv_id":"2412.06831","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A prompt-only LLM framework that generates and executes Python code answers 90 to 93 percent of 100 GTFS transit-data queries without fine-tuning.","lead":"TransitGPT is a chatbot that turns plain-language transit questions into Python code, runs that code on stored GTFS schedule data, and summarizes the result. The paper reports that with extra prompts and an error-fix loop, GPT-4o and Claude-3.5-Sonnet answer 90 and 93 of 100 transit-data tasks correctly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported accuracy may be inflated because the 12 few-shot prompt examples are never stated to be disjoint from the 100 benchmark tasks; if they overlap, TransitGPT+ sees answer code during evaluation.","rationale":"The paper's most valuable contributions are the clear architecture description and the open-source implementation; these stand regardless of evaluation quibbles. However, the central empirical claim—90% (GPT-4o) and 93% (Claude-3.5-Sonnet) accuracy on 'a wide range' of GTFS queries—is only as strong as the benchmark. The reader flagged ground-truth correctness and manual grading. I agree those are weaknesses, but the more sharply falsifiable issue is the possible overlap between the few-shot examples and the test set, which the paper does not rule out. Since dynamic few-shot selection chooses the three most similar examples to the user's query, if any benchmark task resembles a few-shot item, the model is effectively given the answer code. This is a standard contamination failure in LLM evaluation and would directly inflate the reported numbers. The concrete check can settle it in minutes using the released code. A second, independent validation on real user queries would test representativeness, but the disjointness check is the necessary first step. The category totals in Table 1 and Table 2 differ (e.g., Accessibility 10 vs 9, Performance 14 vs 15), a minor housekeeping issue that does not change the verdict but reinforces the need for the proposed audit. Given the framework is open-source and the architecture plausible, the appropriate verdict remains CONDITIONAL: accept the contribution but require the contamination check and, ideally, independent grading before the accuracy numbers are cited. My read is 'no change' relative to the reader, hence UNCHANGED.","tokens_in":19484,"tokens_out":9413,"duration_ms":91900,"concrete_test":"Check the open-source GitHub repository: compare the 12 few-shot query/response pairs used in the Main Prompt against the 100 benchmark task queries and inputs. If any test item matches a few-shot example verbatim or after placeholder substitution, recompute the reported alpha with those items removed; if the corrected accuracy falls below roughly 80% for either model, the headline claim is not supported. Also, for a random subset of 20 tasks, have an independent GTFS expert produce reference outputs without seeing Yi, and blind-grade TransitGPT outputs against these; if agreement with the expert references is substantially lower than 90%, the manual grading was too lenient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 Step 2(iii) introduces 12 curated few-shot question-answer pairs that are dynamically selected into the Main LLM prompt. Section 4.1 describes a 100-task benchmark with queries Qi, code Yi, and outputs Oi, partly generated from the same LLM style. The paper never states that the 12 few-shot examples are excluded from or non-overlapping with the 100 benchmark tasks. If any of the few-shot pairs (or templated variants) appear in the test set, the Main LLM receives the reference code for those tasks before generating its own code, guaranteeing success on those items and inflating the 90%/93% accuracies. Even without exact overlap, the benchmark was authored by the team that designed the prompt (including the five Custom Functions), so tasks are likely aligned with the exact code style and helper functions the system is prompted to use; the reported accuracy may measure reproduction of the authors' coding solutions rather than general GTFS question-answering. Compounding this, the ground truth Oi is produced by executing Yi, which was drafted by an LLM and manually corrected, and grading is manual with no rubric or inter-rater reliability (Section 4.2). The central claim that TransitGPT 'can answer a wide range' therefore rests on an evaluation whose internal validity is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents TransitGPT, an open-source framework that answers natural-language queries about GTFS Static feeds by prompting an LLM to generate Python code, executing that code in a sandbox against a preprocessed feed object, and summarizing the results with a second LLM. The architecture includes a moderation step, dynamic few-shot selection from 12 curated examples, custom fuzzy-matching/geocoding functions, an error-handling retry loop, and a summary module. The authors evaluate GPT-4o and Claude-3.5-Sonnet in a zero-shot baseline and in a 'TransitGPT+' configuration on an author-created benchmark of 100 tasks across eight categories and five U.S. transit feeds, reporting accuracies of 90% and 93% for the TransitGPT+ configuration, together with token and time metrics. The paper also states its scope limitations, including GTFS-Static-only data and no routing functionality.","tokens_in":19741,"tokens_out":6609,"duration_ms":62762,"significance":"The contribution is timely and practical: a prompt-only program-synthesis solution for transit data, a new 100-task benchmark, and open-source code that others can run and extend. The paper is careful to distinguish its approach from RAG and full-feed prompting, and the prompt design details (data types, custom functions, feed samples) are concrete and valuable. The strongest aspects are the open-source code and benchmark release and the clearly described architecture. However, the headline accuracy numbers currently rest on an evaluation whose internal validity is not fully established: the benchmark ground truth is generated with LLM assistance and manual correction, grading is manual without inter-rater reliability, there are no confidence intervals or repeated runs, and the few-shot examples are not stated to be disjoint from the test tasks. These are fixable with additional analysis and documentation, but they must be addressed before the paper's central claim is taken as established.","major_comments":[{"comment":"The paper never states that the 12 curated few-shot question-answer pairs are disjoint from the 100 benchmark tasks (Qi, Yi, Oi). If any few-shot pair or a templated variant of it appears among the benchmark tasks, the Main LLM in TransitGPT+ would receive the reference code Yi before generating its own code for that task, guaranteeing success on those items and inflating the reported 90% and 93% accuracies. Please explicitly confirm disjointness, list the 12 few-shot examples in an appendix (or otherwise make them public), and re-run the evaluation if any overlap exists.","section":"Section 3, Step 2(iii); Section 4.1"},{"comment":"The ground-truth outputs Oi are produced by executing Yi, where Yi is 'request[ed] from an LLM' and then 'manually correct[ed]' by the authors. The paper does not say which LLM drafted Yi, whether the drafting model was one of the two evaluated models, or whether the manual corrections were substantial. Because the benchmark queries and the prompt (including the five Custom Functions) were authored by the same team, this creates a risk that the evaluation measures the model's ability to reproduce the authors' coding style and helper-function usage rather than general GTFS question-answering. Please specify the drafting model, report the rate and extent of manual corrections, and provide independent validation of a random sample of Oi values (e.g., by a second annotator or by an alternative computation method).","section":"Section 4.1"},{"comment":"The accuracy results are reported from a single run per task with binary manual grading, and no confidence intervals, repeated runs, or inter-rater reliability statistics are given. With per-category sample sizes of 8-22, the category-level rates (e.g., Fare Analysis: 75% for both configurations for GPT-4o; Performance Metrics: 93% baseline vs. 87% TransitGPT+ for Claude-3.5-Sonnet) are noisy, and the overall 90%/93% estimates lack uncertainty bounds. Please report confidence intervals (or repeated runs with variance) and provide a grading rubric, plus a second grader on at least a subset to compute inter-rater agreement.","section":"Section 4.2 and Table 2"}],"minor_comments":[{"comment":"The sentence 'TransitGPT+ outperforms the baseline in all categories except for both LLMs' is not supported by Table 2; for Claude-3.5-Sonnet, Basic Data Operations (80 vs. 80) and Fare Analysis (88 vs. 88) tie, and Performance Metrics decreases (93 vs. 87), while GPT-4o's Fare Analysis ties (75 vs. 75). Please correct this sentence.","section":"Section 4.2, Table 2 text"},{"comment":"Because the baseline is zero-shot with zero retries and TransitGPT+ adds both few-shot examples and retries, the observed improvement cannot be attributed to either component; an ablation would clarify the individual contributions.","section":"Section 4.2"},{"comment":"The paper should pin the exact versions or download dates of the five GTFS feeds used in the benchmark, as well as the code version (e.g., a commit hash or archived DOI), so that the benchmark is reproducible.","section":"Section 4.1 and Appendix B"},{"comment":"There are small presentation issues: footnote 3 has 'poltly.express' instead of 'plotly.express', the find_stops_by_street description says 'Maint Street.' instead of 'Main Street', and footnote 6 has 'Nominatum' instead of 'Nominatim'.","section":"Section 3 and footnotes"},{"comment":"The distinction between the prior 'GTFS Retrieval' benchmark (Devunuri et al., 2024b) and the new 100-task benchmark should be stated explicitly, since Table 1's categories appear to overlap with the earlier work; please clarify which tasks are new to this paper.","section":"Section 2.3 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's fit with the journal is reasonable, but the contribution is primarily an applied systems/NLP paper; the benchmark and code release could be valuable data/software contributions if the validity issues are resolved. The main risk is internal validity of the evaluation, which I recommend be treated as a gating issue for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read on TransitGPT. It's a solid, well-scoped systems paper: prompt an LLM to write Python against a serialized GTFS feed, execute it in a sandbox, retry on errors, then summarize. The authors ship open-source code and a 100-task benchmark, and the architecture is described in enough detail to reproduce. The head-to-head of GPT-4o and Claude-3.5-Sonnet with and without few-shot and retries is a reasonable evaluation design, and the reported gains (74→90 for GPT-4o, 84→93 for Claude) are plausible. What's genuinely new is the combination of dynamic few-shot prompting, a sandbox execution loop, ReAct-style error correction, and domain-specific fuzzy-matching functions for GTFS stop/route names—plus the benchmark itself, which others can reuse. No fundamental scientific advance, but it doesn't pretend to be one; it's an applied contribution with a working artifact.\n\nThe soft spots are concentrated in the evaluation. First, the benchmark's internal validity is under-documented. The 100 tasks are authored by the same team that designed the prompt and the five custom functions, and the ground-truth code Yi was drafted by an LLM and manually corrected. That creates a real risk that the benchmark measures reproduction of the authors' coding style rather than general GTFS question-answering. The paper never states whether the 12 curated few-shot examples are disjoint from the 100 benchmark tasks. If any overlap, TransitGPT+ would see the reference code for those tasks during evaluation, which would inflate the 90/93 numbers. I don't think this is deliberate, but the authors need to rule it out explicitly. Second, grading is manual with no rubric or inter-rater reliability, and per-category counts are tiny (8–22 tasks), so category-level accuracy is noisy. Confidence intervals or repeated runs would help. Third, the scope is deliberately narrow—GTFS Static only, no routing, no realtime—which is fine, but it should temper how the results are framed.\n\nNone of this is a load-bearing flaw. The central claim that generated code can answer a range of transit queries holds up in broad strokes. The paper deserves a serious referee and a request for revision that addresses the disjointness and grading reliability. I'd cite it if I were working on LLM+GTFS, and I'd bring it to reading group as a good example of an applied LLM agent benchmark and its evaluation pitfalls.\n\nRecommendation: send it to peer review, not desk reject. With the overlap question answered and some reliability analysis added, it's a solid contribution to transit informatics.","headline":"Useful, well-scoped engineering paper with a plausible architecture and benchmark, but the evaluation's internal validity needs tightening before the 90/93 accuracy numbers can be read as robust.","tokens_in":20265,"tokens_out":3487,"would_cite":true,"duration_ms":31329,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By prompting a large language model to write and run Python, TransitGPT answers ordinary-language questions about GTFS transit-schedule data, achieving 90–93% on a 100-task benchmark.","keywords":["GTFS","Large Language Models","Program Synthesis","Prompt Engineering","Public Transit Data","Natural Language Interfaces","Code Generation","TransitGPT"],"falsifier":"Run TransitGPT on a held-out set of 50–100 transit questions written by riders and planners who have never seen the prompt set, with answers checked by an independent transit agency, and compare the task accuracy to the reported 90–93%; a large gap would show the benchmark was tuned from within. A second check: have multiple raters independently grade the original 100 outputs and measure inter-annotator agreement on the binary correct/incorrect scores.","tokens_in":19281,"feed_emoji":"🚌","tokens_out":7267,"duration_ms":64754,"temperature":0.7,"pith_summary":"TransitGPT is a chatbot framework that lets people ask plain-language questions about public transit systems and get answers computed from GTFS schedule data. The key move is to have a large language model (LLM) write Python code that reads a preloaded GTFS feed, run that code in a sandbox, and then summarize the output for the user. No fine-tuning is involved; all guidance comes from prompts, including a schema description, sample rows, custom fuzzy-matching functions, and a few relevant examples selected per query. The authors built a 100-task benchmark across eight categories and report that the full system answers 90% of tasks correctly with one LLM and 93% with another, up from 74–84% in a zero-shot baseline. If that performance holds beyond the benchmark, the contribution is a low-cost recipe for making complex standardized datasets conversational.","feed_headline":"A chatbot answers transit-data questions 90% of the time","feed_subtitle":"TransitGPT turns plain questions into Python code that runs against GTFS schedule feeds—no programming needed.","key_machinery":"The load-bearing mechanism is the code-generation pipeline split into four stages: a lightweight moderation model blocks off-topic or malicious queries; a main LLM receives the user question, conversation history, three TF-IDF-selected few-shot examples, and a five-module system prompt (role, task instructions, GTFS field data types, sample feed rows, and custom fuzzy-matching functions); a sandboxed server executes the generated Python against a preprocessed Feed object; and a summary LLM turns the code's output dictionary into a human-friendly response. The full pipeline is prompt-only, with no fine-tuning and no access to the actual feed during generation.","core_discovery":"The paper's central claim is that a prompted LLM can serve as a reliable natural-language interface to GTFS Static data without seeing the feed and without fine-tuning. The system guides the LLM to output Python code that uses pandas, numpy, and geospatial libraries against a preprocessed Feed object, with strict instructions on GTFS data types, time conventions (seconds since midnight), and how to build a result dictionary with an answer, additional information, and an optional visualization. When the code errors, the traceback is sent back to the LLM for up to three correction rounds; a summary model then rewrites the raw output into a natural answer with stated assumptions. On the authors' 100-task benchmark, the full configuration scored 90% with GPT-4o and 93% with Claude-3.5-Sonnet, versus 74% and 84% without few-shot examples or retries, suggesting that dynamic examples and the error loop are what close most of the gap.","pith_inferences":["The pattern may generalize beyond transit: any domain with many interrelated standardized tables and a well-documented schema—permit records, clinical trial tables, building inventories—could get the same treatment, because the hard part the LLM is doing is schema reasoning, not arithmetic.","Reported accuracy is likely an optimistic bound, since the benchmark's ground-truth answers were drafted by an LLM and corrected by the same team that wrote the prompts; real-world questions phrased by transit riders would be a stricter test and might score lower.","A concrete stress test not in the paper: benchmark performance on a single very large feed, such as one with millions of stop-time rows, where join-heavy queries written by the LLM may time out and push the three-minute execution limit.","The framework deliberately excludes door-to-door routing; a hybrid that hands routing queries to a dedicated router while keeping TransitGPT for system-level questions would cover a larger share of how users actually interact with transit information."],"forward_implications":["Transit riders and planners can run system-level analyses—route lengths, headways, stop counts, fare comparisons—without writing code or mastering the GTFS specification.","Because the feed is never put into the model's context, the architecture sidesteps context-length limits and can work with feeds larger than any LLM window.","The open-source 100-task benchmark, organized into eight categories, gives other teams a shared test set for comparing natural-language interfaces to GTFS data.","The same design should transfer to other standardized mobility specifications, such as shared-bikeshare feeds or real-time transit feeds, by swapping the schema description and Feed object.","Each generated script is a reusable, commented analysis step, so a single chat answer also leaves behind a starting point for deeper exploration."],"supporting_citations":[{"why":"Prior benchmark showing LLMs can answer GTFS semantics questions and write retrieval code; this paper extends that line into a full chatbot.","marker":"Devunuri et al. (2024b)"},{"why":"Source of the dynamic few-shot prompting technique used to select the three closest example query-response pairs.","marker":"Brown et al. (2020)"},{"why":"ReAct framework that inspired the error-handling loop that feeds code errors back to the main LLM for retries.","marker":"Yao et al. (2023)"},{"why":"Toolformer describes describing functions to LLMs so they can call tools; underlies the custom fuzzy-matching functions exposed in the prompt.","marker":"Schick et al. (2023)"},{"why":"Retrieval-augmented generation approach the paper explicitly rejects because GTFS files are too inter-related to chunk.","marker":"Lewis et al. (2021)"},{"why":"Demonstrated that LLMs can generate correct pandas/numpy code, which is the capability TransitGPT relies on.","marker":"Zan et al. (2022)"}],"fun_headline_variants":["Ask transit questions in plain English, AI writes the code","No coding needed: LLM turns questions into transit queries","AI chatbot for GTFS data: ask, get Python, get answer","TransitGPT: your words, its code, real GTFS answers","Chat with your transit data: no SQL, no Python, just ask"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground-truth outputs were built by an LLM and manually corrected by the authors, and all 100 task scores were assigned by manual grading; if those ground truths or grades are systematically skewed toward the same prompt style TransitGPT uses, the 90% and 93% accuracy figures may not reproduce on independent real-world questions.","fun_headline_variants_meta":{"raw":{"variants":["Ask transit questions in plain English, AI writes the code","No coding needed: LLM turns questions into transit queries","AI chatbot for GTFS data: ask, get Python, get answer","TransitGPT: your words, its code, real GTFS answers","Chat with your transit data: no SQL, no Python, just ask"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2157,"prompt_tokens":912,"completion_tokens":1245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1155}},"tokens_in":528,"tokens_out":1245,"duration_ms":9091,"temperature":1.0,"reasoning_tokens":1155,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:40:12.592431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TransitGPT on a held-out set of 50–100 transit questions written by riders and planners who have never seen the prompt set, with answers checked by an independent transit agency, and compare the task accuracy to the reported 90–93%; a large gap would show the benchmark was tuned from within. A second check: have multiple raters independently grade the original 100 outputs and measure inter-annotator agreement on the binary correct/incorrect scores.","supporting_citations":[],"review_version":1}