{"id":"570aea05-b39e-45ee-b2ff-9705b377db11","arxiv_id":"2501.03904","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a new San Antonio GTFS benchmark, GPT-4 and GPT-3.5-turbo score between 48% and 98% accuracy on understanding tasks and 60% to 90% on information retrieval, with errors concentrated in ambiguous and multi-file questions.","lead":"This study tested how two ChatGPT models answer hundreds of questions about transit data formats and schedule files, using San Antonio's bus system. The measured accuracies, ranging from 48% to 98%, show the models are useful but not reliable enough to run a transit system without careful design and tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hand-written answer key is not demonstrably unambiguous: the paper's own Table 2 includes a GTFS question with two valid answers, so the reported accuracy ranges may not be trustworthy.","rationale":"The paper's central claim is an empirical accuracy range over a newly constructed benchmark. The entire evaluation reduces to matching model outputs against a single hand-authored key, so the most load-bearing condition is that the key is correct and unambiguous. The paper itself provides a concrete counterexample in Table 2: the shape_dist_travelled question has two plausible correct answers under the official GTFS reference, and the field is misspelled relative to the spec. This is not a disagreement with consensus; it is an internal ambiguity in the evaluation instrument. If the key contains even a small fraction of such items, the reported category-level accuracies shift, and the comparison between GPT-3.5-turbo and GPT-4 could invert for some categories. The reader identified the same weak assumption, and I agree. The unsupported 8% prompt-engineering claim in Section 7.3 and the discrepancy between the stated 3275 MCQs and the 3195 that sum from the reported experiment sizes are additional reporting problems, but they are secondary to the validity of the labels. Because the authors' own sample demonstrates the concern, and no released artifacts allow external re-checking, the appropriate posture remains conditional: require release of the key and an independent audit before the numbers are relied upon. The reader's CONDITIONAL verdict is therefore unchanged.","tokens_in":9666,"tokens_out":4501,"duration_ms":41958,"concrete_test":"Obtain the authors' full question bank and answer key (via release or appendix), then have two independent GTFS-specification experts mark, for each of the 195 core MCQs, all options that are valid under the current GTFS reference and the VIA feed snapshot used. Compute the fraction of questions where the official key disagrees with the expert-labeled set or where more than one option is valid. If that fraction is non-negligible (e.g., >2%), recompute Table 3 and the Section 8 ranges with corrected keys and report per-question flags. For Experiment V, have a second annotator re-judge a random 20-question subsample under a written rubric and report Cohen's kappa; if kappa < 0.8, the retrieval accuracies are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claims (Section 8: 47.97–98.44% understanding, 60.53–90.48% retrieval) are computed as exact-match or semantic-match against a hand-written answer key (Sections 5.1, 5.3). That key must be correct and unambiguous. Table 2 shows this assumption does not hold even in the released sample: the question 'In which file does the shape_dist_travelled attribute appear in GTFS?' lists stops.txt, shapes.txt, trips.txt, and stop_times.txt as single-choice options, but the official GTFS reference defines shape_dist_traveled (the standard spelling; the paper uses a non-standard double-l variant) as an optional field in both shapes.txt and stop_times.txt. A forced single answer is therefore not uniquely determined; the key's selected answer is one of at least two defensible answers. If this ambiguity pattern extends to the other 194 core questions, every accuracy number in Table 3 is computed against a contestable label, and the comparison between GPT-3.5-turbo and GPT-4 loses meaning for affected categories. In addition, Experiment V accuracies rest on 'semantic equivalence' judgments with no rubric, no second annotator, and no reported inter-annotator agreement, which can systematically inflate retrieval accuracy. The manuscript provides no released question bank, answer key, or VIA feed snapshot, so an independent check of the key is currently impossible. This is an internal validation concern, not a disagreement with LLM consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a case study evaluating two OpenAI chat models (GPT-3.5-turbo and GPT-4/GPT-4o) on two tasks: understanding GTFS semantics and retrieving information from a reduced San Antonio VIA GTFS feed. The authors construct 195/780/444/1776 multiple-choice questions for Experiments I-IV plus 80 short-answer retrieval questions for Experiment V, score by exact match and semantic equivalence, and report accuracy ranges of 47.97-98.44% for understanding and 60.53-90.48% for retrieval. They conclude that LLMs are promising for transit applications but require prompt engineering and fine-tuning. The paper also contains a qualitative discussion of LLM inconsistency and prompt engineering.","tokens_in":9951,"tokens_out":8986,"duration_ms":77400,"significance":"The task design, separating pre-trained 'understanding' from context-provided 'information retrieval,' is a useful framing, and using a real GTFS feed as the retrieval corpus is a strength. If the reported numbers were reliable, the paper would provide a practical benchmark for transit agencies considering LLM-based passenger information systems. However, the paper provides no code, data, or answer key; the ground-truth key is shown to contain at least one ambiguous item; and the single-run evaluation with no confidence intervals or significance tests does not support the comparative claims. The contribution is therefore preliminary rather than conclusive, with modest novelty beyond the prior GTFS benchmark it follows ([31]).","major_comments":[{"comment":"The sample Attribute Mapping question in Table 2, 'In which file does the shape_dist_travelled attribute appear in GTFS?', is not uniquely answerable: the official GTFS Schedule reference defines shape_dist_traveled as an optional field in both shapes.txt and stop_times.txt (using a single-'l' spelling), so both answer (b) and answer (d) are defensible. Since all Experiment I-IV accuracies in Table 3 are exact matches against a hand-written answer key that is not released in the paper, a single ambiguous item in a small category can move the reported percentages by several points (e.g., one question is 7.1% of the 14-question Term Definition category). The authors should release the full question bank, answer key, and GTFS snapshot, or provide a second-annotator agreement study showing that every question has a unique correct answer.","section":"Section 5.1, Table 2"},{"comment":"Section 5.3 says that Experiment V accuracy is based on 'semantic equivalence' between the model output and the ground-truth answer, but no scoring rubric, no second annotator, and no inter-annotator agreement are reported. The retrieval questions are free-form and the paper itself documents substantial LLM output variability in Section 7.2, so the reported retrieval accuracies (60.53-90.48%) and the claimed GPT-4o advantage ('average performance increase of approximately 15%' in Section 6.2) are not verifiable as stated. A defined scoring protocol or released scored transcripts is needed before these numbers can be interpreted.","section":"Section 5.3, Section 6.2"},{"comment":"All accuracy numbers in Table 3 and Figure 1 appear to come from a single run per model, with no confidence intervals, repeated runs, or significance tests. Some categories contain very few items, and the paper itself acknowledges in Section 6.1.1 that the 14-question Term Definition category is too small for reliable assessment. Nevertheless, Sections 6.1.2 and 6.2 draw comparative conclusions such as GPT-4o being more robust and 'consistently outperforming' GPT-3.5-turbo. Given the sampling variability at these sample sizes and the model nondeterminism the authors describe in Section 7.2, the observed differences are not established; repeated runs or statistical significance testing are required to support the central comparison.","section":"Section 6, Table 3, Figure 1"}],"minor_comments":[{"comment":"The paper states that the five experiments comprise 3275 multiple-choice questions, but the counts given in Section 5.1 sum to 3195 (195 + 780 + 444 + 1776 = 3195). The number 3275 is repeated in Section 7.1 and should be corrected.","section":"Section 5"},{"comment":"The manuscript uses 'GPT-4' in the text and '4o' in Table 3 and Figure 1 without stating whether the same model version and API endpoint were used across experiments. Please clarify the exact model identifiers and configuration (temperature, max tokens) for each run.","section":"Sections 5.1, 6.2, Table 3, Figure 1"},{"comment":"The claim that 'performance of our experiments may improve by up to ≈ 8% through optimized prompting' is unsupported: no prompt-engineering experiment is reported, and the cited references [37, 38] are general introductions rather than evidence for this specific figure. Remove the quantitative claim or provide the supporting experiment.","section":"Section 7.3"},{"comment":"The assertion that findings from a three-route, 34-trip, 60-stop subset of one agency's feed 'can be applied to other cities' overgeneralizes; the authors should temper this claim given the severely reduced retrieval dataset.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful benchmark idea but is, in its current form, a workshop-level report. The decisive issue is the ambiguous ground-truth key, which undermines the central accuracy claims; because the authors did not release the question bank, the reviewer cannot determine whether the ambiguity is isolated to the Table 2 sample or systemic. A revised version that releases materials, adds a second annotator for the answer key and semantic-equivalence scoring, and reports statistical uncertainty would address the main concerns. The unsupported 8% prompt-engineering claim and the 3275/3195 arithmetic error can be fixed locally."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent, modest extension of Devunuri et al.'s GTFS benchmark to San Antonio, with a useful 'none of these' augmentation and a three-route retrieval set. The qualitative conclusion—general LLMs can answer many GTFS questions but degrade on ambiguous and multi-step retrieval—is plausible and matches what we'd expect. But the headline accuracy numbers are computed against a hand-written answer key, and the key is not unambiguous even in the ten released samples. Table 2's shape_dist_travelled question lists shapes.txt and stop_times.txt as separate single-choice options, while the GTFS reference defines that field (standard spelling shape_dist_traveled) in both files. So the \"correct\" answer is contestable, and if this kind of ambiguity spreads across the 195 core MCQ set, the 47.97–98.44% range is not trustworthy.\n\nWhat the paper does well: the benchmark is externally grounded—GTFS Schedule docs and a real VIA feed snapshot, not the models' outputs. The design cleanly separates understanding from retrieval, and the retrieval questions span simple to complex joins. Credit also for building on [31] rather than ignoring it.\n\nThe soft spots beyond the key: no released question bank, answer key, or feed snapshot, so nothing can be independently checked. No confidence intervals or repeated runs; several categories have 14–30 questions, so single-digit differences are noise. The unsupported \"up to ≈8% via prompt engineering\" in Section 7.3 is a bare assertion. Experiment V's \"semantic equivalence\" has no rubric or second annotator, which can inflate retrieval accuracy. Also, Section 5 says 3275 MCQs but the four MCQ experiments sum to 3195; the 3275 is only correct if you include the 80 retrieval questions, so the labeling is loose.\n\nNone of these kill the paper's qualitative message, but they do mean the reported numbers should be treated as indicative, not as a solid benchmark. If the authors fix the answer key (and ideally release it), add basic error bars, and either drop or substantiate the prompt claim, this becomes a useful city-specific data point.\n\nWho it's for: people working on LLM evaluation for domain-specific QA or transit informatics. It deserves a serious referee—probably a workshop-level one—because the design is honest and the flaws are fixable. I'd recommend conditional acceptance with major revision.","headline":"A competent, modest GTFS benchmark extension whose headline accuracy numbers rest on a hand-written answer key that already shows one ambiguous question in the paper's own sample.","tokens_in":10459,"tokens_out":2351,"would_cite":false,"duration_ms":21060,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that ChatGPT models score 47.97% to 98.44% on GTFS transit understanding questions and 60.53% to 90.48% on GTFS retrieval, concluding that fine-tuning is necessary before real-world deployment.","keywords":["large language models","GTFS","public transportation","ChatGPT","question answering","information retrieval","San Antonio VIA","benchmark"],"falsifier":"Recruit two independent GTFS experts to write the answer key for the same 195 core multiple-choice questions and 80 retrieval questions from the same VIA feed snapshot and GTFS Schedule documentation, without seeing the authors' key, and compare agreement; if the experts frequently disagree with the paper's key, every reported accuracy figure is called into question. A second, cheaper check is to rerun each zero-shot prompt dozens of times at the same temperature and look at the spread in accuracy, which would test whether the reported single-run numbers are stable.","tokens_in":9457,"feed_emoji":"🚌","tokens_out":6628,"duration_ms":59142,"temperature":0.7,"pith_summary":"This paper asks whether off-the-shelf ChatGPT models can understand and work with public transit data, using San Antonio's VIA system and the standard GTFS feed format as the testbed. It reports that GPT-3.5-turbo and GPT-4/4o score between 47.97% and 98.44% accuracy on understanding questions built from the GTFS specification, and between 60.53% and 90.48% on information-retrieval questions answered from a reduced VIA feed. The authors conclude that the models already know a good deal of transit semantics and can look up simple facts, but that multi-step retrieval, ambiguous 'none of these' variants, and semantically similar categories drag accuracy down, so careful engineering and fine-tuning are needed before real deployment.","feed_headline":"ChatGPT scores 48–98% on GTFS transit quiz","feed_subtitle":"But accuracy falls to 64% on multi-step transit queries, so cities should fine-tune before deploying.","key_machinery":"The argument is carried by a two-task benchmarking design on a public data standard. The understanding task uses 195 multiple-choice questions (later expanded to 444, and augmented into 1,776 variants by swapping in a 'none of these' option) authored from the official GTFS Schedule documentation, scored by exact match. The retrieval task feeds the model slices of the San Antonio VIA GTFS feed — three routes, 34 trips, 60 stops — and asks 80 short-answer questions that require search, filtering, sorting, grouping, and joining across files, scored by semantic equivalence. This setup lets the authors separate what the model already knows from what it can extract from supplied data, and the accuracy gap between the two tasks provides the evidence for the conclusion that both data coverage and architecture matter.","core_discovery":"The central finding is a measured capability profile, not a single number: pre-trained ChatGPT answers straightforward GTFS questions well (best 98.44% on attribute mapping), but performance drops to roughly chance on categorical mapping, and adding a 'none of these' option costs about ten points of accuracy on average, with GPT-4 more robust than GPT-3.5. On retrieval, the models handle simple lookups in one or two GTFS files well (best 90.48%) but average only about 64% when a question requires joining data across several files such as stops.txt, stop_times.txt, trips.txt, and routes.txt. The paper takes this pattern as evidence that LLM failures in this domain are partly missing pre-training coverage and partly reasoning limits, and that real transit deployments should not assume a general-purpose model is reliable without fine-tuning and prompt engineering.","pith_inferences":["An implication the authors leave implicit is that the same 80-question retrieval set could be run against other GTFS feeds, not just San Antonio's, to test whether these accuracy ranges generalize across cities without retraining.","Because the paper reports single-run zero-shot results, its accuracy ranges are likely upper bounds for deployment; a natural extension is to sample each prompt repeatedly and report variance, which would quantify the inconsistency the authors discuss in Section 7.2.","If Categorical Mapping's near-chance performance is really caused by semantic similarity, then fine-tuning on GTFS-specific categories or using a retrieval-augmented taxonomy should move that category well above 50%; that prediction is testable and follows directly from the paper's diagnosis.","The authors' estimate that prompt engineering can add about 8% accuracy suggests a cheaper intervention than full fine-tuning, and measuring whether optimized prompts close most of the gap on complex retrieval would directly inform transit agencies' deployment choices."],"forward_implications":["Direct users of a general-purpose ChatGPT should expect high accuracy on simple GTFS definitions but near-chance performance on categorical distinctions like Rail versus Light Rail.","A transit agency that hands an LLM a GTFS feed can rely on it for single-file lookups but should verify multi-file join answers, where average accuracy falls to about 64%.","Because adding a 'none of these' option cuts accuracy by around 10 points, benchmark designers and agencies should treat forced-choice multiple choice as systematically easier than open-ended questions.","GPT-4's smaller performance drop on augmented questions and its consistent edge in retrieval suggest that newer, larger models are the safer default for transit experiments, though still not deployment-ready."],"supporting_citations":[{"why":"supplies the benchmark design the paper follows: ChatGPT evaluated on GTFS semantics and retrieval, including the MCQ and retrieval question formats.","marker":"[31]"},{"why":"documents GPT-4, the model whose understanding and retrieval performance is measured.","marker":"[11]"},{"why":"identifies ChatGPT as the representative LLM used for the experiments.","marker":"[10]"},{"why":"gives the Transformer architecture that underlies the ChatGPT models being tested.","marker":"[33]"},{"why":"describes the GPT scaling approach and zero-shot capabilities that motivate testing pre-trained models as-is.","marker":"[36]"},{"why":"argues that LLMs could reshape intelligent transportation, the broader claim this case study tests.","marker":"[32]"}],"fun_headline_variants":["LLMs: 98% on simple transit queries, 64% on multi-step","ChatGPT transit quiz: 98% simple, 64% multi-file","San Antonio: LLMs need fine-tuning for transit reliability","GPT-4 outperforms GPT-3.5 on GTFS queries, study finds","Pre-trained LLMs struggle with multi-file transit queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-written answer key is correct and unambiguous for all 195 core multiple-choice and 80 retrieval questions, so that every accuracy number in the paper is a true measure of the model's knowledge rather than of quirks in the key.","fun_headline_variants_meta":{"raw":{"variants":["LLMs: 98% on simple transit queries, 64% on multi-step","ChatGPT transit quiz: 98% simple, 64% multi-file","San Antonio: LLMs need fine-tuning for transit reliability","GPT-4 outperforms GPT-3.5 on GTFS queries, study finds","Pre-trained LLMs struggle with multi-file transit queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000885,"raw_usage":{"total_tokens":3797,"prompt_tokens":894,"completion_tokens":2903,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2807}},"tokens_in":510,"tokens_out":2903,"duration_ms":21194,"temperature":1.0,"reasoning_tokens":2807,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:43:56.877564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recruit two independent GTFS experts to write the answer key for the same 195 core multiple-choice questions and 80 retrieval questions from the same VIA feed snapshot and GTFS Schedule documentation, without seeing the authors' key, and compare agreement; if the experts frequently disagree with the paper's key, every reported accuracy figure is called into question. A second, cheaper check is to rerun each zero-shot prompt dozens of times at the same temperature and look at the spread in accuracy, which would test whether the reported single-run numbers are stable.","supporting_citations":[{"cited_title":"Chatgpt for gtfs: benchmarking llms on gtfs semantics and retrieval,","cited_arxiv_id":null,"evidence_quote":"supplies the benchmark design the paper follows: ChatGPT evaluated on GTFS semantics and retrieval, including the MCQ and retrieval question formats."},{"cited_title":"Introducing chatgpt,","cited_arxiv_id":null,"evidence_quote":"identifies ChatGPT as the representative LLM used for the experiments."}],"review_version":1}