Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Application and Evaluation of Large Language Models for Forecasting the Impact of Traffic Incidents

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pretrained LLM given 24 curated examples matches random forest, trained on 2,221 incidents, at forecasting traffic incident impact — with no task-specific training.

desk verdict Solid empirical benchmark with a new ICL example-selection method; the headline claim is credible only if the k and feature-set choices were made on the validation split, which the paper never states. read the letter →

arxiv 2507.04803 v1 pith:4JEHH4K7 submitted 2025-07-07 cs.AI

George Jagadeesh , Srikrishna Iyer , Michal Polanowski , Kai Xin Thia This is my paper · ORCID
classification cs.AI
keywords trafficincidentimpactpredictionlargelanguagemodelsin-contextlearningnear-boundaryexampleselectionseverityclassificationmacro-averageF1scorefree-textlogsPEMS-BAYdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large language model, used off the shelf, can replace a conventionally trained machine-learning model for one practical forecasting task: predicting whether a just-reported traffic incident will have a mild, moderate, or severe effect on traffic 15 or 30 minutes later. On a test set of 556 real Bay Area incidents, the best LLM (GPT-4.1) matches random forest exactly at the 15-minute horizon (macro-F1 0.59 for both), and the best 30-minute LLM (Claude 3.7 Sonnet, 0.48) nearly matches random forest (0.49). The LLMs saw only 24 examples in their prompts, whereas the ML models were trained on 2,221 labeled incidents. The ingredient the authors identify as decisive is a curated method of choosing those 24 examples from near the class boundaries of the labeled set, which they show consistently beats random example selection. If the claim holds, incident impact prediction becomes deployable in regions that lack the thousands of labeled incidents that conventional models require, and free-text dispatcher logs can be used directly instead of being discarded.

What carries the argument

The load-bearing mechanism is the paper's method for selecting in-context learning examples, which it introduces because random example selection is known to give poor in-context learning accuracy. The method first drops outliers — incidents closer, in the normalized feature space, to the centroid of another class than to their own class. From the remaining incidents of each class it keeps the 50% closest to the neighboring class boundary, then draws $m/3$ examples per class. Thirty candidate prompts of $m$ examples are scored on a 60-incident validation set, and the $k$ best prompts are merged into the final system prompt, here $m=12$ and $k=2$, giving 24 examples (8 per class). The second mechanism is the impact label itself: the speed-decrease ratio $\Delta v(i,t)$ computed from sensor speeds on the 2-mile upstream stretch relative to the pre-incident speed, thresholded at 0.2 and 0.5 to define mild, moderate, and severe. Together the two mechanisms convert a prediction problem that normally needs thousands of labels into a prompt-construction problem.

What would settle it

Re-run the pipeline with the top-5 feature choice and the 24-example setting frozen before any test-set label is observed — decide them strictly on the 60-incident validation set — then measure GPT-4.1's macro-F1 on the 556 held-out incidents at the 15-minute horizon. If it falls below 0.59, or below random forest's score on the same split, the claimed parity was an artifact of test-set-informed model selection. An independent replication on a different region, re-selecting examples on that region's own training pool, would additionally test whether the 24-example method transfers.

Watch

Extended reading notes

Core claim

The paper's central claim is that a fully LLM-based pipeline predicts incident impact as accurately as the best conventional machine-learning baseline, despite never being trained or fine-tuned on the task. The pipeline has two stages: a small LLM (GPT-4o mini) reads the California Highway Patrol text logs generated up to the prediction time and extracts incident features, which are combined with two traffic-speed features (pre-incident relative speed and the speed-decrease ratio at the prediction time) into a natural-language prompt addressed to a frontier LLM; the prompt's system message defines the three impact classes and embeds 24 worked examples. Impact itself is defined empirically: the average speed-decrease ratio over a 2-mile upstream stretch, relative to the speed that prevailed before the incident, bucketed into mild ($\Delta v \leq 0.2$), moderate ($0.2 < \Delta v \leq 0.5$), and severe ($\Delta v > 0.5$). The headline numbers are a macro-F1 of 0.59 for GPT-4.1 at 15 minutes, identical to random forest's 0.59, and 0.48 for Claude 3.7 Sonnet at 30 minutes against random forest's 0.49, with XGBoost at 0.55 and 0.49. The paper further claims that its near-boundary example-selection method gives every tested LLM a substantial, consistent accuracy gain over randomly selected examples, and that a reduced five-feature set outperforms the full feature set for the ML baselines.

Load-bearing premise

The load-bearing premise is that the two empirical design choices — cutting the feature set to the top 5 (Table II) and fixing the prompt at 24 examples (Table IV) — were made on the validation set alone, so that the 556-incident test set stayed genuinely unseen; the paper never states which split those tables used, and if test-set outcomes leaked into those choices, every reported macro-F1 score is optimistically biased.

Editorial extensions

If this is right

  • A traffic agency could deploy impact prediction in a new region from a small pool of labeled incidents — 24 curated examples and a 60-incident validation set — instead of the roughly 2,000-plus labeled incidents the machine-learning baselines required.
  • Unstructured incident logs become usable as-is: the pipeline turns California Highway Patrol dispatcher messages into features, so the free-text record that conventional models discard becomes an input.
  • Prompt curation is a first-order lever: the proposed near-boundary selection beats random selection for all three LLMs at both horizons, so how the examples are chosen matters as much as which LLM is used.
  • Parity with the best machine-learning model is exact at the 15-minute horizon (GPT-4.1 and random forest both at macro-F1 0.59) but only approximate at 30 minutes (Claude 3.7 Sonnet 0.48 vs random forest 0.49).
  • The five useful features — incident time, vehicles involved, lanes blocked, pre-incident relative speed, and the initial speed-decrease ratio — carry essentially all the predictive signal; dozens of other LLM-extracted features add nothing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Generalizing the recipe: the same two-stage design — a small model extracts features from free text, a frontier model classifies with curated near-boundary examples — is directly testable on other event-impact problems with scarce labels and rich text logs, such as emergency-response incidents, power outages, or IT service outages.
  • If parity survives a strictly held-out test, the economic decision between LLM and gradient-boosted deployment stops being about accuracy and becomes about API cost per prediction versus the cost of collecting thousands of labels.
  • The finding that the many other LLM-extracted features (weather, tow truck, ambulance, truck involvement) add no accuracy is ambiguous: either those events genuinely carry little signal beyond lanes and vehicles, or extraction and alignment noise cancels their value. A human-verified feature set would separate the two explanations, and if the second holds, better extraction could push the LLM past
  • A cheap transfer test the authors did not run: reuse the identical 24-example prompt and class definitions on another city's incident logs with no re-selection, to measure how far the prompt's knowledge generalizes across regions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a fully LLM-based pipeline for predicting whether a just-occurred traffic incident will have mild, moderate, or severe impact on upstream traffic at 15 and 30 minutes after the incident. It uses GPT-4o mini to extract structured features from California Highway Patrol incident logs, combines them with two speed-based traffic features into a natural-language prompt, and uses in-context learning with examples selected by a near-boundary sampling heuristic. Three LLMs (Claude 3.7 Sonnet, Gemini 2.0 Flash, GPT-4.1) are compared with Random Forest and XGBoost on 556 held-out incidents. The central claim is that the best LLM matches the best machine-learning model, with macro-F1 scores of 0.59 versus 0.59 at 15 minutes and 0.48 versus 0.49 at 30 minutes, and that the proposed ICL example selection method is better than random selection. The paper is an empirical benchmark rather than a theoretical contribution.

Significance. If the headline comparison holds under proper split discipline, the paper provides a practically valuable datapoint: a pre-trained LLM using only 24 in-context examples can match a Random Forest trained on roughly 2,200 labeled incidents, while also consuming free-text incident logs directly. The problem formulation is clear, the data sources are public and real, the baselines are standard, and the paper transparently prioritizes macro-F1 over weighted-F1 for an imbalanced three-class problem. The proposed near-boundary ICL example selection is simple and is tested against a random-sampling baseline, which is a useful methodological contribution. The main weaknesses are experimental-hygiene issues: the splits used for feature selection and for choosing the number of examples are not stated, and the headline equality is asserted from point estimates without uncertainty quantification. These concerns are fixable and do not undermine the value of the study if resolved.

major comments (3)
  1. [Section V.C, Table II] The split used to compute the two feature-set accuracies in Table II is not stated. The text says only that the authors examine the prediction accuracies of two sets of features and then concludes that the top 5 features are used for the rest of the experiments. Because this feature choice carries into all Table III results, Table II must be computed on the training or validation partition. If it was computed on the 556-incident test set, the feature set is selected using test labels, and the reported macro-F1 scores are optimistically biased. Please state the split explicitly and, if necessary, redo the feature selection without consulting the test set.
  2. [Section V.C, Table IV] The analysis that fixes k=2 is not tied to a declared split. Section V.B describes a 60-incident validation set drawn from the training set, but the paragraph introducing Table IV does not say whether the macro-F1 values in that table were obtained on this validation set or on the test set. The choice k=2 determines the final prompt and is then fixed for the Table III comparison. For GPT-4.1 at the 15-minute horizon, Table IV shows macro-F1 rising from 0.46 with 0 examples to 0.59 with 24 examples; if these numbers were computed on the test set, the prompt is effectively tuned on the same data that later supports the headline match. The manuscript must either state clearly that Table IV is validation-only or repeat the Table III evaluation under a k chosen without test labels.
  3. [Section V.C, Table III] The central finding that the best LLM matches the best machine-learning model is based on point estimates only. The text reports LLM results as averages of 3 runs but gives no standard deviations, confidence intervals, or significance tests. This is load-bearing because the severe class contains only about 12 of the 556 test incidents (2.1% in Table I), so macro-F1 is strongly influenced by small-class counts, and the 0.48 versus 0.49 difference at 30 minutes is plausibly within sampling noise. Please report per-class F1 scores, per-run values, and bootstrap or other confidence intervals, and avoid claiming equivalence without showing that the intervals overlap or that an equivalence test was used.
minor comments (5)
  1. [Section III.A] The moderate impact class is defined as 0.2 > ∆v(i,t) ≤ 0.5, which is unsatisfiable; it should be 0.2 < ∆v(i,t) ≤ 0.5. The mild and severe definitions are otherwise consistent with the description.
  2. [Table I] Because the severe class is only 2.1% to 2.4% of the test set, the table should also include absolute counts. With 556 test incidents, the severe class contains roughly 12 incidents, which is important context for interpreting macro-F1 differences.
  3. [Figure 5] The claim that the proposed example selection method gives a substantial and consistent improvement over random sampling is not supported by numeric values, error bars, or statistical tests in the text. Please include a table with the underlying macro-F1 values for each model and horizon.
  4. [Section V.B] Reporting only that the temperature parameter is set to 0 is not enough for reproducibility. Please provide the exact API model snapshots and evaluation dates, and note that temperature 0 does not guarantee identical outputs across API runs.
  5. [Section IV.C] The sampling strategy refers to distances in the normalized feature space, but the normalization method and distance metric are not specified. Please define the feature normalization and the distance used for centroid and near-boundary calculations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical benchmark with held-out evaluation; the only self-citation is non-load-bearing background; the split ambiguity is a leakage concern, not a derivation-level reduction.

full rationale

The paper is an empirical benchmark study with no derivation chain that reduces to its inputs. The sole self-citation [3] is background on delay prediction and is not used to justify any load-bearing premise or to forbid alternatives. The central comparison in Table III is evaluated on a held-out 556-incident test set against machine-learning models trained on the disjoint 2221-incident training split. In-context examples are selected from the training set, and m=12 and k=2 are set after validation-based analysis using the 60-incident validation set described in Section V.B. The ambiguity about which split produced Table II and Table IV is a possible data-leakage or correctness risk, not circularity: even if test labels influenced the choice of feature set or the number of in-context examples, the reported macro-F1 scores would be optimistically biased, but the comparison would still be an empirical measurement rather than an equation-level equivalence between output and input. No fitted parameter is renamed as a prediction, no known result is repackaged under new coordinates, and no uniqueness claim is imported from the authors' prior work. The paper is self-contained against external benchmarks and its findings stand or fall on experimental validity, not on circular reasoning.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new entities; the model relies on standard LLM ICL with hand-designed sampling heuristics and fitted prompt hyperparameters (m, k).

free parameters (3)
  • m (examples per prompt set) = 12
    Set empirically; varying k with m=12 produced the chosen 24-example prompt.
  • k (number of prompt sets combined) = 2
    Chosen from Table IV to balance macro-F1 vs prompt length; 24 examples final.
  • Top-5 feature set = time, #vehicles, #lanes blocked, pre-incident relative speed, initial speed decrease ratio
    Selected based on random forest feature importance and marginal gains in Table II; selection split not clearly stated.
assumptions (3)
  • domain assumption Ground-truth impact is defined by speed decrease ratio thresholds (0.2, 0.5) against historical averages.
    This operationalization is cited to prior empirical methods and is a modeling choice, not a derived law.
  • domain assumption The initial speed decrease ratio measured at the 5-min prediction time is assumed to be a reliable predictor of impact at 15/30 min.
    The feature importance analysis confirms this, but the task consequently becomes partly a nowcast.
  • ad hoc to paper Outlier exclusion and near-boundary selection (50% nearest to neighbor class) improve ICL.
    Introduced as the method; validated only by the comparison in Figure 5, with limited detail on distance metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Application and Evaluation of Large Language Models for Forecasting the Impact of Traffic Incidents." pith.science (2026). https://pith.science/paper/4JEHH4K7

@misc{pith2026250704803,
  author       = {Pith},
  title        = {Pith review of: Application and Evaluation of Large Language Models for Forecasting the Impact of Traffic Incidents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4JEHH4K7}},
  note         = {Machine review of arXiv:2507.04803}
}
read the original abstract

This study examines the feasibility of applying large language models (LLMs) for forecasting the impact of traffic incidents on the traffic flow. The use of LLMs for this task has several advantages over existing machine learning-based solutions such as not requiring a large training dataset and the ability to utilize free-text incident logs. We propose a fully LLM-based solution that predicts the incident impact using a combination of traffic features and LLM-extracted incident features. A key ingredient of this solution is an effective method of selecting examples for the LLM's in-context learning. We evaluate the performance of three advanced LLMs and two state-of-the-art machine learning models on a real traffic incident dataset. The results show that the best-performing LLM matches the accuracy of the most accurate machine learning model, despite the former not having been trained on this prediction task. The findings indicate that LLMs are a practically viable option for traffic incident impact prediction.

Figures

Figures reproduced from arXiv: 2507.04803 by the authors.

Figure 1
Figure 1. shows the proposed LLM-based solution for traffic incident impact prediction. The main aspects of the solution are discussed below [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. provides an overview of the proposed method. It involves selecting a small number 𝑚 of examples from the labelled set of incidents according to a judicious sampling strategy (described below) to construct a prompt and evaluating the prompt on a validation set using an LLM. The validation set is a small subset of the labelled set. 30 different prompts each with 𝑚 examples are evaluated against the validation set. The… view at source ↗
Figure 4
Figure 4. shows a summary of the impact predictions made by GPT 4.1, in the form of a confusion matrix, for the 15- minute horizon in one of the experiment runs. We find that, in general, a relatively large number of mild-impact incidents are misclassified as moderate resulting in low precision and therefore a low F1 score for the moderate class, which brings down the overall macro-average F1 score. We also observe that, in g… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Comparison of the proposed example selection method and random sampling [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages

  1. [1]

    Overview of traffic incident duration analysis and prediction,

    R. Li, F. C. Pereira, and M. E. Ben-Akiva, “Overview of traffic incident duration analysis and prediction,” European Transport Research Review, vol. 10, no. 2, pp. 1–13, 2018

    work page 2018

  2. [2]

    DG-Trans: Dual-level Graph Transformer for Spatiotemporal Incident Impact Prediction on Traffic Networks

    Y. Sun, K. Fu, and C.-T. Lu, “DG-Trans: Dual-level Graph Transformer for Spatiotemporal Incident Impact Prediction on Traffic Networks,” arXiv preprint arXiv:2303.12238, 2023

    work page Pith review arXiv 2023

  3. [3]

    Predicting the Impact of Traffic Incidents: An Evaluative Analysis,

    G. R. Jagadeesh, J. K. Ong, and C. C. Chong, “Predicting the Impact of Traffic Incidents: An Evaluative Analysis,” in Emerging Cutting-Edge Developments in Intelligent Traffic and Transportation Systems , IOS Press, 2024, pp. 231–242

    work page 2024

  4. [4]

    Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions,

    D. Mahmud et al., “Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions,” IEEE Transactions on Intelligent Transportation Systems (Early Access), 2025

    work page 2025

  5. [5]

    Arterial incident duration prediction using a bi-level framework of extreme gradient-tree boosting

    A.-S. Mihaita, Z. Liu, C. Cai, and M. -A. Rizoiu, “Arterial incident duration prediction using a bi-level framework of extreme gradient-tree boosting,” arXiv preprint arXiv:1905.12254, 2019

    work page Pith review arXiv 1905

  6. [6]

    A Data -Driven Approach to Estimate and Predict the Traffic Incidents’ Queue Length,

    B. Ghosh and J. Dauwels, “A Data -Driven Approach to Estimate and Predict the Traffic Incidents’ Queue Length,” presented at the 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2023, pp. 2517–2522

    work page 2023

  7. [7]

    Text analysis in incident duration prediction,

    F. C. Pereira, F. Rodrigues, and M. Ben -Akiva, “Text analysis in incident duration prediction,” Transportation Research Part C: Emerging Technologies, vol. 37, pp. 177–192, 2013

    work page 2013

  8. [8]

    Mining traffic incidents to forecast impact,

    M. Miller and C. Gupta, “Mining traffic incidents to forecast impact,” in the Proceedings of the ACM SIGKDD International Workshop on Urban Computing, 2012, pp. 33–40

    work page 2012

Show all 18 references
  1. [9]

    Traffic accident duration prediction using text mining and ensemble learning on expressways,

    J. Chen and W. Tao, “Traffic accident duration prediction using text mining and ensemble learning on expressways,” Scientific Reports, vol. 12, no. 1, p. 21478, 2022

    2022

  2. [10]

    Traffic incident duration prediction using BERT representation of text,

    P. Agrawal, A. Franklin, D. Pawar, and S. PK, “Traffic incident duration prediction using BERT representation of text,” presented at the 2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall), IEEE, 2021, pp. 1–5

    2021

  3. [11]

    Enhancing Traffic Incident Management with Large Language Models: A Hybrid Machine Learning Approach for Severity Classification,

    A. Grigorev, K. Saleh, Y. Ou, and A. -S. Mihăiţă, “Enhancing Traffic Incident Management with Large Language Models: A Hybrid Machine Learning Approach for Severity Classification,” International Journal of Intelligent Transportation Systems Research, vol. 23, no. 1, pp. 259–280, 2025

    2025

  4. [12]

    A methodological approach for estimating temporal and spatial extent of delays caused by freeway accidents,

    Y. Chung and W. W. Recker, “A methodological approach for estimating temporal and spatial extent of delays caused by freeway accidents,” IEEE Transactions on Intelligent Transportation Systems , vol. 13, no. 3, pp. 1454–1461, 2012

    2012

  5. [13]

    Forecasting spatiotemporal impact of traffic incidents on road networks,

    B. Pan, U. Demiryurek, C. Shahabi, and C. Gupta, “Forecasting spatiotemporal impact of traffic incidents on road networks,” presented at the 2013 IEEE 13th International Conference on Data Mining, IEEE, 2013, pp. 587–596

    2013

  6. [14]

    Data Curation Alone Can Stabilize In-context Learning,

    T.-Y. Chang and R. Jia, “Data Curation Alone Can Stabilize In-context Learning,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 8123–8144

    2023

  7. [15]

    Diffusion Convolutional Recurrent Neural Network: Data -Driven Traffic Forecasting,

    Y. Li, R. Yu, C. Shahabi, and Y. Liu, “Diffusion Convolutional Recurrent Neural Network: Data -Driven Traffic Forecasting,” International Conference on Learning Representations, 2018

    2018

  8. [16]

    (30 April 2025)

    California Department of Transportation. (30 April 2025). Freeway Performance Measurement System (PeMS). Available: http://pems.dot.ca.gov

    2025

  9. [17]

    A comparative study of machine learning algorithms to predict road accident severity,

    S. Ahmed, M. A. Hossain, M. M. I. Bhuiyan, and S. K. Ray, “A comparative study of machine learning algorithms to predict road accident severity,” presented at the 20th International Conference on Ubiquitous Computing and Communications , IEEE, 2021, pp. 390 – 397

    2021

  10. [18]

    Robust Prompt Optimization for Large Language Models Against Distribution Shifts,

    M. Li, W. Wang, F. Feng, Y. Cao, J. Zhang, and T. -S. Chua, “Robust Prompt Optimization for Large Language Models Against Distribution Shifts,” in the Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 1539–1554 Figure 5. Comparis...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.