REVIEW 5 major objections 6 minor 1 cited by
Predicting person-level injury severity using crash narratives: A balanced approach with roadway classification and natural language process techniques
T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Adding police-written crash narratives to structured crash data improves person-level injury severity prediction, with TF-IDF plus XGBoost performing best.
desk verdict Systematic road-class comparison and a credible ISS validation are real positives, but the paper's central claim—narrative beats structured—is never tested because no structured-only baseline is reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the concatenation of TF-IDF term-weight vectors derived from crash narratives with the top-100 structured features selected by Random Forest importance, fed into XGBoost. TF-IDF assigns higher weight to words that are frequent within a narrative but rare across the corpus, so phrases like 'ejected', 'unresponsive', or 'head-on' can act as direct severity signals; Word2Vec serves as the semantic-context alternative. SMOTE oversampling is applied to the training partition to address the rarity of fatal and serious injuries, and the evaluation metric that carries the comparison is macro-averaged F1, which treats all four severity classes equally. External validation is provided by cross-tabulating predicted severity with Injury Severity Score categories from linked trauma records.
What would settle it
Take a random sample of crash-involved persons, obtain their true injury outcomes from hospital or trauma records (including no-injury cases), and compare TF-IDF plus XGBoost predictions against those ground-truth labels; if the narrative-augmented model does not outperform structured-only models, or if the predicted severity gradient against ISS disappears, the paper's central claim would be refuted.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is that structured crash variables alone leave predictive signal on the table, and that police-written narratives recover much of it. Across 17 roadway-stratified datasets (eight detailed functional classes, four binary pairings, and one pooled set) and 102 model configurations, models incorporating narrative data consistently outperformed those relying solely on structured data, and TF-IDF plus XGBoost won in most subgroups. The authors also show that the predicted severity categories align monotonically with clinical Injury Severity Scores: average ISS falls from 23.0 for predicted Fatal to 10.6 for Possible Injury, supporting the model's external validity and its proposed use for imputing missing severities in the ~67,000-record dataset.
Load-bearing premise
The load-bearing assumption is that the records with reported injury severity are representative of all persons in crashes, including the majority whose severity is missing and the excluded 'No Injury' cases; if that subset is biased, the model's performance and imputations will not transfer.
Editorial extensions
If this is right
- Transportation agencies can apply the trained TF-IDF plus XGBoost model to impute injury severities for the over 50% of Kentucky crash records with missing labels, months before official statistics are released.
- Per-roadway-class model selection matters: the paper shows that grouping rare road types into binary pairs (e.g., Freeway vs. Non-Freeway) stabilizes predictions for rare outcomes like fatal crashes, so practitioners should not default to a single pooled model.
- Because predicted severity tracks clinical ISS, the framework can serve as a proxy for injury outcomes in settings where hospital data are unavailable, supporting triage and emergency-response planning.
- The sensitivity-analysis capability lets safety officials simulate policy changes, such as posted speed limit adjustments, and estimate their effect on the distribution of predicted injury severities.
Reading between the lines
- The paper's result suggests that in this domain, discriminative vocabulary (TF-IDF) beats semantic embeddings (Word2Vec) for short police narratives; an untested extension is whether a transformer-based encoder fine-tuned on crash reports would further improve macro F1, or whether the narrative signal is mostly lexical.
- Because the study excludes the 'No Injury' class and trains only on records with reported severity, the real-world performance depends on the labeled subset being representative; a direct test is to apply the model to a sample with complete clinical labels and compare class distributions.
- The roadway-stratification design implicitly tests domain adaptation; one could also add road class as a categorical feature in a single pooled model, which would let the model share statistical strength across road types instead of training separate models.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops person-level injury severity prediction models for Kentucky crash records (2019–2023) by combining structured crash data with police-written crash narratives. Two text representations, TF-IDF and Word2Vec, are paired with three ensemble classifiers (XGBoost, Random Forest, AdaBoost) under three roadway classification schemes, producing 102 models. To handle class imbalance, SMOTE is applied to training data. The central claim is that narrative-augmented models consistently outperform structured-only models, with TF-IDF plus XGBoost best in most subgroups. The paper also validates the group-3 XGBoost model against trauma-registry ISS scores and proposes the framework for imputing missing severity labels.
Significance. If the central claim were established, the paper would offer a practical framework for crash severity imputation and a useful comparison of roadway stratification strategies, and its external validation against ISS data is a genuine strength. The multi-dataset design across 17 roadway subgroups and the use of a held-out test set for final evaluation are also positive elements. However, the reported results do not currently support the headline added-value claim, because no structured-only baseline is presented. The external ISS validation is informative about the narrative-enhanced model's clinical alignment but does not substitute for a controlled comparison. The practical imputation claim additionally rests on unresolved data-accounting and missingness issues. With the addition of the missing comparison and clarification of the pipeline, the contribution could be meaningful for transportation safety practice.
major comments (5)
- [§5.1, Figures 4–5, Tables 2–3] The paper's central claim (Abstract and §1) is that narrative-augmented models consistently outperform structured-only models, yet no structured-only baseline is reported anywhere in §5. Figures 4–5 and Tables 2–3 rank only narrative-enhanced models by group, and the text does not report macro-F1, precision, recall, or accuracy for a structured-only XGBoost, Random Forest, or AdaBoost model on any of the 17 datasets. Since §1 states that models were first developed using only structured crash data, these results should already exist; they need to be reported with the same pipeline, evaluation protocol, and metrics as the narrative-enhanced models. Without this baseline, the added-value claim is unmeasured.
- [§6 vs. §3.2 and Table 1] The Conclusion states that in the 67,000-record dataset only about 3,200 cases had reported severities, whereas §3.2 and Table 1 describe a person-level dataset with roughly 32,000 severity-labeled records (e.g., 16,583 male and 15,713 female records), and Table 5 lists test-set supports that sum to 26,800 for the group-3 model. These numbers cannot all be correct. The authors must reconcile the data accounting, because the practical imputation claim depends on knowing how many records are labeled and how the modeling subset relates to the full dataset.
- [§4.6.2] The Word2Vec pipeline is incomplete: the paper states that the embedding dimension is 100 and that a 200-word narrative yields a 200×100 matrix, but it never specifies how the per-word vectors are aggregated into a single document-level vector for the classifiers. Options such as mean pooling, max pooling, concatenation, or training a document-embedding model are not described. The training corpus, window size, minimum count, and number of epochs are also omitted. Without this information, the Word2Vec results in Figure 4 and Table 2 cannot be reproduced.
- [§4.4] The feature-selection procedure does not state whether the Random Forest importance scores used to select the top 100 features are computed on the training split only. If the importance ranking is computed on the full dataset before the 70/15/15 split into training, validation, and test sets, then information from the validation and test sets leaks into the feature selection and the reported metrics are optimistically biased. Please clarify that importance is computed inside the training portion, or modify the procedure accordingly.
- [§3.2 and §6] Because the 'No Injury' class is excluded and only records with a reported severity are used for modeling, the proposal to use the model to impute missing severities in the full dataset assumes that the labeled subset is representative of all persons involved in crashes. If missingness is correlated with severity or with crash characteristics, the imputed distribution will be biased. The manuscript should either provide evidence that labels are missing at random or restrict the imputation claim to the population of records with reported severities.
minor comments (6)
- [Abstract] There is a typo in the phrase 'sem antic meaning'; please proofread the text for similar spacing errors.
- [Table 1] In the Speed Limit 30–50 row, the percentage '51.36' appears as '5136'; the missing decimal point should be restored so that the row sums to 100.
- [§5.1] The sentence beginning 'As shown in Tables 4 and 5, except for Group 2 within the Urban class...' refers to the ISS cross-tabulations, not to the TF-IDF/Word2Vec comparison; the intended references appear to be Tables 2 and 3 or Figures 4 and 5.
- [§2] The citation 'Jiang et al. (6)' does not match reference [6], which is Christoforou et al.; the transformer-based discussion appears to refer to reference [4] (Jiang et al.). In addition, the text 'Do et al.' corresponds to reference [10] by Du et al.; please correct the author names in the citations.
- [Figures 2 and 3] Figures 2 and 3 are not described in sufficient detail in the text. In particular, Figure 2 does not illustrate how a document-level representation is obtained from per-word Word2Vec embeddings, which is a central methodological question.
- [General] The paper would benefit from a data and code availability statement; no repository or release is mentioned, which limits reproducibility given the complexity of the preprocessing pipeline.
Circularity Check
No significant circularity: model predictions are tested on held-out data and externally validated against ISS; the missing structured-only baseline is an evidentiary gap, not a circular derivation.
full rationale
The paper's central derivation is a supervised machine-learning pipeline: structured features are combined with TF-IDF or Word2Vec narrative representations and fed to XGBoost, Random Forest, or AdaBoost, with SMOTE applied only to the training set and evaluation performed on a stratified held-out test set (Sections 4.2, 4.3, and 5.1). No fitted parameter is relabeled as a prediction, and the test set is not used for model selection. The ISS clinical validation in Section 5.2 is an external benchmark that does not depend on the prediction targets used for training, so it provides independent evidence rather than a circular check. The only self-citation is reference [23], used as a source for roadway functional classes; it is not load-bearing for the central claim. No uniqueness theorem or ansatz is imported from the authors' prior work, and no known result is merely renamed. The main weakness—the Abstract and Section 1 claim that narrative-enhanced models 'consistently outperform those relying solely on structured data' is never supported by reported structured-only baseline results, and Section 6's '~3200' labeled cases appears inconsistent with the roughly 32,000 labeled persons implied by Table 1—is a missing-baseline and data-accounting problem, not a circularity. Therefore no circular steps are identified, and the derivation chain is self-contained with respect to the evaluation performed.
Assumptions & free parameters
free parameters (5)
- Number of top structured features retained =
100
- Word2Vec embedding dimension =
100
- Missingness exclusion threshold =
0.50
- SMOTE k-nearest neighbors
- Train/validation/test split ratio =
70/15/15
assumptions (5)
- domain assumption Police narratives contain predictive information beyond structured fields
- domain assumption The labeled subset of crashes is representative of the full crash population
- domain assumption Macro F1 is the appropriate primary metric for model ranking
- domain assumption ISS linked trauma registry data accurately reflects clinical injury severity
- standard math Standard ML assumptions of i.i.d. samples and meaningful feature representations
Cite this review
Pith. "Pith review of Predicting person-level injury severity using crash narratives: A balanced approach with roadway classification and natural language process techniques." pith.science (2026). https://pith.science/paper/7YUBDUYG
@misc{pith2026250907845,
author = {Pith},
title = {Pith review of: Predicting person-level injury severity using crash narratives: A balanced approach with roadway classification and natural language process techniques},
year = {2026},
howpublished = {\url{https://pith.science/paper/7YUBDUYG}},
note = {Machine review of arXiv:2509.07845}
}
read the original abstract
Predicting injuries and fatalities in traffic crashes plays a critical role in enhancing road safety, improving emergency response, and guiding public health interventions. This study investigates the added value of unstructured crash narratives (written by police officers at the scene) when combined with structured crash data to predict injury severity. Two widely used Natural Language Processing (NLP) techniques, Term Frequency-Inverse Document Frequency (TF-IDF) and Word2Vec, were employed to extract semantic meaning from the narratives, and their effectiveness was compared. To address the challenge of class imbalance, a K-Nearest Neighbors-based oversampling method was applied to the training data prior to modeling. The dataset consists of crash records from Kentucky spanning 2019 to 2023. To account for roadway heterogeneity, three road classification schemes were used: (1) eight detailed functional classes (e.g., Urban Two-Lane, Rural Interstate, Urban Multilane Divided), (2) four broader paired categories (e.g., Urban vs. Rural, Freeway vs. Non-Freeway), and (3) a unified dataset without classification. A total of 102 machine learning models were developed by combining structured features and narrative-based features using the two NLP techniques alongside three ensemble algorithms: XGBoost, Random Forest, and AdaBoost. Results demonstrate that models incorporating narrative data consistently outperform those relying solely on structured data. Among all combinations, TF-IDF coupled with XGBoost yielded the most accurate predictions in most subgroups. The findings highlight the power of integrating textual and structured crash information to enhance person-level injury prediction. This work offers a practical and adaptable framework for transportation safety professionals to improve crash severity modeling, guide policy decisions, and design more effective countermeasures.
Forward citations
Cited by 1 Pith paper
-
Benchmarking Frontier Large Language Models Against Official Crash Database Coding Using Police Crash Narratives
On 4,194 matched Arkansas fatal crashes, frontier LLMs did not beat a crude keyword-rule baseline at reproducing official crash coding, and agreement varied more by crash attribute than by model.
Reference graph
Works this paper leans on
-
[1]
Khairuddin, M. Z. F., S. Sankaranarayanan, K. Hasikin, N. A. Abd Razak, and R. Omar. Contextualizing injury severity from occupational accident reports using an optimized deep learning prediction model. PeerJ Computer Science, V ol. 10, 2024, p. e1985
work page 2024
-
[2]
Goldberg, D. M. Characterizing accident narratives with word embeddings: Improving accuracy, richness, and generalizability. Journal of safety research, V ol. 80, 2022, pp. 441-455
work page 2022
-
[3]
Behboudi, N., S. Moosavi, and R. Ramnath. Recent advances in traffic accident analysis and prediction: a comprehensive review of machine learning techniques. arXiv preprint arXiv:2406.13968, 2024
arXiv 2024
-
[4]
Jiang, Y ., X. Qu, W. Zhang, W. Guo, J. Xu, W. Yu, and Y . Chen. Analyzing Crash Severity: Human Injury Severity Prediction Method Based on Transformer Model. Vehicles, V ol. 7, No. 1, 2025, p. 5
work page 2025
-
[5]
Fu, C., and H.-T. Tu. Investigating vehicle-vehicle and vehicle–pedestrian crash severity at street intersections with the latent class parameterized correlation bivariate generalized ordered probit. Accident Analysis & Prevention, V ol. 207, 2024, p. 107745
work page 2024
-
[6]
Christoforou, Z., S. Cohen, and M. G. Karlaftis. Vehicle occupant injury severity on highways: An empirical investigation. Accident Analysis & Prevention, V ol. 42, No. 6, 2010, pp. 1606-1620
work page 2010
-
[7]
Sadeghi, P., and A. Goli. Investigating the impact of pavement condition and weather characteristics on road accidents. International Journal of Crashworthiness, V ol. 29, No. 6, 2024, pp. 973-989
work page 2024
-
[8]
Iranitalab, A., and A. Khattak. Comparison of four statistical and machine learning methods for crash severity prediction. Accident Analysis & Prevention, V ol. 108, 2017, pp. 27-36
work page 2017
Show all 46 references
-
[9]
Darkhaneh, M. E., M. Effati, and M. Arabani. Factors affecting the injury severity of head-on crashes on undivided rural roads under different weather conditions. International Journal of Transportation Science and Technology, 2025
2025
-
[10]
Zhang, A
Du, B., C. Zhang, A. Sarkar, J. Shen, A. Telikani, and H. Hu. Identifying factors related to pedestrian and cyclist crashes in ACT, Australia with an extended crash dataset. Accident Analysis & Prevention, V ol. 207, 2024, p. 107742
2024
-
[11]
Eum, Y ., and Y . Chen. Factors Associated with Vulnerable Road Users on Freeways in North Carolina.In International Conference on Transportation and Development 2025, 2025. pp. 183-193
2025
-
[12]
Albayrak
Çeven, S., and A. Albayrak. Traffic accident severity prediction with ensemble learning methods. Computers and Electrical Engineering, V ol. 114, 2024, p. 109101
2024
-
[13]
Sivarak, Z.-J
Ruangkanjanases, A., O. Sivarak, Z.-J. Weng, A. Khan, and S.-C. Chen. Using multilayer perceptron neural network to assess the critical factors of traffic accidents. HighTech and Innovation Journal, V ol. 5, No. 1, 2024, pp. 157-169
2024
-
[14]
Zahid, M
Jamal, A., M. Zahid, M. Tauhidur Rahman, H. M. Al-Ahmadi, M. Almoshaogeh, D. Farooq, and M. Ahmad. Injury severity prediction of traffic crashes with ensemble machine learning techniques: a comparative study. Int J Inj Contr Saf Promot, V ol. 28, No. 4, 2021, pp. 408-427
2021
-
[15]
A Novel Stacking Framework Based On Hybrid of Gradient Boosting-Adaptive Boosting-Multilayer Perceptron for Crash Injury Severity Prediction and Analysis,
J. Niyogisubizo, L. L., Y . Lin, L. Luo, E. Nziyumva and E. Murwanashyaka, . "A Novel Stacking Framework Based On Hybrid of Gradient Boosting-Adaptive Boosting-Multilayer Perceptron for Crash Injury Severity Prediction and Analysis," 2021 IEEE 4th International Conference on E...
2021
-
[16]
Abdel-Aty, J
Cai, Q., M. Abdel-Aty, J. Yuan, J. Lee, and Y . Wu. Real-time crash prediction on expressways using deep generative models. Transportation research part C: emerging technologies, V ol. 117, 2020, p. 102697
2020
-
[17]
A., and H
Rahim, M. A., and H. M. Hassan. A deep learning based traffic crash severity prediction framework. Accid Anal Prev, V ol. 154, 2021, p. 106090
2021
-
[18]
Islam, M. R., D. Wang, and M. Abdel-Aty. Calibrated confidence learning for large-scale real-time crash and severity prediction. npj Sustainable Mobility and Transport, V ol. 1, No. 1, 2024, p. 1
2024
-
[19]
Fan, Z., P . Wang, Y . Zhao, Y . Zhao, B. Ivanovic, Z. Wang, M. Pavone, and H. F. Yang. Learning Traffic Crashes as Language: Datasets, Benchmarks, and What-if Causal Analyses. arXiv preprint arXiv:2406.10789, 2024
2024 arXiv
-
[20]
Li, P., S. Chen, L. Yue, Y . Xu, and D. A. Noyce. Analyzing relationships between latent topics in autonomous vehicle crash narratives and crash severity using natural language processing techniques and explainable XGBoost. Accid Anal Prev, V ol. 203, 2024, p. 107605
2024
-
[21]
Chen, J., W. Tao, Z. Jing, P. Wang, and Y . Jin. Traffic accident duration prediction using multi-mode data and ensemble deep learning. Heliyon, V ol. 10, No. 4, 2024, p. e25957
2024
-
[22]
Wali, B., A. J. Khattak, and N. Ahmad. Injury severity analysis of pedestrian and bicyclist trespassing crashes at non-crossings: A hybrid predictive text analytics and heterogeneity-based statistical modeling approach. Accident Analysis & Prevention, V ol. 150, 2021, p. 105835
2021
-
[23]
Tanzen, and T
Souleyrette, R., R. Tanzen, and T. Wang. Incorporating Crash Severity and Continuous Improvement of SHIFT.In, University of Kentucky Transportation Center, 2023
2023
-
[24]
Rodriguez
Acuna, E., and C. Rodriguez. The treatment of missing values and its effect on classifier accuracy.In Classification, Clustering, and Data Mining Applications: Proceedings of the Meeting of the International Federation of Classification Societies (IFCS), Illinois Institute of ...
2004
-
[25]
A., and S
Imtiaz, S. A., and S. L. Shah. Treatment of missing values in process data analysis. The Canadian Journal of Chemical Engineering, V ol. 86, No. 5, 2008, pp. 838-858
2008
-
[26]
Deep learning.In, MIT press, 2016
Goodfellow, I. Deep learning.In, MIT press, 2016
2016
-
[27]
Kuhn, M., and K. Johnson. Applied predictive modeling. Springer, 2013
2013
-
[28]
Milella, D
Campagner, A., F. Milella, D. Ciucci, and F. Cabitza. Three-way decision in machine learning tasks: a systematic review. Artificial Intelligence Review, V ol. 57, No. 9, 2024, p. 228
2024
-
[29]
Zheng, Y
Yan, Y ., Z. Zheng, Y . Zhang, Y . Zhang, and Y . Yao. CPS-3WS: A critical pattern supported three-way sampling method for classifying class-overlapped imbalanced data. Information Sciences, V ol. 676, 2024, p. 120835
2024
-
[30]
Chawla, N. V ., K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. SMOTE: synthetic minority over- sampling technique. Journal of artificial intelligence research, V ol. 16, 2002, pp. 321-357
2002
-
[31]
Elangovan, K
Sampath, P., G. Elangovan, K. Ravichandran, V . Shanmuganathan, S. Pasupathi, T. Chakrabarti, P. Chakrabarti, and M. Margala. Robust diabetic prediction using ensemble machine learning models with synthetic minority over-sampling technique. Scientific reports, V ol. 14, No. 1,...
2024
-
[32]
Batista, G. E., R. C. Prati, and M. C. Monard. A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD explorations newsletter, V ol. 6, No. 1, 2004, pp. 20-29
2004
-
[33]
Mujahid, M., E. Kına, F. Rustam, M. G. Villar, E. S. Alvarado, I. De La Torre Diez, and I. Ashraf. Data oversampling and imbalanced datasets: An investigation of performance for machine learning and feature engineering. Journal of Big Data, V ol. 11, No. 1, 2024, p. 87
2024
-
[34]
Elisseeff
Guyon, I., and A. Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, V ol. 3, No. Mar, 2003, pp. 1157-1182
2003
-
[35]
Cai, J., J. Luo, S. Wang, and S. Yang. Feature selection in machine learning: A new perspective. Neurocomputing, V ol. 300, 2018, pp. 70-79
2018
-
[36]
Guleria, and N
Kaur, A., K. Guleria, and N. K. Trivedi. Feature selection in machine learning: Methods and comparison.In 2021 International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), IEEE, 2021. pp. 789-795
2021
-
[37]
Research on feature selection methods based on random forest
Wang, Z. Research on feature selection methods based on random forest. Tehnički vjesnik, V ol. 30, No. 2, 2023, pp. 623-633
2023
-
[38]
Random forests
Breiman, L. Random forests. Machine learning, V ol. 45, 2001, pp. 5-32
2001
-
[39]
Li, and H
Li, Y ., T. Li, and H. Liu. Recent advances in feature selection and its applications. Knowledge and Information Systems, V ol. 53, 2017, pp. 551-577
2017
-
[40]
Lv, and N
Yang, Y ., H. Lv, and N. Chen. A survey on ensemble learning under the era of deep learning. Artificial Intelligence Review, V ol. 56, No. 6, 2023, pp. 5545-5589
2023
-
[41]
Qi-Guang, L
Ying, C., M. Qi-Guang, L. Jia-Chen, and G. Lin. Advance and prospects of AdaBoost algorithm. Acta Automatica Sinica, V ol. 39, No. 6, 2013, pp. 745-758
2013
-
[42]
Guestrin
Chen, T., and C. Guestrin. Xgboost: A scalable tree boosting system.In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016. pp. 785-794
2016
-
[43]
Kavzoglu, T., and A. Teke. Predictive performances of ensemble machine learning algorithms in landslide susceptibility mapping using random forest, extreme gradient boosting (XGBoost) and natural gradient boosting (NGBoost). Arabian Journal for Science and Engineering, V ol. 4...
2022
-
[44]
Al-Obaydy, W. I., H. A. Hashim, Y . Najm, and A. A. Jalal. Document classification using term frequency-inverse document frequency and K-means clustering. Indonesian Journal of Electrical Engineering and Computer Science, V ol. 27, No. 3, 2022, pp. 1517-1524
2022
-
[45]
Mikolov, T., K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013
2013 arXiv
-
[46]
A., and D
Asensio, J. A., and D. D. Trunkey. Current therapy of trauma and surgical critical care e-book. Elsevier Health Sciences, 2008
2008
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.