REVIEW 4 major objections 5 minor 25 references
Exploring Aviation Incident Narratives Using Topic Modeling and Clustering Techniques
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that Latent Dirichlet Allocation extracts the most coherent topics from over 36,000 NTSB aviation incident narratives, with a coherence score of 0.597, and that the extracted themes and clusters reveal recurring incident…
desk verdict A modest, under-specified topic-modeling comparison on 20 years of NTSB narratives; the headline ranking rests on an unreported K=10, so treat the numbers as provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the coherence score C_v computed over topics produced by four models, each fed the same preprocessed narrative corpus. C_v measures semantic coherence by asking whether the top words of a topic tend to co-occur within documents, and higher values are taken as better topics. The topic models themselves are standard: LDA treats each document as a mixture of topics drawn from a Dirichlet prior; pLSA is its probabilistic precursor trained with expectation-maximization; LSA applies singular value decomposition to the document-term matrix; and NMF factorizes it into non-negative topic and term matrices. K-means then groups the TF-IDF or embedding representations into ten clusters, and t-SNE visualizes them. The paper's ranking claim rests entirely on the C_v score, so the way coherence is computed is the single most load-bearing piece of the argument.
What would settle it
Re-run the pipeline with the number of topics varied from 5 to 20 for every model, record C_v at each value, and repeat K-means at the same numbers. If the coherence-maximizing number of topics differs between models or from 10, or if LDA loses its top position at that coherence-maximizing value, then the reported ranking and the ten labeled themes are an artifact of the fixed choice rather than a property of the corpus.
Extended reading notes
Core claim
The study's central claim is that, measured by C_v coherence, LDA performs best on this corpus (0.597), followed by pLSA (0.583), LSA (0.542), and NMF (0.437), and that the topics each model extracts converge on a small set of recurring aviation incident themes: mechanical failures, fuel system issues, and engine and power loss. Each model also adds distinct perspectives, such as NMF's emphasis on helicopter operations and flight conditions, and LSA's connections between wind and weather impact and rudder and brake issues. The ten K-means clusters, visualized with t-SNE, group narratives into categories like runway incidents and landing gear and helicopter rotor problems, offering a complementary structured view of the incident stories.
Load-bearing premise
The strongest results assume that ten topics per model and ten K-means clusters is the right level of detail for the whole corpus, and the paper never gives the rule that picked ten; if a different number of themes fits the data better, the coherence ranking, the topic labels, and the clusters could all change.
Editorial extensions
If this is right
- Safety analysts could use LDA's ten topics as a table of contents for the NTSB corpus and prioritize recurring mechanical, fuel, and engine/power-loss themes without reading each narrative by hand.
- The agreement among four different algorithms on core themes, despite their different assumptions, makes those themes a more credible description of incident narratives than any single model's output.
- On similar aviation narrative corpora, LDA would be a defensible default choice among the four methods tested, because it has the highest coherence score.
- K-means provides a second, cluster-based view (for example, runway incidents versus landing gear and helicopter rotor problems) that can be combined with topic themes to organize safety review workflows.
- The same preprocessing and modeling pipeline can be transferred to other accident-report archives, offering a low-cost way to surface recurring safety themes in domains beyond aviation.
Reading between the lines
- The paper fixes the number of topics at ten for every model without giving a selection rule; if the optimal number of themes varies by model, the coherence ranking could change, so a coherence sweep over a range of topic counts is the most direct extension.
- The topic labels are derived from inspecting top words rather than validated against NTSB's official probable-cause classifications; comparing the algorithmic themes to those official categories would test whether the model boundaries match the agency's own taxonomy.
- Because the corpus spans 2000 to 2020, the same pipeline applied year-by-year could reveal whether themes such as fuel system issues are stable or evolving; the paper lists temporal analysis as future work.
- Cluster stability is untested; running K-means on bootstrap resamples of the narratives would show whether the ten clusters, including the smaller ones, reproduce or are artifacts of initialization.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies four topic-modeling techniques (LDA, NMF, LSA, pLSA) and K-means clustering to NTSB aviation incident narratives (2000-2020, >36,000 records). It reports coherence scores (C_v) for each topic model, with LDA highest at 0.597, followed by pLSA (0.583), LSA (0.542), and NMF (0.437), and it presents 10-topics-per-model interpretations (Tables II and III) plus a t-SNE visualization of ten K-means clusters. The authors claim that the extracted topics reveal recurring themes such as Mechanical Failures, Fuel System Issues, and Engine and Power Loss, and that the comparative analysis demonstrates the versatility of NLP topic-modeling methods for aviation safety.
Significance. If the comparative claim were properly supported, the paper would offer a useful descriptive application of standard topic-modeling tools to a valuable public safety corpus, and its coherence ranking could inform practitioners selecting a topic model for NTSB narratives. The paper's strengths are its use of an identifiable public dataset, a clear preprocessing pipeline, and a straightforward comparison of four algorithms. However, the central quantitative claim—that LDA 'performed best'—rests on an unreported and unjustified choice of K=10 for all models, single-point coherence values without variance or hyperparameter detail, and an informal topic-labeling procedure. As reported, the results are not auditable and the ranking may be an artifact of the modeling choices rather than a robust property of the corpus. The contribution is therefore currently limited to a descriptive demonstration rather than a validated comparative finding.
major comments (4)
- [Section III.G and Table I] The headline claim that 'LDA performed best with a coherence value of 0.597' (Abstract, Section IV.A, Table I) is not supported because the number of topics K is never justified for any model. Section III does not state how K=10 was selected for LDA, NMF, LSA, or pLSA, and Section IV.F also uses ten K-means clusters without a selection criterion. Since C_v coherence is known to be sensitive to the number of topics, and the reported margin between LDA (0.597) and pLSA (0.583) is only 0.014, the ranking could plausibly change if a different K were used. The authors should report a K-selection procedure (e.g., a coherence sweep, elbow, or silhouette analysis) or a sensitivity analysis over K for each model.
- [Section III.G and Table I] The implementation details are insufficient for reproducibility. Section III.G says parameters were 'fine-tuned to optimize performance' but gives no parameter values, search grids, or random seeds, and the coherence scores in Table I are single numbers with no variance, number of runs, or confidence intervals. Additionally, the coherence metric parameters (e.g., window size, top-N words for C_v) are not specified. The authors should report these details, or explicitly state which settings were used, so that the coherence values can be independently audited.
- [Section IV.D and Tables II-III] The paper's topic interpretations are informal and not validated. Section IV.D states that Fig. 6 'helps to determine the optimal number of topics,' but no optimal K is ever reported and no external validation (e.g., comparison with NTSB cause/factor fields, human expert judgement, or held-out evaluation) is provided. The labels in Tables II and III (e.g., 'Mechanical Failures,' 'Fuel System Issues') appear to be the authors' semantic readings of the top words rather than a validated outcome. The authors should either provide a validation procedure for these labels or clearly mark them as illustrative interpretations.
- [Section IV.F and Fig. 7] The K-means clustering results are under-specified. Section IV.F does not state which document representation was used (TF-IDF or Word2Vec, or another), how K was chosen for the ten clusters, or any cluster-quality metric such as silhouette score or within-cluster sum of squares. The t-SNE plot (Fig. 7, also referred to as Fig. 12 in the text) is descriptive and cannot substitute for a quantitative assessment of cluster separation. Without this information, the clustering-based claims about 'commonalities and unique insights' are not verifiable.
minor comments (5)
- [Section III] The section begins with the incomplete phrase 'Before In this section' which appears to be a leftover editing fragment; it should be removed.
- [References and Table III] Some references appear not to be cited in the text (e.g., [12] and [16]), and Table III is placed after the References section in the manuscript layout, which will confuse readers; the table should be moved into the results section and all references should be cited in order.
- [Section IV.F] The text refers to 'Fig. 12' for the t-SNE visualization, but the figure is labeled Fig. 7; the numbering should be corrected.
- [Section V] The conclusion states that clustering 'enhances the roughness of safety analysis'; this appears to be a typo, likely intended to be 'thoroughness' or 'robustness'.
- [Throughout] The acronym for Probabilistic Latent Semantic Analysis is rendered inconsistently as both 'pLSA' and 'PLSA' (e.g., Table I); please standardize.
Circularity Check
No circularity: the coherence ranking is an empirical output of fitted models, not an input or self-citation-derived conclusion.
full rationale
I walked the derivation chain from data acquisition (Section III.A), preprocessing (III.B), model fitting (III.C and III.D), coherence evaluation (III.F and IV.A), topic interpretation (IV.C-E), and clustering (IV.F). The central claim, that LDA achieved the highest C_v coherence of 0.597, is reported as the output of fitting four separate topic models to the NTSB narratives; no equation defines coherence in terms of the desired ranking, and no model parameter is fitted to the coherence values. The topic labels in Tables II and III are manual interpretations of the top words produced by the models, not inputs that force those labels. The paper contains multiple self-citations (e.g., [3], [7], [10], [18], [24]), but these appear in the related-work narrative and are not used to justify the choice of methods or to license the coherence comparison; the methods themselves are attributed to established external sources such as Blei et al. [14], Hofmann [23], Landauer et al. [22], and Lee and Seung [15]. The absence of an explicit justification for using exactly 10 topics, and the lack of coherence hyperparameters, seeds, or variance estimates, are reproducibility and robustness limitations, not circular reasoning: the reported ranking is not equivalent by construction to the arbitrary K choice, and changing K could change the ranking only as an empirical matter, not as a logical tautology. No step was found where a 'prediction' reduces to its own input or where a load-bearing conclusion rests on a self-citation chain. Therefore the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- Number of topics K (all models) =
10 (reported)
- Topic model hyperparameters (LDA priors/passes, NMF regularization, pLSA iterations, K-means initialization) =
Unreported
- Text preprocessing settings (stopword list, lemmatizer, vocabulary filters, TF-IDF/Word2Vec settings) =
Unreported
assumptions (5)
- domain assumption The NTSB query returned the complete intended corpus of incidents from 2000 to 2020 with no further filtering needed.
- domain assumption Cleaning steps (stopword removal, lemmatization, punctuation/URL removal) preserve the incident-related signal.
- ad hoc to paper Ten topics is the appropriate granularity for every model.
- domain assumption Coherence value C_v is a sufficient measure of topic quality for comparing models.
- standard math Standard bag-of-words topic model assumptions (exchangeability, Dirichlet/multinomial structure) apply to incident narratives.
Cite this review
Pith. "Pith review of Exploring Aviation Incident Narratives Using Topic Modeling and Clustering Techniques." pith.science (2026). https://pith.science/paper/2EIXQDRZ
@misc{pith2026250107924,
author = {Pith},
title = {Pith review of: Exploring Aviation Incident Narratives Using Topic Modeling and Clustering Techniques},
year = {2026},
howpublished = {\url{https://pith.science/paper/2EIXQDRZ}},
note = {Machine review of arXiv:2501.07924}
}
read the original abstract
Aviation safety is a global concern, requiring detailed investigations into incidents to understand contributing factors comprehensively. This study uses the National Transportation Safety Board (NTSB) dataset. It applies advanced natural language processing (NLP) techniques, including Latent Dirichlet Allocation (LDA), Non-Negative Matrix Factorization (NMF), Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis (pLSA), and K-means clustering. The main objectives are identifying latent themes, exploring semantic relationships, assessing probabilistic connections, and cluster incidents based on shared characteristics. This research contributes to aviation safety by providing insights into incident narratives and demonstrating the versatility of NLP and topic modelling techniques in extracting valuable information from complex datasets. The results, including topics identified from various techniques, provide an understanding of recurring themes. Comparative analysis reveals that LDA performed best with a coherence value of 0.597, pLSA of 0.583, LSA of 0.542, and NMF of 0.437. K-means clustering further reveals commonalities and unique insights into incident narratives. In conclusion, this study uncovers latent patterns and thematic structures within incident narratives, offering a comparative analysis of multiple-topic modelling techniques. Future research avenues include exploring temporal patterns, incorporating additional datasets, and developing predictive models for early identification of safety issues. This research lays the groundwork for enhancing the understanding and improvement of aviation safety by utilising the wealth of information embedded in incident narratives.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
The Nature and Costs of Civil Aviation Flight Training Safety Occurrences,
A. Somerville, T. Lynar, and G. Wild, “The Nature and Costs of Civil Aviation Flight Training Safety Occurrences,” vol. 12, pp. 100182 – 100182, Jun. 2023, doi: https://doi.org/10.1016/j.treng.2023.100182
arXiv 2023
-
[2]
N. AZIIDA, S. MALEK, F. AZIZ, K. S. IBRAHIM, and S. KASIM, “Predicting 30 -Day Mortality after an Acute Coronary Syndrome (ACS) using Machine Learning Methods for Feature Selection, Classification and Visualisation,” Sains Malaysiana, vol. 50, no. 3, pp. 753–768, Mar. 2021, doi: https://doi.org/10.17576/jsm-2021-5003-17
-
[3]
Nanyonga, A., & Wild, G. (2023, December). Impact of Dataset Size & Data Source on Aviation Safety Incident Prediction Models with Natural Language Processing. In 2023 Global Conference on Information Technologies and Communications (GCITC) (pp. 1 -7). IEEE
work page 2023
-
[4]
Augmenting Topic Finding in the NASA Aviation Safety Reporting System using Topic Modeling,
C. Paradis, R. Kazman, M. Davies, and B. Hooey, “Augmenting Topic Finding in the NASA Aviation Safety Reporting System using Topic Modeling,” AIAA Scitech 2021 Forum, Jan. 2021, doi: https://doi.org/10.2514/6.2021-1981
-
[5]
Exploring Topic Coherence over Many Models and Many Topics,
K. Stevens, P. Kegelmeyer, D. Andrzejewski, and D. Buttler, “Exploring Topic Coherence over Many Models and Many Topics,” ACLWeb, Jul. 01, 2012. https://aclanthology.org/D12-1087 (accessed May 06, 2024)
work page 2012
-
[6]
Text mining of accident reports using semi -supervised keyword extraction and topic modelling,
A. Ahadh, G. V. Binish, and R. Srinivasan, “Text mining of accident reports using semi -supervised keyword extraction and topic modelling,” Process Safety and Environmental Protection, vol. 155, pp. 455–465, Nov. 2021, doi: https://doi.org/10.1016/j.psep.2021.09.022
-
[7]
Nanyonga, A., Wasswa, H., & Wild, G. (2024, March). Comparative Study of Deep Learning Architectures for Textual Damage Level Classification. In 2024 11th International Conference on Signal Processing and Integrated Networks (SPIN) (pp. 421-426). IEEE
work page 2024
-
[8]
The Limits of Expertise: Rethinking Pilot Error and the Causes of Airline Accidents,
T. G. C. Griffin, “The Limits of Expertise: Rethinking Pilot Error and the Causes of Airline Accidents,” Ergonomics, vol. 51, no. 7, pp. 1119– 1120, Jun. 2008, doi: https://doi.org/10.1080/00140130701680445
Show all 25 references
-
[9]
Natural language processing for aviation safety reports: From classification to interactive analysis,
L. Tanguy, N. Tulechki, A. Urieli, E. Hermann, and C. Raynal, “Natural language processing for aviation safety reports: From classification to interactive analysis,” Computers in Industry, vol. 78, pp. 80 –95, May 2016, doi: https://doi.org/10.1016/j.compind.2015.09.005
2016 doi
-
[10]
(2023, December)
Nanyonga, A., Wasswa, H., & Wild, G. (2023, December). Aviation Safety Enhancement via NLP & Deep Learning: Classifying Flight Phases in ATSB Safety Reports. In 2023 Global Conference on Information Technologies and Communications (GCITC) (pp. 1 -5). IEEE
2023
-
[11]
Phase of Flight Classification in Aviation Safety Using LSTM, GRU, and BiLSTM: A Case Study with ASN Dataset,
A. Nanyonga, H. Wasswa, O. Molloy, Ugur Turhan, and G. Wild, “Phase of Flight Classification in Aviation Safety Using LSTM, GRU, and BiLSTM: A Case Study with ASN Dataset,” Dec. 2023, doi: https://doi.org/10.1109/hdis60872.2023.10499521
2023
-
[12]
Text Mining for Sentiment Analysis of Twitter Data
S. Wakade, S., Shekar, C., Liszka, J. K., and Chan, C.-C., “Text Mining for Sentiment Analysis of Twitter Data.” [Online]. Available: http://jaiku.com
-
[13]
Temporal topic modeling applied to aviation safety reports: A subject matter expert review,
S. D. Robinson, “Temporal topic modeling applied to aviation safety reports: A subject matter expert review,” Safety Science, vol. 116, pp. 275–286, Jul. 2019, doi: https://doi.org/10.1016/j.ssci.2019.03.014
2019 doi
-
[14]
Latent dirichlet allocation,
D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” Journal of Machine Learning Research, vol. 3, pp. 993 –1022, Mar. 2003, doi: https://doi.org/10.5555/944919.944937
2003
-
[15]
Learning the parts of objects by non - negative matrix factorization,
D. D. Lee and H. S. Seung, “Learning the parts of objects by non - negative matrix factorization,” Nature, vol. 401, no. 6755, pp. 788 – 791, Oct. 1999, doi: https://doi.org/10.1038/44565
1999 doi
-
[16]
Using lda2vec Topic Modeling to Identify Latent Topics in Aviation Safety Reports,
Y. Luo and H. Si, “Using lda2vec Topic Modeling to Identify Latent Topics in Aviation Safety Reports,” Jun. 2019, doi: https://doi.org/10.1109/icis46139.2019.8940271
2019
-
[17]
A Survey of Topic Modeling in Text Mining,
R. Alghamdi and K. Alfalqi, “A Survey of Topic Modeling in Text Mining,” International Journal of Advanced Computer Science and Applications, vol. 6, no. 1, 2015, doi: https://doi.org/10.14569/ijacsa.2015.060121
2015
-
[18]
Comparative Analysis of Topic Modeling Techniques on ATSB Text Narratives Using Natural Language Processing,
A. Nanyonga, H. Wasswa, Ugur Turhan, K. Joiner, and G. Wild, “Comparative Analysis of Topic Modeling Techniques on ATSB Text Narratives Using Natural Language Processing,” Mar. 2024, doi: https://doi.org/10.1109/inocon60754.2024.10511951
2024
-
[19]
Document classification using nonnegative matrix factorization and underapproximation,
M. W. Berry, N. Gillis, and François Glineur, “Document classification using nonnegative matrix factorization and underapproximation,” CiteSeer X (The Pennsylvania State University), May 2009, doi: https://doi.org/10.1109/iscas.2009.5118379
2009
-
[20]
Hazard analysis: A deep learning and text mining framework for accident prevention,
B. Zhong, X. Pan, P. E. D. Love, J. Sun, and C. Tao, “Hazard analysis: A deep learning and text mining framework for accident prevention,” Advanced Engineering Informatics, vol. 46, p. 101152, Oct. 2020, doi: https://doi.org/10.1016/j.aei.2020.101152. Models Topic 1 Topic 2 To...
2020
-
[21]
Application of structural topic modelling to aviation safety data,
R. L. Rose, T. G. Puranik, D. N. Mavris, and A. H. Rao, “Application of structural topic modelling to aviation safety data,” Reliability Engineering & System Safety, vol. 224, p. 108522, Aug. 2022, doi: https://doi.org/10.1016/j.ress.2022.108522
2022
-
[22]
An introduction to latent semantic analysis,
T. K. Landauer, P. W. Foltz, and D. Laham, “An introduction to latent semantic analysis,” Discourse Processes, vol. 25, no. 2–3, pp. 259–284, Jan. 1998, doi: https://doi.org/10.1080/01638539809545028
1998 doi
- [23]
-
[24]
Topic Modeling Analysis of Aviation Accident Reports: A Comparative Study between LDA and NMF Models,
A. Nanyonga, H. Wasswa, and G. Wild, “Topic Modeling Analysis of Aviation Accident Reports: A Comparative Study between LDA and NMF Models,” arXiv (Cornell University), Dec. 2023, doi: https://doi.org/10.1109/smartgencon60755.2023.10442471
2023
-
[25]
An Efficient Topic Modeling Approach for Text Mining and Information Retrieval through K-means Clustering,
J. Rashid, S. M. A. Shah, and A. Irtaza, “An Efficient Topic Modeling Approach for Text Mining and Information Retrieval through K-means Clustering,” January 2020, vol. 39, no. 1, pp. 213–222, Jan. 2020, doi: https://doi.org/10.22581/muet1982.2001
2020
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.