REVIEW 4 major objections 4 minor 20 references
Is BERTopic Better than PLSA for Extracting Key Topics in Aviation Safety Reports?
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read BERTopic beats PLSA in coherence and interpretability on aviation safety reports.
desk verdict A useful but methodologically thin applied comparison; the paper's own cited reference undercuts its coherence metric, so it needs real revision before it can support its central claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the C_v coherence score, a metric that measures how semantically similar the top words in a topic are and that the paper takes as correlated with human judgment. The argument runs through two competing pipelines: BERTopic, where transformer-based sentence embeddings are clustered with HDBSCAN and labelled via class-based TF-IDF, and PLSA, where the EM algorithm estimates topic-word and document-topic distributions from a bag-of-words matrix. The same C_v metric is applied to the topic words each method outputs, and expert review is used to check whether the numbers match human interpretation.
What would settle it
Re-run the comparison with the same number of topics for both models, have aviation experts rate or label blinded topic lists, and also compute alternative coherence scores; if experts cannot tell the models apart or the ordering flips, the superiority claim falls.
Extended reading notes
Core claim
The central claim is that BERTopic outperforms PLSA for extracting key topics from NTSB aviation safety reports. The evidence is a head-to-head comparison on the same preprocessed corpus: both models produced six topics; BERTopic's topics scored 0.41 on the C_v coherence metric versus PLSA's 0.37, and expert inspection found BERTopic's topic groupings—takeoff incidents, fuel-system failures, helicopter operations, structural failures—more coherent than PLSA's overlapping word lists. The authors attribute the gap to BERTopic's use of transformer embeddings and density-based clustering, which preserves contextual and semantic relationships, whereas PLSA's bag-of-words representation only captures statistical co-occurrence. They also report a scalability difference, with BERTopic completing the run in 1.2 hours against PLSA's 2.6 hours.
Load-bearing premise
The load-bearing premise is that the coherence score used to compare the two models, and the small 0.04 gap it shows, is a fair and meaningful measure of which model produces better topics for aviation safety.
Editorial extensions
If this is right
- Analysts working with NTSB-style narratives can expect clearer, more actionable topics from BERTopic than from PLSA, with less manual clean-up.
- The measured 0.04 coherence gap and the 1.2-hour versus 2.6-hour run time suggest BERTopic is the more practical option for large safety-report corpora, assuming GPU resources for embeddings are available.
- PLSA's broader, statistically dominant topics may still suit small datasets or tasks where simple frequency patterns are the goal.
- BERTopic's topic labels align with recognized aviation incident categories, which could feed trend monitoring and prevention efforts in safety-critical domains.
Reading between the lines
- The paper fixes PLSA at six topics while BERTopic's clustering algorithm chooses clusters from the data, so the reported coherence gap could be sensitive to matching topic counts more strictly.
- Because C_v is computed from top-word lists, part of BERTopic's advantage may come from its class-based TF-IDF producing tighter word sets rather than from the embeddings alone; swapping clustering algorithms would isolate the source.
- The same comparison protocol could extend to other safety-critical report corpora, such as rail, maritime, or healthcare incident reports, where contextual topic labels could support similar expert review.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a comparative study of BERTopic and PLSA for topic extraction from over 36,000 NTSB aviation safety reports (2000–2020). Both models are implemented with standard Python libraries: PLSA uses Gensim with an EM-estimated probabilistic model, and BERTopic uses transformer embeddings with HDBSCAN clustering. The evaluation compares topic coherence via the C_v metric, interpretability via unspecified manual inspection by aviation safety experts, and scalability via reported runtimes. The paper's central claim, stated in the abstract and Section IV.C, is that BERTopic outperforms PLSA with a C_v score of 0.41 versus 0.37 and superior interpretability as 'validated by aviation safety experts.' The conclusion recommends BERTopic for large, context-rich aviation safety corpora.
Significance. If the findings were robust, this paper would provide a useful practical comparison for aviation-safety analysts choosing between a classical probabilistic topic model and a transformer-based neural topic model on a real, publicly available dataset. The study has strengths: it uses a large real-world corpus, transparent preprocessing, widely used libraries, and reports concrete runtime figures (Table III). However, the central quantitative evidence is a single, small coherence difference with no uncertainty quantification, and the qualitative evidence is asserted without documentation. The manuscript's own citation [19] does not support the claim that C_v 'correlates well with human judgment'; that reference's central finding is the opposite. As a result, the paper's main conclusion is not currently supported, although it could become supportable with additional analysis and reporting.
major comments (4)
- [Section III.D and reference [19]] The manuscript justifies its primary metric by stating that C_v 'correlates well with human judgment [19]', but reference [19] is Hoyle et al. (2021), 'Is automated topic model evaluation broken? The incoherence of coherence', whose main empirical result is that automated coherence metrics, including C_v, do not reliably track human topic-quality judgments and can be improved artificially without improving human-rated quality. Citing this work as validation of the metric is therefore contradictory. Since the 0.41-versus-0.37 gap is the only quantitative support for the paper's central claim, the authors need to either justify why C_v is valid for this specific comparison despite [19], or replace the coherence-based claim with a validated evaluation.
- [Section IV.C] The headline result is reported as two point estimates, BERTopic C_v = 0.41 and PLSA C_v = 0.37, with no confidence intervals, standard deviations, multiple runs, or significance tests. A difference of 0.04 on the C_v scale is small and uncalibrated; without repeated runs or bootstrapping, it is impossible to know whether the difference is meaningful. The authors should report variability across seeds or bootstrap samples and, if feasible, a statistical test or an explicit statement that the difference is descriptive only.
- [Section III.E] The number of topics for PLSA was set to 6 'based on preliminary experiments', while BERTopic's minimum cluster size was set to 15 with default embeddings. No details are given about how the PLSA topic count was selected, whether the selection involved the coherence scores reported later, or how many topics BERTopic actually produced. If the topic count was tuned on the same evaluation metric, the comparison is biased; at minimum, the authors should report the topic counts for both models and describe the selection procedure, ideally with sensitivity analyses over topic counts and clustering parameters.
- [Section III.D and abstract] The claim of 'superior interpretability as validated by aviation safety experts' is not supported by any reported details: the number of experts, their domain background, the rating instrument or rubric, whether the experts were blind to model identity, inter-annotator agreement, or the actual ratings for each model are all absent. Without this information, the interpretability claim is anecdotal. The authors should provide a formal expert evaluation protocol and report quantitative agreement and per-model interpretability scores, or temper the claim accordingly.
minor comments (4)
- [Section III.C.2] The description of BERTopic omits the class-based TF-IDF step, which is central to how BERTopic constructs topic-word representations and to how the coherence metric is later applied; this should be described explicitly in the methodology.
- [Section IV.F and Table III] The scalability comparison is only summarized as runtimes in Table III (1.2 vs. 2.6 hours), but the text states that PLSA 'struggled with scalability' and mentions memory usage in Section III.D; no memory measurements are reported, so the scalability conclusion should be limited to runtime on the reported hardware.
- [Section II] Reference [18] describes a comparison on a 'stock overflow dataset'; this appears to be a typo for 'Stack Overflow', which is the standard name of the platform and the likely subject of the cited study.
- [Section IV.D] The text refers to 'the six topics identified by BERTopic and PLSA' without clarifying whether BERTopic's HDBSCAN clustering with a minimum cluster size of 15 actually produced exactly six topics or whether the number was enforced; this affects the interpretation of Tables I and II and of the topic-word score figures.
Circularity Check
No circularity: the BERTopic-vs-PLSA comparison is an independent empirical measurement; self-citations are contextual, not load-bearing.
full rationale
The paper's central claim is an empirical comparison of two topic models on a single NTSB corpus, and no step in the derivation chain defines BERTopic's superiority in terms of PLSA's outputs or vice versa. The C_v coherence scores (0.41 vs 0.37) are reported outputs of separate model runs with stated hyperparameters, not quantities fitted to reproduce the conclusion. The expert interpretability assessment is asserted rather than documented, and citation [19] is misused when the paper says C_v 'correlates well with human judgment [19]' because Hoyle et al. is titled 'Is automated topic model evaluation broken? The incoherence of coherence'; however, a mis-citation that weakens evidence is not a circularity step. The paper cites several of the authors' own prior works ([1], [2], [4], [8], [16], [20]), but only for background, review, and previous related comparisons; none is used as the sole justification for the conclusion that BERTopic outperforms PLSA. No equation in the paper reduces to its own input, no fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The comparison is therefore not circular.
Assumptions & free parameters
free parameters (3)
- number of topics for PLSA =
6
- minimum cluster size for BERTopic =
15
- embedding model =
default transformer embeddings
assumptions (3)
- domain assumption C_v coherence metric correlates with human judgment
- domain assumption Preprocessed text preserves semantic content for both models
- domain assumption Expert validation is reliable
Cite this review
Pith. "Pith review of Is BERTopic Better than PLSA for Extracting Key Topics in Aviation Safety Reports?." pith.science (2026). https://pith.science/paper/6GVOHDYC
@misc{pith2026250606328,
author = {Pith},
title = {Pith review of: Is BERTopic Better than PLSA for Extracting Key Topics in Aviation Safety Reports?},
year = {2026},
howpublished = {\url{https://pith.science/paper/6GVOHDYC}},
note = {Machine review of arXiv:2506.06328}
}
read the original abstract
This study compares the effectiveness of BERTopic and Probabilistic Latent Semantic Analysis (PLSA) in extracting meaningful topics from aviation safety reports aiming to enhance the understanding of patterns in aviation incident data. Using a dataset of over 36,000 National Transportation Safety Board (NTSB) reports from 2000 to 2020, BERTopic employed transformer based embeddings and hierarchical clustering, while PLSA utilized probabilistic modelling through the Expectation-Maximization (EM) algorithm. Results showed that BERTopic outperformed PLSA in topic coherence, achieving a Cv score of 0.41 compared to PLSA 0.37, while also demonstrating superior interpretability as validated by aviation safety experts. These findings underscore the advantages of modern transformer based approaches in analyzing complex aviation datasets, paving the way for enhanced insights and informed decision-making in aviation safety. Future work will explore hybrid models, multilingual datasets, and advanced clustering techniques to further improve topic modelling in this domain.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[19]
Is automated topic model evaluation broken? the incoherence of coherence,
A. Hoyle, P. Goel, A. Hian-Cheong, D. Peskov, J. Boyd-Graber, and P. J. A. i. n. i. p. s. Resnik, "Is automated topic model evaluation broken? the incoherence of coherence," vol. 34, pp. 2018-2033, 2021
work page 2018
-
[1]
Classification of Operational Records in Aviation Using Deep Learning Approaches,
A. Nanyonga and G. J. a. p. a. Wild, "Classification of Operational Records in Aviation Using Deep Learning Approaches," 2025
work page 2025
-
[2]
Applications of natural language processing in aviation safety: A review and qualitative analysis,
A. Nanyonga, K. Joiner, U. Turhan, and G. Wild, "Applications of natural language processing in aviation safety: A review and qualitative analysis," in AIAA SCITECH 2025 Forum, 2025, p. 2153
work page 2025
-
[3]
H. Axelborn and J. Berggren, "Topic Modeling for Customer Insights: A Comparative Analysis of LDA and BERTopic in Categorizing Customer Calls," ed, 2023
work page 2023
-
[4]
A. Nanyonga, K. Joiner, U. Turhan, and G. Wild, "Does the Choice of Topic Modeling Technique Impact the Interpretation of Aviation Incident Reports? A Methodological Assessment," 2025
work page 2025
-
[5]
Probabilistic latent semantic analysis,
T. Hofmann, "Probabilistic latent semantic analysis," in UAI, 1999, vol. 99, pp. 289-296
work page 1999
-
[6]
Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling,
Y. Mu, C. Dong, K. Bontcheva, and X. J. a. p. a. Song, "Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling," 2024
work page 2024
-
[7]
Larger residuals, less work: Active document scheduling for latent Dirichlet allocation,
M. Wahabzada and K. Kersting, "Larger residuals, less work: Active document scheduling for latent Dirichlet allocation," in Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, 2011, pp. 475-490: Springer
work page 2011
Show all 20 references
-
[8]
Exploring Aviation Incident Narratives Using Topic Modeling and Clustering Techniques,
A. Nanyonga, H. Wasswa, U. Turhan, K. Joiner, and G. Wild, "Exploring Aviation Incident Narratives Using Topic Modeling and Clustering Techniques," in 2024 IEEE Region 10 Symposium (TENSYMP), 2024, pp. 1-6: IEEE
2024
-
[9]
Probabilistic latent semantic analysis for prediction of gene ontology annotations,
M. Masseroli, D. Chicco, and P. Pinoli, "Probabilistic latent semantic analysis for prediction of gene ontology annotations," in The 2012 international joint conference on neural networks (IJCNN), 2012, pp. 1-8: IEEE
2012
-
[10]
BERTopic: Neural topic modeling with a class-based TF-IDF procedure,
M. J. a. p. a. Grootendorst, "BERTopic: Neural topic modeling with a class-based TF-IDF procedure," 2022
2022
-
[11]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. J. a. p. a. Devlin, "Bert: Pre-training of deep bidirectional transformers for language understanding," 2018
2018
-
[12]
An Analysis of Research Trends on the Metaverse Using BERTopic Modeling,
Y. Kim and H. J. I. J. o. C. Kim, "An Analysis of Research Trends on the Metaverse Using BERTopic Modeling," vol. 19, no. 4, pp. 61-72, 2023
2023
-
[13]
Leveraging state-of-the-art topic modeling for news impact analysis on financial markets: a comparative study,
W. Chen, F. Rabhi, W. Liao, and I. J. E. Al-Qudah, "Leveraging state-of-the-art topic modeling for news impact analysis on financial markets: a comparative study," vol. 12, no. 12, p. 2605, 2023
2023
-
[14]
A topic modeling comparison between lda, nmf, top2vec, and bertopic to demystify twitter posts,
R. Egger and J. J. F. i. s. Yu, "A topic modeling comparison between lda, nmf, top2vec, and bertopic to demystify twitter posts," vol. 7, p. 886498, 2022
2022
-
[15]
Discovering latent themes in aviation safety reports using text mining and network analytics,
Y. Xing et al., "Discovering latent themes in aviation safety reports using text mining and network analytics," 2024
2024
-
[16]
Impact of Dataset Size & Data Source on Aviation Safety Incident Prediction Models with Natural Language Processing,
A. Nanyonga and G. Wild, "Impact of Dataset Size & Data Source on Aviation Safety Incident Prediction Models with Natural Language Processing," in 2023 Global Conference on Information Technologies and Communications (GCITC), 2023, pp. 1-7: IEEE
2023
-
[17]
A guide to text analysis with latent semantic analysis in R with annotated code: Studying online reviews and the stack exchange community,
D. Gefen, J. E. Endicott, J. E. Fresneda, J. Miller, and K. R. J. C. o. t. A. f. I. S. Larsen, "A guide to text analysis with latent semantic analysis in R with annotated code: Studying online reviews and the stack exchange community," vol. 41, no. 1, p. 21, 2017
2017
-
[18]
Developing & Comparing Various Topic Modeling Algorithms on a Stack Overflow Dataset,
R. IBRAIMOH, K. O. DEBRAH, and E. NWAMBUONWO, "Developing & Comparing Various Topic Modeling Algorithms on a Stack Overflow Dataset," 2024
2024
-
[20]
Topic Modeling Analysis of Aviation Accident Reports: A Comparative Study between LDA and NMF Models,
A. Nanyonga, H. Wasswa, and G. Wild, "Topic Modeling Analysis of Aviation Accident Reports: A Comparative Study between LDA and NMF Models," in 2023 3rd International Conference on Smart Generation Computing, Communication and Networking (SMART GENCON), 2023, pp. 1-2: IEEE
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.