Pith. sign in

REVIEW 4 major objections 4 minor 20 references

Is BERTopic Better than PLSA for Extracting Key Topics in Aviation Safety Reports?

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read BERTopic beats PLSA in coherence and interpretability on aviation safety reports.

desk verdict A useful but methodologically thin applied comparison; the paper's own cited reference undercuts its coherence metric, so it needs real revision before it can support its central claim. read the letter →

arxiv 2506.06328 v1 pith:6GVOHDYC submitted 2025-05-30 cs.IR cs.CL

classification cs.IRcs.CL
keywords BERTopicPLSAtopicmodellingaviationsafetyNTSBreportscoherenceC_vtransformerembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a modern transformer-based topic model, BERTopic, extracts better topics from aviation safety narratives than the older probabilistic model PLSA. Using more than 36,000 National Transportation Safety Board reports from 2000 to 2020, the authors find that BERTopic achieves a topic coherence score (C_v) of 0.41, compared with 0.37 for PLSA. Aviation safety experts also judged BERTopic's topics, such as 'engine failures' and 'helicopter operations,' to be clearer and more interpretable, and BERTopic processed the full dataset in about half the time. The paper's point is that, for this kind of complex, context-heavy corpus, transformer-based topic modelling gives more usable results, even though simpler probabilistic methods remain valuable in other settings.

What carries the argument

The central object is the C_v coherence score, a metric that measures how semantically similar the top words in a topic are and that the paper takes as correlated with human judgment. The argument runs through two competing pipelines: BERTopic, where transformer-based sentence embeddings are clustered with HDBSCAN and labelled via class-based TF-IDF, and PLSA, where the EM algorithm estimates topic-word and document-topic distributions from a bag-of-words matrix. The same C_v metric is applied to the topic words each method outputs, and expert review is used to check whether the numbers match human interpretation.

What would settle it

Re-run the comparison with the same number of topics for both models, have aviation experts rate or label blinded topic lists, and also compute alternative coherence scores; if experts cannot tell the models apart or the ordering flips, the superiority claim falls.

Watch

Extended reading notes

Core claim

The central claim is that BERTopic outperforms PLSA for extracting key topics from NTSB aviation safety reports. The evidence is a head-to-head comparison on the same preprocessed corpus: both models produced six topics; BERTopic's topics scored 0.41 on the C_v coherence metric versus PLSA's 0.37, and expert inspection found BERTopic's topic groupings—takeoff incidents, fuel-system failures, helicopter operations, structural failures—more coherent than PLSA's overlapping word lists. The authors attribute the gap to BERTopic's use of transformer embeddings and density-based clustering, which preserves contextual and semantic relationships, whereas PLSA's bag-of-words representation only captures statistical co-occurrence. They also report a scalability difference, with BERTopic completing the run in 1.2 hours against PLSA's 2.6 hours.

Load-bearing premise

The load-bearing premise is that the coherence score used to compare the two models, and the small 0.04 gap it shows, is a fair and meaningful measure of which model produces better topics for aviation safety.

Editorial extensions

If this is right

  • Analysts working with NTSB-style narratives can expect clearer, more actionable topics from BERTopic than from PLSA, with less manual clean-up.
  • The measured 0.04 coherence gap and the 1.2-hour versus 2.6-hour run time suggest BERTopic is the more practical option for large safety-report corpora, assuming GPU resources for embeddings are available.
  • PLSA's broader, statistically dominant topics may still suit small datasets or tasks where simple frequency patterns are the goal.
  • BERTopic's topic labels align with recognized aviation incident categories, which could feed trend monitoring and prevention efforts in safety-critical domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper fixes PLSA at six topics while BERTopic's clustering algorithm chooses clusters from the data, so the reported coherence gap could be sensitive to matching topic counts more strictly.
  • Because C_v is computed from top-word lists, part of BERTopic's advantage may come from its class-based TF-IDF producing tighter word sets rather than from the embeddings alone; swapping clustering algorithms would isolate the source.
  • The same comparison protocol could extend to other safety-critical report corpora, such as rail, maritime, or healthcare incident reports, where contextual topic labels could support similar expert review.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports a comparative study of BERTopic and PLSA for topic extraction from over 36,000 NTSB aviation safety reports (2000–2020). Both models are implemented with standard Python libraries: PLSA uses Gensim with an EM-estimated probabilistic model, and BERTopic uses transformer embeddings with HDBSCAN clustering. The evaluation compares topic coherence via the C_v metric, interpretability via unspecified manual inspection by aviation safety experts, and scalability via reported runtimes. The paper's central claim, stated in the abstract and Section IV.C, is that BERTopic outperforms PLSA with a C_v score of 0.41 versus 0.37 and superior interpretability as 'validated by aviation safety experts.' The conclusion recommends BERTopic for large, context-rich aviation safety corpora.

Significance. If the findings were robust, this paper would provide a useful practical comparison for aviation-safety analysts choosing between a classical probabilistic topic model and a transformer-based neural topic model on a real, publicly available dataset. The study has strengths: it uses a large real-world corpus, transparent preprocessing, widely used libraries, and reports concrete runtime figures (Table III). However, the central quantitative evidence is a single, small coherence difference with no uncertainty quantification, and the qualitative evidence is asserted without documentation. The manuscript's own citation [19] does not support the claim that C_v 'correlates well with human judgment'; that reference's central finding is the opposite. As a result, the paper's main conclusion is not currently supported, although it could become supportable with additional analysis and reporting.

major comments (4)
  1. [Section III.D and reference [19]] The manuscript justifies its primary metric by stating that C_v 'correlates well with human judgment [19]', but reference [19] is Hoyle et al. (2021), 'Is automated topic model evaluation broken? The incoherence of coherence', whose main empirical result is that automated coherence metrics, including C_v, do not reliably track human topic-quality judgments and can be improved artificially without improving human-rated quality. Citing this work as validation of the metric is therefore contradictory. Since the 0.41-versus-0.37 gap is the only quantitative support for the paper's central claim, the authors need to either justify why C_v is valid for this specific comparison despite [19], or replace the coherence-based claim with a validated evaluation.
  2. [Section IV.C] The headline result is reported as two point estimates, BERTopic C_v = 0.41 and PLSA C_v = 0.37, with no confidence intervals, standard deviations, multiple runs, or significance tests. A difference of 0.04 on the C_v scale is small and uncalibrated; without repeated runs or bootstrapping, it is impossible to know whether the difference is meaningful. The authors should report variability across seeds or bootstrap samples and, if feasible, a statistical test or an explicit statement that the difference is descriptive only.
  3. [Section III.E] The number of topics for PLSA was set to 6 'based on preliminary experiments', while BERTopic's minimum cluster size was set to 15 with default embeddings. No details are given about how the PLSA topic count was selected, whether the selection involved the coherence scores reported later, or how many topics BERTopic actually produced. If the topic count was tuned on the same evaluation metric, the comparison is biased; at minimum, the authors should report the topic counts for both models and describe the selection procedure, ideally with sensitivity analyses over topic counts and clustering parameters.
  4. [Section III.D and abstract] The claim of 'superior interpretability as validated by aviation safety experts' is not supported by any reported details: the number of experts, their domain background, the rating instrument or rubric, whether the experts were blind to model identity, inter-annotator agreement, or the actual ratings for each model are all absent. Without this information, the interpretability claim is anecdotal. The authors should provide a formal expert evaluation protocol and report quantitative agreement and per-model interpretability scores, or temper the claim accordingly.
minor comments (4)
  1. [Section III.C.2] The description of BERTopic omits the class-based TF-IDF step, which is central to how BERTopic constructs topic-word representations and to how the coherence metric is later applied; this should be described explicitly in the methodology.
  2. [Section IV.F and Table III] The scalability comparison is only summarized as runtimes in Table III (1.2 vs. 2.6 hours), but the text states that PLSA 'struggled with scalability' and mentions memory usage in Section III.D; no memory measurements are reported, so the scalability conclusion should be limited to runtime on the reported hardware.
  3. [Section II] Reference [18] describes a comparison on a 'stock overflow dataset'; this appears to be a typo for 'Stack Overflow', which is the standard name of the platform and the likely subject of the cited study.
  4. [Section IV.D] The text refers to 'the six topics identified by BERTopic and PLSA' without clarifying whether BERTopic's HDBSCAN clustering with a minimum cluster size of 15 actually produced exactly six topics or whether the number was enforced; this affects the interpretation of Tables I and II and of the topic-word score figures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the BERTopic-vs-PLSA comparison is an independent empirical measurement; self-citations are contextual, not load-bearing.

full rationale

The paper's central claim is an empirical comparison of two topic models on a single NTSB corpus, and no step in the derivation chain defines BERTopic's superiority in terms of PLSA's outputs or vice versa. The C_v coherence scores (0.41 vs 0.37) are reported outputs of separate model runs with stated hyperparameters, not quantities fitted to reproduce the conclusion. The expert interpretability assessment is asserted rather than documented, and citation [19] is misused when the paper says C_v 'correlates well with human judgment [19]' because Hoyle et al. is titled 'Is automated topic model evaluation broken? The incoherence of coherence'; however, a mis-citation that weakens evidence is not a circularity step. The paper cites several of the authors' own prior works ([1], [2], [4], [8], [16], [20]), but only for background, review, and previous related comparisons; none is used as the sole justification for the conclusion that BERTopic outperforms PLSA. No equation in the paper reduces to its own input, no fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The comparison is therefore not circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on hyperparameter choices (topic count, cluster size, embedding model) and on background assumptions about the validity of the coherence metric and the expert validation. No new entities are introduced.

free parameters (3)
  • number of topics for PLSA = 6
    Set to 6 based on preliminary experiments (Section III.E); this hyperparameter affects model output and coherence scores.
  • minimum cluster size for BERTopic = 15
    Set to 15 for HDBSCAN clustering (Section III.E); affects number and granularity of topics.
  • embedding model = default transformer embeddings
    BERTopic uses pre-trained transformer embeddings; the specific model is not named.
assumptions (3)
  • domain assumption C_v coherence metric correlates with human judgment
    Relies on reference [19] to justify using C_v as a proxy for topic quality; this is an unproved background assumption in this context.
  • domain assumption Preprocessed text preserves semantic content for both models
    The same preprocessing (stopword removal, lemmatization) is applied to both, but BERTopic typically works on raw text; preprocessing may affect embeddings and word distributions differently.
  • domain assumption Expert validation is reliable
    The paper claims aviation safety experts validated interpretability but gives no details; assumes expert judgment is reliable and unbiased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is BERTopic Better than PLSA for Extracting Key Topics in Aviation Safety Reports?." pith.science (2026). https://pith.science/paper/6GVOHDYC

@misc{pith2026250606328,
  author       = {Pith},
  title        = {Pith review of: Is BERTopic Better than PLSA for Extracting Key Topics in Aviation Safety Reports?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6GVOHDYC}},
  note         = {Machine review of arXiv:2506.06328}
}
read the original abstract

This study compares the effectiveness of BERTopic and Probabilistic Latent Semantic Analysis (PLSA) in extracting meaningful topics from aviation safety reports aiming to enhance the understanding of patterns in aviation incident data. Using a dataset of over 36,000 National Transportation Safety Board (NTSB) reports from 2000 to 2020, BERTopic employed transformer based embeddings and hierarchical clustering, while PLSA utilized probabilistic modelling through the Expectation-Maximization (EM) algorithm. Results showed that BERTopic outperformed PLSA in topic coherence, achieving a Cv score of 0.41 compared to PLSA 0.37, while also demonstrating superior interpretability as validated by aviation safety experts. These findings underscore the advantages of modern transformer based approaches in analyzing complex aviation datasets, paving the way for enhanced insights and informed decision-making in aviation safety. Future work will explore hybrid models, multilingual datasets, and advanced clustering techniques to further improve topic modelling in this domain.

Figures

Figures reproduced from arXiv: 2506.06328 by the authors.

Figure 1
Figure 1. Methodological framework A. Data Collection Aviation incident and accident investigation reports used in this study were exclusively sourced from the NTSB spanning the years from 2000 to 2020. The dataset, comprising a collection of more than 36,000 records in JSON format, was obtained from https://www.ntsb.gov/Pages/AviationQuery.aspx, and these reports encompass textual narratives, findings, and recommendations fr… view at source ↗
Figure 2
Figure 2. Word Cloud of Topics on BERTopic Model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Word Cloud of Topics on PLSA Model B. Hierarchical Clustering for BERTopic [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: illustrates the hierarchical clustering of topics derived from BERTopic. Using HDBSCAN, the clustering process grouped semantically similar terms, showcasing the model's ability to identify nuanced relationships between words. This clustering aligns with the domain-spe…
Figure 5
Figure 5. Figure 5: Topic word score for BERTopic Model [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Topic word score for PLSA Model E. Topic Word Distributions Tables I and II summarize the top 10 words and the associated themes (single-word topics) for the six topics generated by BERTopic and PLSA. These themes represent the overarching subject for each topic, deriv…
Figure 8
Figure 8. Figure 8: PLSA document word distribution V. CONCLUSION This study compared BERTopic and PLSA in extracting topics from aviation safety reports, focusing on topic coherence, interpretability, scalability, and topic￾specific insights. The results demonstrated that BERTopic outper…
Figure 7
Figure 7. Figure 7: BERTopic Distance Map 2) Word Distribution of Topics for PLSA [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

  1. [19]

    Is automated topic model evaluation broken? the incoherence of coherence,

    A. Hoyle, P. Goel, A. Hian-Cheong, D. Peskov, J. Boyd-Graber, and P. J. A. i. n. i. p. s. Resnik, "Is automated topic model evaluation broken? the incoherence of coherence," vol. 34, pp. 2018-2033, 2021

  2. [1]

    Classification of Operational Records in Aviation Using Deep Learning Approaches,

    A. Nanyonga and G. J. a. p. a. Wild, "Classification of Operational Records in Aviation Using Deep Learning Approaches," 2025

  3. [2]

    Applications of natural language processing in aviation safety: A review and qualitative analysis,

    A. Nanyonga, K. Joiner, U. Turhan, and G. Wild, "Applications of natural language processing in aviation safety: A review and qualitative analysis," in AIAA SCITECH 2025 Forum, 2025, p. 2153

  4. [3]

    Topic Modeling for Customer Insights: A Comparative Analysis of LDA and BERTopic in Categorizing Customer Calls,

    H. Axelborn and J. Berggren, "Topic Modeling for Customer Insights: A Comparative Analysis of LDA and BERTopic in Categorizing Customer Calls," ed, 2023

  5. [4]

    Does the Choice of Topic Modeling Technique Impact the Interpretation of Aviation Incident Reports? A Methodological Assessment,

    A. Nanyonga, K. Joiner, U. Turhan, and G. Wild, "Does the Choice of Topic Modeling Technique Impact the Interpretation of Aviation Incident Reports? A Methodological Assessment," 2025

  6. [5]

    Probabilistic latent semantic analysis,

    T. Hofmann, "Probabilistic latent semantic analysis," in UAI, 1999, vol. 99, pp. 289-296

  7. [6]

    Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling,

    Y. Mu, C. Dong, K. Bontcheva, and X. J. a. p. a. Song, "Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling," 2024

  8. [7]

    Larger residuals, less work: Active document scheduling for latent Dirichlet allocation,

    M. Wahabzada and K. Kersting, "Larger residuals, less work: Active document scheduling for latent Dirichlet allocation," in Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, 2011, pp. 475-490: Springer

Show all 20 references
  1. [8]

    Exploring Aviation Incident Narratives Using Topic Modeling and Clustering Techniques,

    A. Nanyonga, H. Wasswa, U. Turhan, K. Joiner, and G. Wild, "Exploring Aviation Incident Narratives Using Topic Modeling and Clustering Techniques," in 2024 IEEE Region 10 Symposium (TENSYMP), 2024, pp. 1-6: IEEE

  2. [9]

    Probabilistic latent semantic analysis for prediction of gene ontology annotations,

    M. Masseroli, D. Chicco, and P. Pinoli, "Probabilistic latent semantic analysis for prediction of gene ontology annotations," in The 2012 international joint conference on neural networks (IJCNN), 2012, pp. 1-8: IEEE

  3. [10]

    BERTopic: Neural topic modeling with a class-based TF-IDF procedure,

    M. J. a. p. a. Grootendorst, "BERTopic: Neural topic modeling with a class-based TF-IDF procedure," 2022

  4. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. J. a. p. a. Devlin, "Bert: Pre-training of deep bidirectional transformers for language understanding," 2018

  5. [12]

    An Analysis of Research Trends on the Metaverse Using BERTopic Modeling,

    Y. Kim and H. J. I. J. o. C. Kim, "An Analysis of Research Trends on the Metaverse Using BERTopic Modeling," vol. 19, no. 4, pp. 61-72, 2023

  6. [13]

    Leveraging state-of-the-art topic modeling for news impact analysis on financial markets: a comparative study,

    W. Chen, F. Rabhi, W. Liao, and I. J. E. Al-Qudah, "Leveraging state-of-the-art topic modeling for news impact analysis on financial markets: a comparative study," vol. 12, no. 12, p. 2605, 2023

  7. [14]

    A topic modeling comparison between lda, nmf, top2vec, and bertopic to demystify twitter posts,

    R. Egger and J. J. F. i. s. Yu, "A topic modeling comparison between lda, nmf, top2vec, and bertopic to demystify twitter posts," vol. 7, p. 886498, 2022

  8. [15]

    Discovering latent themes in aviation safety reports using text mining and network analytics,

    Y. Xing et al., "Discovering latent themes in aviation safety reports using text mining and network analytics," 2024

  9. [16]

    Impact of Dataset Size & Data Source on Aviation Safety Incident Prediction Models with Natural Language Processing,

    A. Nanyonga and G. Wild, "Impact of Dataset Size & Data Source on Aviation Safety Incident Prediction Models with Natural Language Processing," in 2023 Global Conference on Information Technologies and Communications (GCITC), 2023, pp. 1-7: IEEE

  10. [17]

    A guide to text analysis with latent semantic analysis in R with annotated code: Studying online reviews and the stack exchange community,

    D. Gefen, J. E. Endicott, J. E. Fresneda, J. Miller, and K. R. J. C. o. t. A. f. I. S. Larsen, "A guide to text analysis with latent semantic analysis in R with annotated code: Studying online reviews and the stack exchange community," vol. 41, no. 1, p. 21, 2017

  11. [18]

    Developing & Comparing Various Topic Modeling Algorithms on a Stack Overflow Dataset,

    R. IBRAIMOH, K. O. DEBRAH, and E. NWAMBUONWO, "Developing & Comparing Various Topic Modeling Algorithms on a Stack Overflow Dataset," 2024

  12. [20]

    Topic Modeling Analysis of Aviation Accident Reports: A Comparative Study between LDA and NMF Models,

    A. Nanyonga, H. Wasswa, and G. Wild, "Topic Modeling Analysis of Aviation Accident Reports: A Comparative Study between LDA and NMF Models," in 2023 3rd International Conference on Smart Generation Computing, Communication and Networking (SMART GENCON), 2023, pp. 1-2: IEEE

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.