Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Enhancing BERTopic with Intermediate Layer Representations

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read For each of three datasets, some embedding configuration beats BERTopic's default on both topic coherence and topic diversity.

desk verdict A useful, reproducible sweep of embedding layers and pooling for BERTopic, but the headline claim is weaker than it looks because the baseline may not be BERTopic's actual default and the best configuration is picked from the same metrics it is compared on. read the letter →

arxiv 2505.06696 v1 pith:XJ74FCZE submitted 2025-05-10 cs.CL

classification cs.CL
keywords BERTopictopicmodelingintermediatelayerembeddingstransformerhiddenstatespoolingstrategiescoherencediversitystopwordremoval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether BERTopic's standard document embedding—the final transformer layer with mean pooling—is actually the best choice for topic quality. It compares 18 configurations (six ways of extracting hidden states from the transformer, crossed with three pooling methods) on three datasets, measuring topic coherence with NPMI and topic diversity. The central claim is that every dataset has at least one embedding configuration that beats BERTopic's default on both metrics, so the default is not a safe optimum. The paper also finds that removing stop words improves coherence in 48 of 54 settings and diversity in 51 of 54, and that the best configuration changes with the dataset and with stop-word removal.

What carries the argument

The machinery is a configuration grid over transformer hidden states. Six retrieval strategies—embedding-layer output, last layer, second-to-last layer, sum of all layers, sum of last four layers, and concatenation of last four layers—are each combined with mean, max, and CLS pooling to produce document embeddings, which BERTopic clusters and scores. The comparison is carried by two metrics: NPMI topic coherence (from -1 to 1, measuring whether the top words of a topic co-occur more than chance) and topic diversity (the fraction of unique words across all topics). The paper also varies the input by removing stop words and by switching from static to dynamic topic modeling, giving 270 static runs and further dynamic runs.

What would settle it

Re-running the 18 configurations many times on the same datasets and computing confidence intervals for NPMI and topic diversity would settle it: if the gap between the best and default configuration is inside the interval, the claimed improvement is not distinguishable from noise.

Watch

Extended reading notes

Core claim

The central discovery is that BERTopic's default embedding choice leaves measurable performance on the table. On the 20 Newsgroups, Trump Tweets, and United Nations General Debate corpora, at least one of the 18 tested configurations yields higher NPMI topic coherence and higher topic diversity than the last-layer-with-mean-pooling default; on Trump Tweets, for example, the sum of all layers with max pooling raises coherence from 0.061 to 0.072 and diversity from 0.667 to 0.693. The best configuration is not universal: it shifts across datasets and after stop-word removal, and the embedding-layer output with mean pooling—not the last layer—gives the highest coherence on two of the three original datasets. CLS pooling is the weakest choice in most settings, max pooling most often maximizes diversity, and in the dynamic topic modeling extension the same pattern holds: some alternative configuration beats the default on each dataset.

Load-bearing premise

The paper assumes the differences between configurations are larger than the noise in the three-run averages, yet it reports no variance, confidence intervals, or significance tests.

Editorial extensions

If this is right

  • Embedding configuration should be treated as a hyperparameter of BERTopic; for any new corpus, a small search over layer and pooling choices can improve topic coherence and diversity at modest cost.
  • CLS pooling is a consistently weak choice for BERTopic topic quality; mean and max pooling are safer starting points.
  • Stop-word removal is broadly beneficial across embedding configurations: coherence improves in 48 of 54 settings and diversity in 51 of 54.
  • The same non-default advantage appears in dynamic topic modeling, where alternative configurations beat the default on both tested datasets and can surpass LDA Seq.
  • Because the best configuration is dataset-dependent, there is no single 'better default' the paper can recommend; the practical message is to explore rather than to switch universally.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper selects the best configuration using the same scores it later reports, so the size of the true advantage over the default is probably optimistic; a held-out evaluation or multiple train/test splits would be needed to turn 'a configuration exists' into a reliable selection rule.
  • If the layer effect is real, it suggests topical information is distributed across transformer layers rather than concentrated in the final layer, and the poorest CLS results hint that the [CLS] token's sentence-level summarization loses the word-level signal BERTopic's class-based TF-IDF needs.
  • A practical extension would be a cheap two-stage search: run a few aggregating configurations first (for example, sum-all-layers and concat-last-four with mean and max pooling), pick the best by coherence, and only then tune topic counts; this would keep the improvement while reducing the 1.5-hour full cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents an empirical study of BERTopic with 18 embedding configurations, formed by crossing six hidden-state retrieval strategies (last layer, embedding layer, sum of all layers, second-to-last layer, sum of last four layers, and concatenation of last four layers) with three pooling methods (Mean, Max, CLS). Evaluations are conducted on three datasets (20 Newsgroups, Trump Tweets, United Nations General Debates) using NPMI topic coherence and topic diversity, with additional experiments on stop-word removal and dynamic topic modeling. The central claim is that, for each dataset, there exists an embedding configuration that outperforms BERTopic's default setting (last layer with mean pooling), and that the best configuration varies by dataset. The paper also reports that CLS pooling performs worst, Max pooling often yields highest diversity, and stop-word removal improves metrics in most cases.

Significance. If the central claim is statistically and methodologically sound, the paper would offer practical guidance for BERTopic practitioners and contribute to the understudied question of how layer choice and pooling affect neural topic modeling. The study covers a broad configuration space, uses three diverse datasets, extends to dynamic topic modeling, and provides publicly available code and result files, which are commendable for reproducibility. However, the significance is currently limited by the absence of any variability or significance analysis, a post-hoc best-configuration selection procedure, and a potentially incorrect baseline definition, all of which directly affect the validity of the headline claim.

major comments (3)
  1. [Section 4 (Experiments)] The baseline configuration is not BERTopic's actual default. The text states: 'we used the same model as Grootendorst (2022), specifically the all-mpnet-base-v2 model from the Sentence Transformers library.' However, the BERTopic documentation and the Grootendorst (2022) paper specify all-MiniLM-L6-v2 as the default sentence-transformer model. Consequently, the 'default' row in Tables 2 and 3 (Last Layer, Mean pooling) is a non-default base encoder, and the claim that a configuration 'performs better than the default setting of BERTopic' is not established by the reported experiments. The authors should either rerun the full comparison with the actual default embedding model or explicitly re-scope the claim to improvements over their all-mpnet-base-v2 pipeline.
  2. [Section 5.1 and Tables 2-3] No measure of variability is reported. Each score is an average of 3 runs at each of 5 topic counts, but the paper provides no standard deviations, confidence intervals, or significance tests. Several of the improvements over the stated default are extremely small (e.g., Table 2, 20 Newsgroups topic coherence: 0.144 vs. 0.141; Trump Tweets topic coherence: 0.072 vs. 0.061). In addition, the best configuration is selected post hoc from the same 18 configurations using the same metrics that define the comparison, so the observed gain is partly a selection artifact. Without variance reporting, significance testing, or an out-of-sample evaluation scheme (e.g., train/validation/test splits), the central claim that 'it is possible to find' a better configuration is not supported beyond the specific runs shown.
  3. [Section 5.2 (Influence of stop words removal)] The stop-word analysis, including the statements that removal improves topic coherence in 48/54 cases and topic diversity in 51/54 cases, is presented without any statistical significance testing. The improvement counts are derived from comparisons of averaged metric values across topic counts, and no variance is given. Given the small average differences observed elsewhere in the paper, it is unclear whether these aggregate improvements are robust or simply reflect run-to-run noise. The authors should provide per-configuration variability estimates and, ideally, a paired test across the matched experimental conditions.
minor comments (5)
  1. [Section 4 (Experiments)] The statement 'For each embedding configuration, three runs were performed and the results averaged, resulting in a total of 270 individual runs' appears inconsistent with the experimental design: 18 configurations × 3 datasets × 5 topic counts × 3 runs equals 810 runs, not 270 unless the sentence refers to a single dataset. Please clarify the total.
  2. [Table 4] Several cells in Table 4 are missing spacing (e.g., '0.1740.856', '0.096 0.881 0.1740.856'), making the table difficult to read.
  3. [Figure 1] The caption 'Sum( )' in Figure 1(b) appears truncated or incomplete; the intended formula should be stated fully.
  4. [Table 1] The column header 'V ocab Size' contains an unexpected space; should be 'Vocab Size'.
  5. [Section 2 (Related work)] The related-work discussion is relevant but would benefit from explicitly connecting the cited layer-analysis findings (e.g., Ma et al., 2019; Jawahar et al., 2019) to the specific hypothesis that intermediate layers may help in topic modeling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the study is an empirical comparison, not a derivation, and the default BERTopic baseline is externally defined rather than constructed from the outcomes.

full rationale

The paper makes no formal derivation chain: it evaluates 18 embedding configurations against BERTopic's default, reporting NPMI and diversity values from experiments. The claim that 'for each dataset, it is possible to find an embedding configuration that performs better than the default setting' is an empirical existence claim supported by the measured tables, not a quantity derived from its own inputs. The default setting (last layer, mean pooling) is defined by BERTopic/Grootendorst, not by the paper's results, and the alternative configurations are distinct architectural choices. Selecting the best configuration on the same metric used for comparison is a statistical selection issue that can inflate apparent gains, but it does not make the comparison equivalent to its inputs by construction; the maximum of 18 measured values need not exceed the default unless the data show it. No load-bearing self-citations, imported uniqueness theorems, ansatz smuggling via citation, or renaming of known results appear. The paper's use of Grootendorst (2022) is for dataset/pipeline comparability, and the LDA/NMF comparison values are explicitly attributed to that prior work rather than generated inside the paper. Any concerns about the baseline model choice or missing variance measures are correctness or robustness concerns, not circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities and no continuous free parameters. Its implicit assumptions are about evaluation validity: coherence and diversity as quality proxies, dataset representativeness, default BERTopic pipeline fairness, and the sufficiency of the averaging scheme. The post-hoc selection of the best configuration is the most consequential choice, acting as a discrete fitted parameter.

free parameters (1)
  • per-dataset best configuration = selected post hoc from Tables 2-3
    The claim 'there exists a configuration that performs better than default' relies on picking the best of 18 configurations per dataset using the same metrics that define success; this discrete selection is a post-hoc fit.
assumptions (5)
  • domain assumption NPMI and topic diversity are valid proxies for topic quality.
    Used as the sole quantitative evaluation in Section 4 and Tables 2-4; the paper itself notes these metrics may not fully capture quality.
  • domain assumption The three datasets are representative enough to support a general practical claim.
    Section 3 selects Trump Tweets, UN debates, and 20 Newsgroups; whether these cover the space of corpora is assumed.
  • domain assumption BERTopic's default pipeline is a fair baseline for all configurations.
    The evaluation pipeline from Grootendorst (2022) is reused without re-tuning; this assumes default hyperparameters are fair across embeddings.
  • domain assumption all-mpnet-base-v2 is a suitable base model for the comparison.
    A single Sentence Transformers model is used; results may not transfer to other transformer models.
  • domain assumption Run-to-run and topic-count averaging is sufficient to rank configurations.
    Section 4 averages 3 runs at 5 topic counts; no standard deviations are reported, so the ranking stability is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing BERTopic with Intermediate Layer Representations." pith.science (2026). https://pith.science/paper/XJ74FCZE

@misc{pith2026250506696,
  author       = {Pith},
  title        = {Pith review of: Enhancing BERTopic with Intermediate Layer Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJ74FCZE}},
  note         = {Machine review of arXiv:2505.06696}
}
read the original abstract

BERTopic is a topic modeling algorithm that leverages transformer-based embeddings to create dense clusters, enabling the estimation of topic structures and the extraction of valuable insights from a corpus of documents. This approach allows users to efficiently process large-scale text data and gain meaningful insights into its structure. While BERTopic is a powerful tool, embedding preparation can vary, including extracting representations from intermediate model layers and applying transformations to these embeddings. In this study, we evaluate 18 different embedding representations and present findings based on experiments conducted on three diverse datasets. To assess the algorithm's performance, we report topic coherence and topic diversity metrics across all experiments. Our results demonstrate that, for each dataset, it is possible to find an embedding configuration that performs better than the default setting of BERTopic. Additionally, we investigate the influence of stop words on different embedding configurations.

Figures

Figures reproduced from arXiv: 2505.06696 by the authors.

Figure 1
Figure 1. Visualization of embedding configurations [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Topic coherence and topic diversity scores on the United Nations dataset (without stop words) across [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the top eight topics with words representing them for the worst (Embedding Layer with [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Frequency of selected topics estimated from Trump Tweets dataset over the years. Topics have been [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 13 canonical work pages

  1. [1]

    Saqib Aziz, Michael Dowling, Helmi Hammami, and Anke Piepenbrink. 2022. https://doi.org/10.1111/eufm.12326 Machine learning in finance: A topic modeling approach . European Financial Management, 28(3):744--770

  2. [2]

    Steven Bird. 2006. https://doi.org/10.3115/1225403.1225421 NLTK : The N atural L anguage T oolkit . In Proceedings of the COLING / ACL 2006 Interactive Presentation Sessions , pages 69--72, Sydney, Australia. Association for Computational Linguistics

  3. [3]

    David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. https://doi.org/http://dx.doi.org/10.1162/jmlr.2003.3.4-5.993 Latent dirichlet allocation . Journal of machine Learning research, 3(Jan):993--1022

  4. [4]

    Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL, 30:31--40

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

  6. [6]

    Dieng, Francisco J

    Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. https://doi.org/10.1162/tacl_a_00325 Topic modeling in embedding spaces . Transactions of the Association for Computational Linguistics, 8:439--453

  7. [7]

    Kawin Ethayarajh. 2019. https://doi.org/10.18653/v1/D19-1006 How contextual are contextualized word representations? C omparing the geometry of BERT , ELM o, and GPT -2 embeddings . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN...

  8. [8]

    Max Falkenberg, Alessandro Galeazzi, Maddalena Torricelli, Niccol \`o Di Marco, Francesca Larosa, Madalina Sas, Amin Mekacher, Warren Pearce, Fabiana Zollo, Walter Quattrociocchi, and Andrea Baronchelli. 2022. https://doi.org/10.1038/s41558-022-01527-x Growing polarization around climate change on social media . Nature Climate Change, 12(12):1114--1121

Show all 35 references
  1. [9]

    Yansong Feng and Mirella Lapata. 2010. https://aclanthology.org/N10-1125/ Topic models for image annotation and text illustration . In Human Language Technologies: The 2010 Annual Conference of the North A merican Chapter of the Association for Computational Linguistics , page...

  2. [10]

    Yuan Gong and Christian Poellabauer. 2017. https://doi.org/10.1145/3133944.3133945 Topic modeling based multi-modal depression detection . AVEC '17, page 69–76, New York, NY, USA. Association for Computing Machinery

  3. [11]

    Derek Greene and P\' a draig Cunningham. 2006. https://doi.org/10.1145/1143844.1143892 Practical solutions to the problem of diagonal dominance in kernel document clustering . In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 377–384, New ...

  4. [12]

    Maarten Grootendorst. 2022. https://arxiv.org/abs/2203.05794 Bertopic: Neural topic modeling with a class-based tf-idf procedure . Preprint, arXiv:2203.05794

  5. [13]

    Stefano Guizzardi, Maria Teresa Colangelo, Prisco Mirandola, and Carlo Galli. 2023. https://doi.org/10.2217/rme-2023-0096 Modeling new trends in bone regeneration, using the bertopic approach . Regenerative Medicine, 18(9):719--734

  6. [14]

    Timothy Hospedales, Shaogang Gong, and Tao Xiang. 2012. https://doi.org/10.1007/s11263-011-0510-7 Video behaviour mining using a dynamic topic model . International Journal of Computer Vision, 98(3):303--323

  7. [15]

    Ganesh Jawahar, Beno \^i t Sagot, and Djam \'e Seddah. 2019. https://doi.org/10.18653/v1/P19-1356 What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651--3657, Florence, Italy...

  8. [16]

    Eunji Jeon, Naeun Yoon, and So Young Sohn. 2023. https://doi.org/10.1016/j.techfore.2022.122130 Exploring new digital therapeutics technologies for psychiatric disorders using bertopic and patentsberta . Technological Forecasting and Social Change, 186:122130

  9. [17]

    Lingyao Li, Zihui Ma, Lizhou Fan, Sanggyu Lee, Huizi Yu, and Libby Hemphill. 2024. https://doi.org/10.1007/s10639-023-12256-9 Chatgpt in education: a discourse analysis of worries and concerns on social media . Education and Information Technologies, 29(9):10729--10762

  10. [18]

    Xiaofei Ma, Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang. 2019. https://arxiv.org/abs/1910.07973 Universal text representation from bert: An empirical study . Preprint, arXiv:1910.07973

  11. [19]

    Bayode Ogunleye, Tonderai Maswera, Laurence Hirsch, Jotham Gaudoin, and Teresa Brunsdon. 2023. https://doi.org/10.3390/app13020797 Comparison of topic modelling approaches in the banking context . Applied Sciences, 13(2)

  12. [20]

    Pentti Paatero and Unto Tapper. 1994. https://api.semanticscholar.org/CorpusID:121690164 Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† . Environmetrics, 5:111--126

  13. [21]

    Giovanni Puccetti, Alessio Miaschi, and Felice Dell ' Orletta. 2021. https://doi.org/10.18653/v1/2021.deelio-1.6 How do BERT embeddings organize linguistic knowledge? In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration ...

  14. [22]

    Alessandro Raganato and J \"o rg Tiedemann. 2018. https://doi.org/10.18653/v1/W18-5431 An analysis of encoder representations in transformer-based machine translation . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : Analyzing and Interpreting Neural Networks for NLP...

  15. [23]

    Raju, Vasudeva, Bharath Kumar Bolla, Deepak Kumar Nayak, and Jyothsna Kh

    S. Raju, Vasudeva, Bharath Kumar Bolla, Deepak Kumar Nayak, and Jyothsna Kh. 2022. https://doi.org/10.1109/I2CT54291.2022.9824873 Topic modelling on consumer financial protection bureau data: An approach using bert based embeddings . In 2022 IEEE 7th International conference f...

  16. [24]

    Steven Ramondt, Peter Kerkhof, and Eva-Maria Merz. 2022. https://doi.org/10.1016/j.tmrv.2021.10.001 Blood donation narratives on social media: A topic modeling study . Transfusion Medicine Reviews, 36(1):58--65

  17. [25]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

  18. [26]

    Alexandra Schofield, M a ns Magnusson, and David Mimno. 2017. https://aclanthology.org/E17-2069/ Pulling out the stops: Rethinking stopword removal for topic models . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistic...

  19. [27]

    Jennifer Sleeman, Tim Finin, and Milton Halem. 2021. https://doi.org/10.3389/fdata.2021.601529 Understanding cybersecurity threat trends through dynamic topic modeling . Frontiers in Big Data, 4

  20. [28]

    Robert Tibshirani. 1996. http://www.jstor.org/stable/2346178 Regression shrinkage and selection via the lasso . Journal of the Royal Statistical Society. Series B (Methodological), 58(1):267--288

  21. [29]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

  22. [30]

    Yueshen Xu, Yuyu Yin, and Jianwei Yin. 2017. https://doi.org/10.1016/j.engappai.2017.04.009 Tackling topic general words in topic modeling . Engineering Applications of Artificial Intelligence, 62:124--133

  23. [31]

    Liang Yao, Yin Zhang, Baogang Wei, Wenjin Zhang, and Zhe Jin. 2018. https://doi.org/10.1109/TKDE.2017.2787158 A topic modeling approach for traditional chinese medicine prescriptions . IEEE Transactions on Knowledge and Data Engineering, 30(6):1007--1021

  24. [32]

    Hao Zhang, Gunhee Kim, and Eric P. Xing. 2015. https://doi.org/10.1145/2783258.2783293 Dynamic topic modeling for monitoring market competition from online text and image data . In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mini...

  25. [33]

    Elaine Zosa and Mark Granroth-Wilding. 2019. https://doi.org/10.26615/978-954-452-056-4_159 Multilingual dynamic topic model . In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1388--1396, Varna, Bulgaria. INCOMA Ltd

  26. [34]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.