{"id":"fc9d3189-684d-4f61-97af-60806a711d5b","arxiv_id":"2505.06696","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across three datasets, no single embedding configuration beats BERTopic's default on all metrics, but every dataset has some intermediate-layer or pooling choice that scores higher.","lead":"BERTopic is a popular topic modeling tool that turns documents into clusters using sentence embeddings. This paper tried 18 different ways of building those embeddings, by taking representations from different transformer layers and pooling them differently, and found that on each of three datasets some non-default setting scored higher on topic coherence or diversity than the default.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4's baseline may not be BERTopic's actual default: it uses all-mpnet-base-v2, whereas BERTopic and Grootendorst (2022) default to all-MiniLM-L6-v2; if correct, the headline claim compares against a non-default model.","rationale":"I focused on the identity of the baseline because it is prior to the statistical concern. If the paper is not comparing against BERTopic's default embedding model, the claim 'better than the default setting of BERTopic' is not established even in the ideal noiseless case. The statement in Section 4 that Grootendorst used all-mpnet-base-v2 is checkable and appears inconsistent with BERTopic's documented default of all-MiniLM-L6-v2. The reader's concern about missing variance and selection from the same metric remains valid, especially given small delta values such as 0.003 on 20 Newsgroups, but it would not fix a baseline mismatch. I do not see fabrication or internal inconsistency; the layer/pooling exploration is useful as a study of a specific base model. My recommendation is to retain the conditional verdict, pending verification of the baseline and, ideally, variance reporting.","tokens_in":10537,"tokens_out":11300,"duration_ms":109978,"concrete_test":"Check the default embedding model in the BERTopic repository and in Grootendorst (2022), e.g., by inspecting the BERTopic source code for the default SentenceTransformer model and the cited paper's experimental setup. If the default is all-MiniLM-L6-v2, rerun the Section 5.1 sweep with all-MiniLM-L6-v2 as the base model for all 18 configurations and compare against (all-MiniLM-L6-v2, last layer, mean pooling) on the same three datasets. If no configuration beats that actual default on both NPMI and diversity for every dataset, the central claim fails. The same rerun should report run-to-run variance or confidence intervals to settle the reader's noise concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that an embedding configuration can beat the default BERTopic setting. The paper's comparison baseline is stated in Section 4: 'we used the same model as Grootendorst (2022), specifically the all-mpnet-base-v2 model from the Sentence Transformers library.' Publicly available BERTopic documentation and the Grootendorst (2022) paper use all-MiniLM-L6-v2 as the default sentence-transformer model. If that is accurate, all 18 configurations, including the 'default' (last layer, mean pooling), are built on a non-default base encoder. The experiments therefore establish improvements only within an all-mpnet-base-v2 pipeline, not over BERTopic as actually shipped. This is more load-bearing than the variance issue, because even with unlimited runs the comparison would still be against the wrong baseline. The variance concern is still real: no standard deviations are reported, and Table 2 shows a best-coherence gain of only 0.003 on 20 Newsgroups, which may be selection noise from choosing the best of 18 configurations on the same metric used for evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of BERTopic with 18 embedding configurations, formed by crossing six hidden-state retrieval strategies (last layer, embedding layer, sum of all layers, second-to-last layer, sum of last four layers, and concatenation of last four layers) with three pooling methods (Mean, Max, CLS). Evaluations are conducted on three datasets (20 Newsgroups, Trump Tweets, United Nations General Debates) using NPMI topic coherence and topic diversity, with additional experiments on stop-word removal and dynamic topic modeling. The central claim is that, for each dataset, there exists an embedding configuration that outperforms BERTopic's default setting (last layer with mean pooling), and that the best configuration varies by dataset. The paper also reports that CLS pooling performs worst, Max pooling often yields highest diversity, and stop-word removal improves metrics in most cases.","tokens_in":10725,"tokens_out":3683,"duration_ms":35970,"significance":"If the central claim is statistically and methodologically sound, the paper would offer practical guidance for BERTopic practitioners and contribute to the understudied question of how layer choice and pooling affect neural topic modeling. The study covers a broad configuration space, uses three diverse datasets, extends to dynamic topic modeling, and provides publicly available code and result files, which are commendable for reproducibility. However, the significance is currently limited by the absence of any variability or significance analysis, a post-hoc best-configuration selection procedure, and a potentially incorrect baseline definition, all of which directly affect the validity of the headline claim.","major_comments":[{"comment":"The baseline configuration is not BERTopic's actual default. The text states: 'we used the same model as Grootendorst (2022), specifically the all-mpnet-base-v2 model from the Sentence Transformers library.' However, the BERTopic documentation and the Grootendorst (2022) paper specify all-MiniLM-L6-v2 as the default sentence-transformer model. Consequently, the 'default' row in Tables 2 and 3 (Last Layer, Mean pooling) is a non-default base encoder, and the claim that a configuration 'performs better than the default setting of BERTopic' is not established by the reported experiments. The authors should either rerun the full comparison with the actual default embedding model or explicitly re-scope the claim to improvements over their all-mpnet-base-v2 pipeline.","section":"Section 4 (Experiments)"},{"comment":"No measure of variability is reported. Each score is an average of 3 runs at each of 5 topic counts, but the paper provides no standard deviations, confidence intervals, or significance tests. Several of the improvements over the stated default are extremely small (e.g., Table 2, 20 Newsgroups topic coherence: 0.144 vs. 0.141; Trump Tweets topic coherence: 0.072 vs. 0.061). In addition, the best configuration is selected post hoc from the same 18 configurations using the same metrics that define the comparison, so the observed gain is partly a selection artifact. Without variance reporting, significance testing, or an out-of-sample evaluation scheme (e.g., train/validation/test splits), the central claim that 'it is possible to find' a better configuration is not supported beyond the specific runs shown.","section":"Section 5.1 and Tables 2-3"},{"comment":"The stop-word analysis, including the statements that removal improves topic coherence in 48/54 cases and topic diversity in 51/54 cases, is presented without any statistical significance testing. The improvement counts are derived from comparisons of averaged metric values across topic counts, and no variance is given. Given the small average differences observed elsewhere in the paper, it is unclear whether these aggregate improvements are robust or simply reflect run-to-run noise. The authors should provide per-configuration variability estimates and, ideally, a paired test across the matched experimental conditions.","section":"Section 5.2 (Influence of stop words removal)"}],"minor_comments":[{"comment":"The statement 'For each embedding configuration, three runs were performed and the results averaged, resulting in a total of 270 individual runs' appears inconsistent with the experimental design: 18 configurations × 3 datasets × 5 topic counts × 3 runs equals 810 runs, not 270 unless the sentence refers to a single dataset. Please clarify the total.","section":"Section 4 (Experiments)"},{"comment":"Several cells in Table 4 are missing spacing (e.g., '0.1740.856', '0.096 0.881 0.1740.856'), making the table difficult to read.","section":"Table 4"},{"comment":"The caption 'Sum( )' in Figure 1(b) appears truncated or incomplete; the intended formula should be stated fully.","section":"Figure 1"},{"comment":"The column header 'V ocab Size' contains an unexpected space; should be 'Vocab Size'.","section":"Table 1"},{"comment":"The related-work discussion is relevant but would benefit from explicitly connecting the cited layer-analysis findings (e.g., Ma et al., 2019; Jawahar et al., 2019) to the specific hypothesis that intermediate layers may help in topic modeling.","section":"Section 2 (Related work)"}],"recommendation":"major_revision","confidential_remarks":"The baseline model issue is the most serious concern. The authors state they used 'the same model as Grootendorst (2022), specifically the all-mpnet-base-v2 model,' but the BERTopic paper and official documentation use all-MiniLM-L6-v2 as the default. This could be an innocent mistake inherited from the evaluation pipeline they used, but it directly undermines the headline claim. If the authors can rerun the comparison with the actual default model, or convincingly show that all-mpnet-base-v2 was the default at the time of the cited work, the paper would be salvageable. The lack of any variance analysis is also a concern for a journal publication, especially given the small metric differences. I recommend major revision over rejection because both issues are addressable with additional experiments or re-analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the systematic sweep: 18 layer-retrieval and pooling configurations inside BERTopic, on three datasets, with code and raw results public. Prior work looked at layer-wise embeddings on probing tasks, not inside BERTopic. That is a real gap, and this paper fills it carefully. The tabular results are clear, the stop-word analysis is a reasonable add-on, and the limitations section is honest about metrics not capturing topic quality. Credit where due: this is a reproducible empirical study, and the authors do not oversell the metrics themselves.\n\nThe soft spots are substantial. First, the stress-test note is right: the paper says it uses the same model as Grootendorst (2022), all-mpnet-base-v2, but BERTopic's documented default is all-MiniLM-L6-v2. That makes the 'better than the default setting of BERTopic' claim a comparison against a strawman default. The internal comparison among configurations on the same base model is still informative, but the headline overreaches. Second, there is no variance reporting or significance testing. Three runs per topic count, five counts, averaged, with no standard deviation. Some gaps are tiny — 0.003 on 20 Newsgroups coherence — and choosing the best of 18 configurations on the same metric you then report is textbook selection bias. An out-of-sample or multi-split evaluation would fix it.\n\nThe central claim does not collapse. Within the all-mpnet-base-v2 pipeline, there are configurations that beat last-layer mean pooling on both coherence and diversity, and that pattern survives across datasets and in the dynamic setting. But the paper's framing as an 'enhancement' of BERTopic is not supported until the baseline is corrected to the actual default, or the claim is narrowed to 'within our base model.'\n\nWho is this for? Practitioners using BERTopic who want cheap empirical guidance on embedding choices. It is not a theoretical paper, and it does not need to be. The literature engagement is adequate, and the self-citation is not a problem.\n\nMy recommendation: send it to peer review, but flag the baseline issue as the primary required revision, and require variance or significance analysis before accept. As it stands it is a solid workshop-level empirical report that needs one more pass to become a reliable citable claim.","headline":"A useful, reproducible sweep of embedding layers and pooling for BERTopic, but the headline claim is weaker than it looks because the baseline may not be BERTopic's actual default and the best configuration is picked from the same metrics it is compared on.","tokens_in":11271,"tokens_out":1573,"would_cite":true,"duration_ms":17591,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For each of three datasets, some embedding configuration beats BERTopic's default on both topic coherence and topic diversity.","keywords":["BERTopic","topic modeling","intermediate layer embeddings","transformer hidden states","pooling strategies","topic coherence","topic diversity","stop word removal"],"falsifier":"Re-running the 18 configurations many times on the same datasets and computing confidence intervals for NPMI and topic diversity would settle it: if the gap between the best and default configuration is inside the interval, the claimed improvement is not distinguishable from noise.","tokens_in":10308,"feed_emoji":"📊","tokens_out":6576,"duration_ms":59988,"temperature":0.7,"pith_summary":"This paper tests whether BERTopic's standard document embedding—the final transformer layer with mean pooling—is actually the best choice for topic quality. It compares 18 configurations (six ways of extracting hidden states from the transformer, crossed with three pooling methods) on three datasets, measuring topic coherence with NPMI and topic diversity. The central claim is that every dataset has at least one embedding configuration that beats BERTopic's default on both metrics, so the default is not a safe optimum. The paper also finds that removing stop words improves coherence in 48 of 54 settings and diversity in 51 of 54, and that the best configuration changes with the dataset and with stop-word removal.","feed_headline":"Other embeddings beat BERTopic's default on every dataset","feed_subtitle":"Choosing the right transformer layer and pooling method raises topic coherence and diversity, but the best pick varies by corpus.","key_machinery":"The machinery is a configuration grid over transformer hidden states. Six retrieval strategies—embedding-layer output, last layer, second-to-last layer, sum of all layers, sum of last four layers, and concatenation of last four layers—are each combined with mean, max, and CLS pooling to produce document embeddings, which BERTopic clusters and scores. The comparison is carried by two metrics: NPMI topic coherence (from -1 to 1, measuring whether the top words of a topic co-occur more than chance) and topic diversity (the fraction of unique words across all topics). The paper also varies the input by removing stop words and by switching from static to dynamic topic modeling, giving 270 static runs and further dynamic runs.","core_discovery":"The central discovery is that BERTopic's default embedding choice leaves measurable performance on the table. On the 20 Newsgroups, Trump Tweets, and United Nations General Debate corpora, at least one of the 18 tested configurations yields higher NPMI topic coherence and higher topic diversity than the last-layer-with-mean-pooling default; on Trump Tweets, for example, the sum of all layers with max pooling raises coherence from 0.061 to 0.072 and diversity from 0.667 to 0.693. The best configuration is not universal: it shifts across datasets and after stop-word removal, and the embedding-layer output with mean pooling—not the last layer—gives the highest coherence on two of the three original datasets. CLS pooling is the weakest choice in most settings, max pooling most often maximizes diversity, and in the dynamic topic modeling extension the same pattern holds: some alternative configuration beats the default on each dataset.","pith_inferences":["The paper selects the best configuration using the same scores it later reports, so the size of the true advantage over the default is probably optimistic; a held-out evaluation or multiple train/test splits would be needed to turn 'a configuration exists' into a reliable selection rule.","If the layer effect is real, it suggests topical information is distributed across transformer layers rather than concentrated in the final layer, and the poorest CLS results hint that the [CLS] token's sentence-level summarization loses the word-level signal BERTopic's class-based TF-IDF needs.","A practical extension would be a cheap two-stage search: run a few aggregating configurations first (for example, sum-all-layers and concat-last-four with mean and max pooling), pick the best by coherence, and only then tune topic counts; this would keep the improvement while reducing the 1.5-hour full cost."],"forward_implications":["Embedding configuration should be treated as a hyperparameter of BERTopic; for any new corpus, a small search over layer and pooling choices can improve topic coherence and diversity at modest cost.","CLS pooling is a consistently weak choice for BERTopic topic quality; mean and max pooling are safer starting points.","Stop-word removal is broadly beneficial across embedding configurations: coherence improves in 48 of 54 settings and diversity in 51 of 54.","The same non-default advantage appears in dynamic topic modeling, where alternative configurations beat the default on both tested datasets and can surpass LDA Seq.","Because the best configuration is dataset-dependent, there is no single 'better default' the paper can recommend; the practical message is to explore rather than to switch universally."],"supporting_citations":[{"why":"Defines BERTopic, its default last-layer mean-pooled embedding, and the evaluation pipeline the experiments reuse.","marker":"(Grootendorst, 2022)"},{"why":"Provides Sentence-BERT and the all-mpnet-base-v2 model from which the hidden states and pooling options are taken.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Supplies the BERT layer-retrieval conventions (embedding layer, last four layers, sum/concatenation) that define six of the configurations.","marker":"(Devlin et al., 2019)"},{"why":"Defines normalized pointwise mutual information, the topic coherence metric used for all comparisons.","marker":"(Bouma, 2009)"},{"why":"Defines topic diversity as the share of unique words among top words, the second evaluation metric.","marker":"(Dieng et al., 2020)"},{"why":"Prior evidence that intermediate layers and pooling methods shift downstream performance, motivating the configuration grid.","marker":"(Ma et al., 2019)"},{"why":"Shows different BERT layers encode different syntactic and semantic information, the rationale for expecting layer choice to matter.","marker":"(Jawahar et al., 2019)"}],"fun_headline_variants":["Custom embeddings beat BERTopic defaults on every dataset","Better topic models: tuning BERTopic's layers and pooling","BERTopic's default isn't best: new embeddings win on 3 corpora","Layer and pooling choices boost BERTopic, but best varies","Tuning BERTopic embeddings: alternate layers beat defaults"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the differences between configurations are larger than the noise in the three-run averages, yet it reports no variance, confidence intervals, or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["Custom embeddings beat BERTopic defaults on every dataset","Better topic models: tuning BERTopic's layers and pooling","BERTopic's default isn't best: new embeddings win on 3 corpora","Layer and pooling choices boost BERTopic, but best varies","Tuning BERTopic embeddings: alternate layers beat defaults"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000698,"raw_usage":{"total_tokens":3116,"prompt_tokens":868,"completion_tokens":2248,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2161}},"tokens_in":484,"tokens_out":2248,"duration_ms":17073,"temperature":1.0,"reasoning_tokens":2161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:34:57.601940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the 18 configurations many times on the same datasets and computing confidence intervals for NPMI and topic diversity would settle it: if the gap between the best and default configuration is inside the interval, the claimed improvement is not distinguishable from noise.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines normalized pointwise mutual information, the topic coherence metric used for all comparisons."}],"review_version":1}