REVIEW 5 major objections 5 minor 38 references
Semantic-Augmented Latent Topic Modeling with LLM-in-the-Loop
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that a few-shot LLM should refine LDA topics after inference rather than seed the Gibbs sampler, and reports a 5.86% NPMI coherence gain from post-correction.
desk verdict The negative initialization result is the only thing worth keeping, and even it needs better evidence; the 5.86% gain is likely an artifact of the NPMI metric. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is an LLM-in-the-loop wrapper around Gibbs-sampled LDA, using Qwen2-7B-Instruct as the LLM. In the initialization phase, the LLM acts as a semantic coherence evaluator, scoring candidate word clusters with $C_v = \frac{1}{T}\sum_{t=1}^{T}\sum_{w_i,w_j\in t}\mathrm{sim}_{LLM}(w_i,w_j)$ and retaining clusters above a threshold as the starting topic assignments for Gibbs sampling. In the post-correction phase, the LLM is prompted with few-shot examples to judge each topic's word list and output the subset that fits, filtering semantically unrelated terms. This machinery transfers semantic knowledge into the count-based model while preserving LDA's deterministic, transparent inference; the failure mode the paper identifies is that hard cluster seeds inject external bias, whereas word-level filtering refines the output without restructuring the chain.
What would settle it
Re-run the post-correction step on the same 10,000 Chinese news articles with the same Qwen2-7B-Instruct model, computing NPMI before and after filtering for every topic and every random seed. If the 5.86% aggregate gain is not reproduced across seeds, or if it comes from a minority of topics while many others get worse, the central positive claim would be refuted. The claim would also fail if changing the few-shot examples or the LLM removes the gain, since the paper does not show that the result is stable across prompts and models.
Extended reading notes
Core claim
On the authors' own terms, the discovery is a placement effect: an LLM's semantic judgments help LDA most when applied after the fact, not before. When a few-shot LLM clusters the corpus vocabulary into $T$ topics and those clusters initialize Gibbs sampling, the Markov chain starts with lower perplexity but is overtaken by both random and K-Means initialization by the fourth iteration; at pass 20 its perplexity is 328.91 (eta=None) and 306.44 (eta=0.1), while random initialization reaches 309.36 and 285.04. The authors attribute the failure to cluster-based initialization injecting noise and external bias. When instead a few-shot LLM is asked to filter each LDA topic's word list, removing words it judges semantically unrelated, NPMI topic coherence improves by 5.86%; the worked example in Table 1 shows 'quantum' and 'universe' excised from a baby/mother topic. The conclusion is that the practical benefit of LLM-in-the-loop topic modeling lies in post-correction, not initialization.
Load-bearing premise
The reported 5.86% coherence improvement rests on the assumption that it is a stable aggregate NPMI gain across all topics and seeds, not a statistic that depends on the specific LLM prompt, the chosen model, or a few favorably filtered topics.
Editorial extensions
If this is right
- LLM-guided initialization should not be adopted as a general seeding strategy for LDA, since in this setup it yields the worst final perplexity and coherence among random, K-Means, and LLM initialization.
- LLM-based post-correction is a practical way to raise topic interpretability, with a reported 5.86% NPMI gain on a 10,000-document Chinese news corpus.
- The negative initialization result and the positive post-correction result together indicate that LLM semantic knowledge is best injected as a refinement layer rather than as a replacement for probabilistic inference.
- Direct LLM topic modeling remains brittle at scale: the paper reports that clustering 1,000 words produces repetitive word assignments and noise, so hybrid in-the-loop designs are the practical route.
Reading between the lines
- Beyond the paper: the 5.86% figure is reported as a single aggregate, and a natural follow-up would be to report the per-topic distribution of NPMI changes, since Table 1 shows only one cleaned topic and the gain could in principle be driven by a few topics.
- Beyond the paper: the initialization failure may be specific to hard cluster assignments; an untested variant would use LLM clusters as a soft prior over topic assignments rather than fixed starting values, which could preserve the early-iteration speedup without the injected noise.
- Beyond the paper: post-correction gains likely transfer beyond LDA to other bag-of-words topic models, but the paper does not test this; a comparison across models would show whether the 5.86% is a property of the LLM filter or of the LDA pipeline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an LLM-in-the-loop LDA framework in which a few-shot LLM (Qwen2-7B-Instruct) is used first to produce semantic clusters for initializing Gibbs sampling and then to filter incoherent words from the resulting topics. Experiments on 10,000 Chinese news articles compare random, k-means-cluster, and LLM-guided initialization in terms of perplexity and NPMI coherence, and evaluate post-correction by deleting LLM-flagged words from topics. The manuscript reports that LLM-guided initialization improves early iterations but not convergence, and yields the worst final performance among the three initialization methods, while LLM post-correction gives a 5.86% improvement in topic coherence. The paper concludes that replacing LDA with an LLM is not universally superior and advocates an LLM-in-the-loop design.
Significance. If the empirical claims were fully substantiated, the paper would provide a useful negative result for LLM-based topic-model initialization and a simple, practical recipe for post-correction. The evaluation metrics (perplexity and NPMI) are computed from the corpus alone, so the results are not circular in the sense of being fitted to the LLM's own outputs. The negative result on initialization is reported honestly, which is a strength. However, the current manuscript does not supply the data needed to support the headline 5.86% improvement, and the coherence scores in Table 3 appear incompatible with the stated NPMI definition. No code or data are provided, which further limits reproducibility. The significance of the contribution is therefore prospective rather than demonstrated as written.
major comments (5)
- [§4.3.2 and Table 1] The paper's only positive quantitative claim, a '5.86% improvement in topic coherence', cannot be verified from the manuscript: no aggregate NPMI values for the original and filtered topic sets, no number of topics, no top-k, and no variance are reported. Table 1 illustrates a single topic whose 20-word set is reduced to 15 words, and because NPMI is a pairwise average, deleting words changes the set of pairs over which the average is computed. The reported improvement may therefore be a mechanical artifact of shrinking word sets rather than evidence that LLM filtering helps; a control that keeps the number of words per topic constant is required.
- [§4.2 and Table 3] The coherence values in Table 3 (e.g., -17.8463 at pass=0 for LLM with eta=0.1) are outside the [-1,1] range of NPMI as defined in §4.2. Unless the authors intend an unstated aggregation convention such as summing over pairs or rescaling, the reported numbers are incompatible with the stated metric. This discrepancy must be resolved before any quantitative conclusion from Tables 2–3 can be accepted.
- [Abstract, §1, and §4.3.1] The manuscript contradicts itself about the effect of LLM-guided initialization: the Abstract states it 'has no effect on the convergence and yields the worst performance', the Introduction (§1) claims it can 'significantly impro[ve] the convergence efficiency', and §4.3.1 says it 'enhances the initial performance' even though Table 3 shows it is not the best at pass=0 for coherence. These statements must be reconciled with one consistent interpretation of Tables 2–3.
- [§4.1 and §4.3] The experimental design is too thin to support the paper's general conclusions: only one dataset (10,000 Chinese news articles), one LLM (Qwen2-7B-Instruct), and single runs without seeds, error bars, or significance tests are reported. Tables 2–3 give only pass=0 and pass=20, so neither the claimed 'average descent rate' nor the 5.86% improvement has an associated confidence interval. The authors should add multi-seed runs and confidence intervals, or reframe the contribution as a case study with caveats.
- [§3.1 and Appendix A] The LLM-guided initialization procedure is underspecified: the semantic similarity function simLLM and the threshold for retaining clusters are introduced without instantiation, and the appendix provides prompts but no number of shots, decoding temperature, or deterministic rule for converting LLM clusters into Gibbs sampling topic assignments. Without these details, the central negative result cannot be reproduced or attributed to the method rather than to prompt or implementation choices.
minor comments (5)
- [§4.2] The displayed equations for PMI and NPMI are broken across lines in a way that obscures their meaning; please typeset them as single, unambiguous equations.
- [Table 1] The caption states that deleted words are in bold, but no bold markup is visible in the submitted text; please format the deleted words consistently.
- [§4.1] The manuscript reports the hyperparameter 'eta' but does not state the number of topics T, the Dirichlet alpha, the vocabulary size, or the exact number of Gibbs passes; all of these settings should be reported to allow replication.
- [References] The related-work section cites several self-authored preprints and does not engage with widely used neural or LLM-based topic models such as BERTopic; please add a brief comparison and clearly distinguish published references from preprints.
- [Appendix A] Table 4, the example of LLM-supervised clustering, is not referenced or analyzed in the main text; either add a discussion or remove it.
Circularity Check
No load-bearing circularity: LLM outputs are exogenous semantic inputs, and both headline results are evaluated with corpus-derived NPMI and perplexity metrics.
full rationale
The paper's two central outcomes are empirical rather than derivational. LLM-guided initialization is compared through perplexity and NPMI reported in Tables 2 and 3, and LLM-guided post-correction is reported as a 5.86% NPMI improvement in Section 4.3.2. NPMI is defined from corpus co-occurrence probabilities (Section 4.2), and perplexity is a standard predictive likelihood; neither is defined in terms of the few-shot LLM prompts. The LLM is not assigned fitted parameters that are later renamed predictions. The post-correction prompt filters words by semantic category, while NPMI measures statistical association, so the evaluation is not identical to the LLM's criterion by construction. Although the references include numerous works by the same authors, these are used for background and motivation (e.g., Hong et al. 2025b) and are not the load-bearing justification for the experiments. Concerns about the missing aggregate NPMI values and the mechanical effect of word deletion on a pair-averaged score are reporting-validity issues, not circularity under the definition used here.
Assumptions & free parameters
free parameters (6)
- Number of topics T
- Dirichlet hyperparameter alpha
- Dirichlet hyperparameter beta (eta) =
None or 0.1
- Coherence threshold Cv for retaining LLM clusters
- Number of Gibbs passes =
20
- LLM model and prompt examples =
Qwen2-7B-Instruct
assumptions (4)
- standard math LDA and Gibbs sampling converge to the posterior under mild conditions.
- domain assumption NPMI computed on the corpus is a valid and reliable measure of topic coherence.
- domain assumption The LLM's semantic similarity judgments align with NPMI coherence for the Chinese news corpus.
- domain assumption A single 10,000-document Chinese news corpus is representative enough to generalize conclusions about LLM-based topic modeling.
Cite this review
Pith. "Pith review of Semantic-Augmented Latent Topic Modeling with LLM-in-the-Loop." pith.science (2026). https://pith.science/paper/RD3FXRTS
@misc{pith2026250708498,
author = {Pith},
title = {Pith review of: Semantic-Augmented Latent Topic Modeling with LLM-in-the-Loop},
year = {2026},
howpublished = {\url{https://pith.science/paper/RD3FXRTS}},
note = {Machine review of arXiv:2507.08498}
}
read the original abstract
Latent Dirichlet Allocation (LDA) is a prominent generative probabilistic model used for uncovering abstract topics within document collections. In this paper, we explore the effectiveness of augmenting topic models with Large Language Models (LLMs) through integration into two key phases: Initialization and Post-Correction. Since the LDA is highly dependent on the quality of its initialization, we conduct extensive experiments on the LLM-guided topic clustering for initializing the Gibbs sampling algorithm. Interestingly, the experimental results reveal that while the proposed initialization strategy improves the early iterations of LDA, it has no effect on the convergence and yields the worst performance compared to the baselines. The LLM-enabled post-correction, on the other hand, achieved a promising improvement of 5.86% in the coherence evaluation. These results highlight the practical benefits of the LLM-in-the-loop approach and challenge the belief that LLMs are always the superior text mining alternative.
Reference graph
Works this paper leans on
- [1]
-
[2]
Sicilia, Katherine Atwell, Diane Litman, and Malihe Alikhani
Yuya Asano, Sabit Hassan, Paras Sharma, Anthony B. Sicilia, Katherine Atwell, Diane Litman, and Malihe Alikhani. 2025. https://aclanthology.org/2025.coling-industry.32/ Contextual ASR error handling with LLM s augmentation for goal-oriented conversational AI . In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track...
work page 2025
-
[3]
Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021. https://doi.org/10.18653/v1/2021.acl-short.96 Pre-training is a hot topic: Contextualized document embeddings improve topic coherence . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing...
-
[4]
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993--1022
work page 2003
-
[5]
Jason Chuang, Margaret E Roberts, Brandon M Stewart, Rebecca Weiss, Dustin Tingley, Justin Grimmer, and Jeffrey Heer. 2015. Topiccheck: Interactive alignment for assessing topic model stability. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175--184
work page 2015
-
[6]
Tomoki Doi, Masaru Isonuma, and Hitomi Yanaka. 2024. https://aclanthology.org/2024.acl-srw.3 Topic modeling for short texts with large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 21--33, Bangkok, Thailand. Association for Computational Linguistics
work page 2024
-
[7]
Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794
arXiv 2022
-
[8]
Mengze Hong and Di Jiang. 2025. https://arxiv.org/abs/2506.07149 Technical report: A practical guide to kaldi asr optimization . Preprint, arXiv:2506.07149
arXiv 2025
Show all 38 references
-
[9]
Mengze Hong, Wailing Ng, Chen Jason Zhang, Yuanfeng Song, and Di Jiang. 2025 a . https://arxiv.org/abs/2412.09049 Dial-in llm: Human-aligned llm-in-the-loop intent clustering for customer service dialogues . Preprint, arXiv:2412.09049
2025
-
[10]
Mengze Hong, Wailing Ng, Chen Jason Zhang, Yifei Wang, Yuanfeng Song, and Di Jiang. 2025 b . https://doi.org/10.36227/techrxiv.174495034.42657551/v2 Llm-in-the-loop: Replicating human insight with llms for better machine learning applications
2025
-
[11]
Mengze Hong, Chen Jason Zhang, Chaotao Chen, Rongzhong Lian, and Di Jiang. 2025 c . https://arxiv.org/abs/2412.09034 Dialogue language model with large-scale persona data engineering . Preprint, arXiv:2412.09034
2025 arXiv
-
[12]
Mengze Hong, Chen Jason Zhang, Di Jiang, Yuanfeng Song, Lu Wang, Yuanqin He, Zhiyang Su, and Qing Li. 2025 d . https://arxiv.org/abs/2410.12444 Expanding chatbot knowledge in customer service: Context-aware similar question generation using large language models . Preprint, ar...
2025
-
[13]
Di Jiang and Wilfred Ng. 2013. Mining web search topics with diverse spatiotemporal patterns. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 881--884
2013
-
[14]
Di Jiang, Yuanfeng Song, Rongzhong Lian, Siqi Bao, Jinhua Peng, Huang He, Hua Wu, Chen Zhang, and Lei Chen. 2021 a . Familia: A configurable topic modeling framework for industrial text engineering. In Database Systems for Advanced Applications: 26th International Conference, ...
2021
-
[15]
Di Jiang, Yuanfeng Song, Yongxin Tong, Xueyang Wu, Weiwei Zhao, Qian Xu, and Qiang Yang. 2019. Federated topic modeling. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1071--1080
2019
-
[16]
Di Jiang, Yongxin Tong, and Yuanfeng Song. 2016. Cross-lingual topic discovery from multilingual search engine query log. ACM Transactions on Information Systems (TOIS), 35(2):1--28
2016
-
[17]
Di Jiang, Yongxin Tong, Yuanfeng Song, Xueyang Wu, Weiwei Zhao, Jinhua Peng, Rongzhong Lian, Qian Xu, and Qiang Yang. 2021 b . Industrial federated topic modeling. ACM Transactions on Intelligent Systems and Technology (TIST), 12(1):1--22
2021
-
[18]
Di Jiang, Jan Vosecky, Kenneth Wai-Ting Leung, and Wilfred Ng. 2012. G-wstd: A framework for geographic web search topic discovery. In Proceedings of the 21st ACM international conference on Information and knowledge management, pages 1143--1152
2012
-
[19]
Di Jiang and Lingxiao Yang. 2016. Query intent inference via search engine log. Knowledge and information systems, 49:661--685
2016
-
[20]
Di Jiang, Chen Zhang, and Yuanfeng Song. 2023. Probabilistic topic models: Foundation and application. Springer
2023
-
[21]
Charu Karakkaparambil James, Mayank Nagda, Nooshin Haji Ghassemi, Marius Kloft, and Sophie Fellenz. 2024. https://aclanthology.org/2024.acl-long.11 Evaluating dynamic topic models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...
2024
-
[22]
Jey Han Lau, David Newman, Sarvnaz Karimi, and Timothy Baldwin. 2010. Best topic word selection for topic labelling. In COLING 2010: Posters, pages 605--613
2010
-
[23]
Yawen Li, Di Jiang, Rongzhong Lian, Xueyang Wu, Conghui Tan, Yi Xu, and Zhiyang Su. 2021. Heterogeneous latent topic discovery for semantic text mining. IEEE Transactions on Knowledge and Data Engineering, 35(1):533--544
2021
-
[24]
Yida Mu, Chun Dong, Kalina Bontcheva, and Xingyi Song. 2024. https://aclanthology.org/2024.lrec-main.887 Large language models offer an alternative to the traditional approach of topic modelling . In Proceedings of the 2024 Joint International Conference on Computational Lingu...
2024
-
[25]
Philip Resnik and Eric Hardisty. 2010. Gibbs sampling for the uninitiated. LAMP-153
2010
-
[26]
Roberts and A.F.M
G.O. Roberts and A.F.M. Smith. 1994. https://doi.org/10.1016/0304-4149(94)90134-1 Simple conditions for the convergence of the gibbs sampler and metropolis-hastings algorithms . Stochastic Processes and their Applications, 49(2):207--216
1994 doi
-
[27]
Alexandra Schofield and David Mimno. 2016. Comparing apples to apple: The effects of stemmers on topic models. In Transactions of the Association for Computational Linguistics, volume 4, pages 287--300
2016
-
[28]
Xingyi Song, Johann Petrak, Ye Jiang, Iknoor Singh, Diana Maynard, and Kalina Bontcheva. 2021. Classification aware neural topic model for covid-19 disinformation categorisation. PLoS one, 16(2):e0247086
2021
-
[29]
Yuan-Feng Song, Yuan-Qin He, Xue-Fang Zhao, Han-Lin Gu, Di Jiang, Hai-Jun Yang, and Li-Xin Fan. 2024. A communication theory perspective on prompting engineering methods for large language models. Journal of Computer Science and Technology, 39(4):984--1004
2024
-
[30]
Dominik Stammbach, Vil \'e m Zouhar, Alexander Hoyle, Mrinmaya Sachan, and Elliott Ash. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.581 Revisiting automated topic model evaluation with large language models . In Proceedings of the 2023 Conference on Empirical Methods in ...
2023 doi
-
[31]
Luis Talavera and Javier B \'e jar. 1999. Integrating declarative knowledge in hierarchical clustering tasks. In Advances in Intelligent Data Analysis, pages 211--222, Berlin, Heidelberg. Springer Berlin Heidelberg
1999
-
[32]
Vijay Viswanathan, Kiril Gashteovski, Kiril Gashteovski, Carolin Lawrence, Tongshuang Wu, and Graham Neubig. 2024. https://doi.org/10.1162/tacl_a_00648 Large language models enable few-shot clustering . Transactions of the Association for Computational Linguistics, 12:321--333
2024 doi
-
[33]
Jan Vosecky, Di Jiang, Kenneth Wai-Ting Leung, and Wilfred Ng. 2013. Dynamic multi-faceted topic discovery in twitter. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 879--884
2013
-
[34]
Jan Vosecky, Di Jiang, Kenneth Wai-Ting Leung, Kai Xing, and Wilfred Ng. 2014. Integrating social and auxiliary semantics for multifaceted topic modeling in twitter. ACM Transactions on Internet Technology (TOIT), 14(4):1--24
2014
-
[35]
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024. Large language models are latent variable models: explaining and finding good demonstrations for in-context learning. In Proceedings of the 37th International Conference on Neural Information P...
2024
-
[36]
Haiyang Wu, Daxiang Dong, Xiaoguang Hu, Dianhai Yu, Wei He, Hua Wu, Haifeng Wang, and Ting Liu. 2014. https://doi.org/10.3115/v1/D14-1015 Improve statistical machine translation with context-sensitive bilingual semantic embedding model . In Proceedings of the 2014 Conference o...
2014 doi
-
[37]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[38]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.