REVIEW 4 major objections 5 minor 34 references
Improving Topic Modeling of Social Media Short Texts with Rephrasing: A Case Study of COVID-19 Related Tweets
T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Rewriting tweets into formal language before topic modeling improves the coherence, uniqueness, and diversity of the resulting topics, with the largest gains from the most formal rephrasing, especially for word-co-occurrence-based models.
desk verdict Useful applied test of LLM rephrasing for short-text topic modeling, but the abstract oversells the results and the semantic-fidelity assumption is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The workhorse is 'rephrasing as a preprocessing step': a prompt-driven LLM transformation that turns grammatically messy, shorthand, emotionally charged posts into standardized sentences before topic modeling. The crucial distinction is between a mild 'general' rephrase (fix grammar and tone, keep meaning and entities) and a 'colloquial-to-formal' rephrase (convert to professional, report-like language). The formal version is the load-bearing intervention: by expanding abbreviations, repairing syntax, and replacing slang with standard vocabulary, it gives word-count-based topic models a denser co-occurrence signal, which is exactly the kind of signal classical topic models need. The paper's
What would settle it
On the paper's own example, the original tweet says 'Your Junk' (i.e., the vaccine) causes cytokine storms, while the colloquial-to-formal rephrase omits any mention of the vaccine and replaces the religious accusation with 'documented by divine entities.' If a human annotator judges that the rephrased text no longer expresses the original claim, or if a round-trip entailment test fails, the fidelity premise — and with it the interpretation of the metric improvements as improvements in understanding actual public discourse — is falsified.
Extended reading notes
Core claim
The paper's central claim is that a model-agnostic preprocessing step — having a large language model rewrite each tweet into cleaner, more formal language — improves the topical signal available to downstream topic models. The authors test two prompting strategies, a light 'general' rephrasing and a stronger 'colloquial-to-formal' rephrasing, across four topic models. Across their quantitative metrics, rephrased corpora produce more coherent, more unique, and more diverse topics and less redundant topic words than the original corpus, with the strongest gains for LDA (coherence rising from about 0.31 to 0.50) and consistent gains for an embedding-clustering model. The paper also reports a t
Load-bearing premise
The load-bearing premise is that rephrasing preserves the tweet's meaning — if the LLM inserts claims or drops references, the improved metrics describe rewritten text, not the public discourse the analysis supposedly tracks.
Editorial extensions
If this is right
- For researchers using classical bag-of-words topic models on short text, rephrasing can more than compensate for data sparsity, with coherence scores reaching levels comparable to or higher than those of embedding-based models.
- The improvement pattern suggests that embedding-based models benefit less because they already incorporate external semantic knowledge; rephrasing is most valuable where word co-occurrence statistics matter.
- The colloquial-to-formal scheme can drive topic redundancy to zero, producing maximally distinct topics, but it must be paired with a model that tolerates the more abstract vocabulary.
- In public-health monitoring, rephrased corpora surface risk-related concerns (e.g., death, risk, adverse) that are hard to recover from the original noisy text, enabling finer-grained issue tracking.
Reading between the lines
- The paper frames rephrasing as an alternative to algorithmic innovation, but its own trade-off data suggest a hybrid: keep the original text for sentiment or stance analysis and use the rephrased text only for topic discovery, since the rephrasing that helps topical coherence can erase the emotional tone that sentiment analysis needs.
- If the fidelity condition could be checked automatically (for example, by requiring the rephrased tweet to entail the original), TM-Rephrase becomes a general data-cleaning layer for any noisy short-text corpus — reviews, financial chatter, or political posts — not just public-health discourse.
- The formality level appears to act like a regularizer: too little helps, too much can over-smooth individual voice and even lower coherence for some models. This predicts a continuum of rephrasing strengths, and one could test whether an adaptive prompt that matches formality to tweet complexity performs best.
- A natural extension is to compare rephrasing with other enrichment methods (e.g., adding contextual tweets from the same conversation thread) to see whether the gains come from added context or from formalization alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TM-Rephrase, a model-agnostic preprocessing framework that uses an LLM (Gemini 2.5 Flash) to rewrite short, informal tweets into more standardized or formal text before applying standard topic models. Two prompt strategies are compared—general rephrasing and colloquial-to-formal (C-to-F) rephrasing—on a corpus of 25,027 tweets replying to CDC accounts during the COVID-19 pandemic. Four topic models (LDA, BERTopic, FASTopic, TSCTM) are evaluated with four metrics: topic coherence (C_v), uniqueness (TU), redundancy (TR), and diversity (TD). The authors report that rephrasing improves most metrics for most models, with C-to-F yielding the largest gains and LDA benefiting especially strongly, and they provide qualitative examples of improved keyword interpretability. The central claim is that LLM-based rephrasing is an effective data-centric lever for improving topic modeling of noisy short social media text.
Significance. If the central claim were fully supported, this would be a useful, low-cost contribution to social media text mining: instead of redesigning topic models, practitioners could prepend an LLM rewriting step. The paper has several strengths: it compares four diverse topic models, reports four complementary metrics, gives the exact prompts, uses public repositories for FASTopic and TSCTM (enabling reproducibility of those baselines), and includes explicit qualitative examples. However, the evidence as presented supports only a much narrower conclusion. The abstract's emphasis on LDA is contradicted by Table III, and the paper's own Table VII shows that the rephrasing step does not preserve semantic meaning, undermining the claim that the improved topics describe what users actually wrote. Because the coherence metric is computed against Wikipedia and the C-to-F prompt explicitly requests text resembling official/professional documents, part of the apparent improvement may be an artifact of aligning the rephrased text with the evaluation corpus. The paper's contribution therefore remains promising but not established.
major comments (4)
- [Abstract; Table III] The abstract claims that TM-Rephrase improves three topic-quality metrics and that the C-to-F strategy yields the greatest gains 'especially for the LDA algorithm.' Table III does not support this for LDA. For LDA, compared to the baseline (TU=0.575, TR=0.3048, TD=0.575), both general rephrasing (TU=0.5583, TR=0.3095, TD=0.5583) and C-to-F rephrasing (TU=0.5583, TR=0.3214, TD=0.5583) worsen TU, TR, and TD; only C_v improves. Similar metric-level trade-offs occur for other models (e.g., TSCTM with general rephrasing has worse TU and TR; FASTopic with C-to-F has worse C_v). The summary and Section V.A should be rewritten to report per-model, per-metric effects rather than a blanket 'improves three metrics' claim.
- [Section III.B; Table VII] The framework's stated goal is to 'maintain semantic fidelity' (Section III.B) and the prompts in Table I require 'keeping the meaning exactly the same' / 'Preserve the original meaning.' Table VII directly contradicts this. The General Rephrase inserts 'COVID-19 mRNA injections,' a phrase absent from the original, and the C-to-F Rephrase omits 'Your Junk' (the original's vaccine reference) and changes a specific causal claim into a generic statement about virus exposure. The paper itself acknowledges in Section V.B that C-to-F 'altered the original text by omitting the terms... that explicitly mention... vaccine.' If rephrasing changes content, then the improved topic keywords and metrics describe an LLM-rewritten discourse, not the original tweets. A systematic semantic-preservation evaluation—e.g., human annotation, entailment/NLI, or lexical alignment—is necessary before any claim th
- [Section IV.D; Table I] There appears to be metric-intervention alignment. The C-to-F prompt explicitly requests output 'suitable for inclusion in a public health report or professional summary,' while C_v (Eq. 2) is computed using the Wikipedia corpus. Formalized, report-like language is likely to have higher lexical overlap with Wikipedia-style text than raw tweets do, so the C_v gains may partly reflect prompt-to-corpus alignment rather than better topic structure. To rule this out, the authors should report coherence using an external corpus more similar to the original domain (e.g., a general social media corpus or the original tweet corpus itself) or otherwise demonstrate that the gains are not an artifact of the chosen evaluation reference corpus.
- [Section IV.C; Table III] The quantitative results are reported as single point estimates with no seeds, no variance, and no significance tests. Several differences in Table III are small (e.g., FASTopic general rephrasing C_v = 0.3723 vs. 0.3688; TSCTM general rephrasing TU = 0.9833 vs. 0.9917), and could easily be within run-to-run noise for stochastic topic models. The paper's language ('demonstrate,' 'substantially outperforming') is too strong for single-run comparisons. The authors should report multiple runs or bootstrap confidence intervals, and use paired or model-specific significance tests where appropriate.
minor comments (5)
- [Section I] Typo: 'lack intepretability' should be 'lack interpretability.'
- [Section III] The text says the pipeline is composed of three primary stages, but then lists four (data collection, rephrasing, preprocessing, topic modeling/evaluation). Numbering should be made consistent.
- [Section V.B] The reference to 'Table V, Topic 5' for the C-to-F vaccine-safety topic appears to be a typo; the C-to-F results are in Table VI, not Table V.
- [References] Reference [19] (Lau et al.) appears in the bibliography but does not appear to be cited in the text.
- [Section V.B] The qualitative assessment is informal and based on a single example; the authors themselves note that a more rigorous qualitative study is 'underway.' The current qualitative claims should be framed as illustrative, not as evidence.
Circularity Check
No significant circularity: empirical benchmark with standard external metrics; no fitted prediction or self-citation chain drives the result.
full rationale
TM-Rephrase is an empirical intervention study, not a derivation. It applies an LLM rephrasing prompt to a fixed corpus, then compares standard external topic-quality metrics (C_v, TU, TR, TD) on four published topic models. No parameter is fitted to the outcome and then reported as a prediction, and no uniqueness/self-citation argument is used to force the conclusion. The only self-citations ([21], [26]) appear in related-work lists and are not load-bearing. Two validity concerns are present but they are not circularity in the sense of this audit. First, there is metric/intervention alignment: the C-to-F prompt requests text 'suitable for inclusion in a public health report or professional summary' (Table I), while C_v is computed against the Wikipedia corpus (Sec. IV-D), so some coherence gains may reflect register similarity rather than deeper topic quality. Second, the paper's own Table VII and Section V admit semantic-fidelity violations (inserting 'COVID-19 mRNA injections' and omitting 'Your Junk'), which threatens the claim that rephrased topics describe the original tweets. These are empirical confounds or assumption violations, not cases where an equation reduces to its inputs by construction; indeed FASTopic's C-to-F coherence drops (0.3688 -> 0.3301), showing the reported improvements are not logically forced. Hence no circular step can be quoted, and the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Number of topics K =
8
- Number of top keywords per topic N =
15
- Prompt designs (General vs Colloquial-to-Formal) =
Two hand-written prompts (Table I)
- Coherence background corpus =
English Wikipedia dump
- Rephrasing LLM and version =
gemini-2.5-flash-lite-preview-06-17
assumptions (5)
- domain assumption LLM rephrasing preserves the semantic content of the original tweets (semantic fidelity).
- domain assumption Wikipedia is an appropriate reference corpus for measuring topic coherence of social-media-derived topics.
- domain assumption C_v, TU, TR, TD operationalize 'topic quality' relevant to interpretability.
- domain assumption Topics extracted from rephrased text are comparable to topics from original text (same underlying discourse).
- standard math NPMI-based C_v computation per Röder et al. is a valid coherence measure.
Cite this review
Pith. "Pith review of Improving Topic Modeling of Social Media Short Texts with Rephrasing: A Case Study of COVID-19 Related Tweets." pith.science (2026). https://pith.science/paper/UVSN5PLN
@misc{pith2026251018908,
author = {Pith},
title = {Pith review of: Improving Topic Modeling of Social Media Short Texts with Rephrasing: A Case Study of COVID-19 Related Tweets},
year = {2026},
howpublished = {\url{https://pith.science/paper/UVSN5PLN}},
note = {Machine review of arXiv:2510.18908}
}
read the original abstract
Social media platforms such as Twitter (now X) provide rich data for analyzing public discourse, especially during crises such as the COVID-19 pandemic. However, the brevity, informality, and noise of social media short texts often hinder the effectiveness of traditional topic modeling, producing incoherent or redundant topics that are often difficult to interpret. To address these challenges, we have developed \emph{TM-Rephrase}, a model-agnostic framework that leverages large language models (LLMs) to rephrase raw tweets into more standardized and formal language prior to topic modeling. Using a dataset of 25,027 COVID-19-related Twitter posts, we investigate the effects of two rephrasing strategies, general- and colloquial-to-formal-rephrasing, on multiple topic modeling methods. Results demonstrate that \emph{TM-Rephrase} improves three metrics measuring topic modeling performance (i.e., topic coherence, topic uniqueness, and topic diversity) while reducing topic redundancy of most topic modeling algorithms, with the colloquial-to-formal strategy yielding the greatest performance gains and especially for the Latent Dirichlet Allocation (LDA) algorithm. This study contributes to a model-agnostic approach to enhancing topic modeling in public health related social media analysis, with broad implications for improved understanding of public discourse in health crisis as well as other important domains.
Figures
Reference graph
Works this paper leans on
-
[1]
Public sphere in crisis mode: How the COVID-19 pandemic influenced public discourse and user behaviour in the Swiss Twitter-sphere,
A. Rauchfleisch, D. V ogler, and M. Eisenegger, “Public sphere in crisis mode: How the COVID-19 pandemic influenced public discourse and user behaviour in the Swiss Twitter-sphere,” Javnost - The Public, vol. 28, no. 2, pp. 129–148, 2021
2021
-
[2]
Top concerns of tweeters during the COVID-19 pandemic: Infoveillance study,
A. Abd-Alrazaq, D. Alhuwail, M. Househ, M. Hamdi, and Z. Shah, “Top concerns of tweeters during the COVID-19 pandemic: Infoveillance study,” J. Med. Internet Res., vol. 22, no. 4, p. e19016, Apr. 2020
2020
-
[3]
COVID- Senti: A large-scale benchmark Twitter data set for COVID-19 sentiment analysis,
U. Naseem, I. Razzak, M. Khushi, P. W. Eklund, and J. Kim, “COVID- Senti: A large-scale benchmark Twitter data set for COVID-19 sentiment analysis,” IEEE Trans. Comput. Soc. Syst., vol. 8, no. 4, pp. 1003–1015, 2021
2021
-
[4]
Understanding concerns, senti- ments, and disparities among population groups during the COVID-19 pandemic via Twitter data mining: large-scale cross-sectional study,
C. Zhang, S. Xu, Z. Li, and S. Hu, “Understanding concerns, senti- ments, and disparities among population groups during the COVID-19 pandemic via Twitter data mining: large-scale cross-sectional study,” J. Med. Internet Res., vol. 23, no. 3, p. e26482, 2021
2021
-
[5]
Short-text semantic similarity (stss): Techniques, challenges and future perspectives,
Z. H. Amur, Y . K. Hooi, H. Bhanbhro, K. Dahri, and G. M. Soomro, “Short-text semantic similarity (stss): Techniques, challenges and future perspectives,”Applied Sciences, vol. 13, no. 6, p. 3911, 2023
2023
-
[6]
A systematic review of the use of topic models for short text social media analysis,
C. D. P. Laureti, W. Buntine, and H. Linger, “A systematic review of the use of topic models for short text social media analysis,”Artificial Intelligence Review, vol. 56, no. 12, pp. 14223–14255, 2023
2023
-
[7]
Large language models and their appli- cations in bioinformatics,
O. A. Sarumi and D. Heider, “Large language models and their appli- cations in bioinformatics,” Comput. Struct. Biotechnol. J., vol. 23, pp. 3498–3505, Oct. 2024. :contentReference[oaicite:1]index=1
2024
-
[8]
Large language models in medical and healthcare domain: A review,
Z. Al Nazi and W. Peng, “Large language models in medical and healthcare domain: A review,” Artif. Intell. Rev., 2023. :contentRefer- ence[oaicite:2]index=2
2023
Show all 34 references
-
[9]
Exploring the expertise of large language models in materials science and metallurgical engineer- ing,
C. Bajan and G. Lambard, “Exploring the expertise of large language models in materials science and metallurgical engineer- ing,” Digit. Discov., vol. 4, pp. 500–512, Jan. 2025. :contentRefer- ence[oaicite:4]index=4
2025
-
[10]
Domain specialization as the key to make large language models disruptive: A comprehensive survey,
C. X. Ling, X. Zhao, J. Lu et al., “Domain specialization as the key to make large language models disruptive: A comprehensive survey,” arXiv preprint arXiv:2305.18703, May 2023. :contentRefer- ence[oaicite:6]index=6
2023 arXiv
-
[11]
Latent Dirichlet allocation,
D. M. Blei, A. Y . Ng, and M. I. Jordan, “Latent Dirichlet allocation,” J. Mach. Learn. Res., vol. 3, pp. 993–1022, Jan. 2003
2003
-
[12]
BERTopic: Neural topic modeling with a class- based TF-IDF procedure,
M. Grootendorst, “BERTopic: Neural topic modeling with a class- based TF-IDF procedure,” arXiv preprint arXiv:2203.05794, Mar. 2022. [Online]. Available: https://arxiv.org/abs/2203.05794
2022 arXiv
-
[13]
Fastopic: Pretrained transformer is a fast, adaptive, stable, and transferable topic model,
X. Wu, T. Nguyen, D. Zhang, W. Y . Wang, and A. T. Luu, “Fastopic: Pretrained transformer is a fast, adaptive, stable, and transferable topic model,” in Advances in Neural Information Processing Systems, vol. 37, pp. 84447–84481, 2024
2024
-
[14]
Empirical study of topic modeling in Twitter,
L. Hong and J. G. Wallach, “Empirical study of topic modeling in Twitter,” in Proc. Workshop on Applications for Topic Models: Text and Beyond, 2010, pp. 77–78
2010
-
[15]
Author-topic model for analyzing authors and their publications on large text collections,
Z. Li, C. Chen, and J. Bu, “Author-topic model for analyzing authors and their publications on large text collections,” Proc. Int. Conf. Comput. Intell. Secur., 2010, pp. 574–578
2010
-
[16]
Rajagopal, B
D. Rajagopal, B. Huang, L. Zettlemoyer, and D. S. Weld, ”Commonsense-based topic modeling,” in Proc. 2nd Int. Workshop Issues Sentiment Discovery Opinion Mining (WISDOM), 2013, pp. 1–8
2013
-
[17]
Top2Vec: Distributed representations of topics,
D. Angelov, “Top2Vec: Distributed representations of topics,” arXiv preprint arXiv:2008.09470, Aug. 2020. [Online]. Available: https://arxiv.org/abs/2008.09470
2008 arXiv
-
[18]
Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning,
X. Wu, A. T. Luu, and X. Dong, “Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning,” in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), pp. 2748–2760, Dec. 2022
2022
-
[19]
Automatic labelling of topic models,
J. H. Lau, K. Grieser, D. Newman, and T. Baldwin, “Automatic labelling of topic models,” inProc. 49th Annu. Meeting Assoc. Comput. Linguistics: Human Language Technol. (ACL-HLT), Portland, OR, USA, Jun. 2011, pp. 1536–1545
2011
-
[20]
Topic modeling for short texts with large language models,
T. Doi, M. Isonuma, and H. Yanaka, “Topic modeling for short texts with large language models,” inProc. 62nd Annu. Meeting Assoc. Comput. Linguistics (Vol. 4: Student Res. Workshop), Bangkok, Thailand, Aug. 2024, pp. 21–33
2024
-
[21]
Digital voices of survival: From social media disclosures to support provisions for domestic violence victims,
K. Wang, Z. Fu, W. Xin, L. Zhou, and S. K. Chandrappa, “Digital voices of survival: From social media disclosures to support provisions for domestic violence victims,”arXiv preprint arXiv:2509.12288, 2025
2025 arXiv
-
[22]
Deliberate reasoning in language models as structure-aware planning with an accurate world model,
S. Xiong, A. Payani, Y . Yang, and F. Fekri, “Deliberate reasoning in language models as structure-aware planning with an accurate world model,” inProc. 63rd Annu. Meeting Assoc. Comput. Linguistics (ACL), Vienna, Austria, Jul. 2025, pp. 31900–31931, doi: 10.18653/v1/2025.acl-...
2025 doi
-
[23]
Evaluating large language models for narrative topic labeling,
A. Piper and S. Wu, “Evaluating large language models for narrative topic labeling,” inProc. 5th Int. Conf. Natural Lang. Process. Digital Humanit., May 2025, pp. 281–291
2025
-
[24]
Information seeking and information sharing during the COVID-19 pandemic,
J. Z. Yang, Z. Liu, and J. C. Wong, “Information seeking and information sharing during the COVID-19 pandemic,” Communication Quarterly, vol. 70, no. 1, pp. 1–21, 2022
2022
-
[25]
Long COVID discourse in Canada, the United States, and Europe: Topic modeling and sentiment analysis of Twitter data,
A. G. T. AbuRaed, E. A. Prikryl, G. Carenini, and N. Z. Janjua, “Long COVID discourse in Canada, the United States, and Europe: Topic modeling and sentiment analysis of Twitter data,” Journal of Medical Internet Research, vol. 26, p. e59425, 2024
2024
-
[26]
Dynamic associations between Cen- ters for Disease Control and Prevention social media contents and epidemic measures during COVID-19: Infoveillance study,
S. Yin, S. Chen, and Y . Ge, “Dynamic associations between Cen- ters for Disease Control and Prevention social media contents and epidemic measures during COVID-19: Infoveillance study,” JMIR In- fodemiology, vol. 4, no. 1, p. e49756, 2024. [Online]. Available: https://infodem...
2024
-
[27]
Exploring communication by public health leaders and organizations during the pandemic: A content analysis of COVID-related tweets,
L. James, H. McPhail, L. Foisey, L. Donelle, M. Bauer, and A. Kothari, “Exploring communication by public health leaders and organizations during the pandemic: A content analysis of COVID-related tweets,” Canadian Journal of Public Health, vol. 114, no. 4, pp. 563-583, 2023
2023
-
[28]
Tracking social media discourse about COVID-19: Temporal, geographic, and thematic analysis,
E. Chen, J. Lerman, and E. Ferrara, “Tracking social media discourse about COVID-19: Temporal, geographic, and thematic analysis,” Journal of Medical Internet Research, vol. 22, no. 9, p. e19669, 2020
2020
-
[29]
Interpretable depression detection from social media text using LLM-derived embeddings,
S. Kim, O. Imieye, and Y . Yin, “Interpretable depression detection from social media text using LLM-derived embeddings,”arXiv preprint arXiv:2506.06616, 2025
2025 arXiv
-
[30]
Sentence-BERT: Sentence embeddings using Siamese BERT-networks,
N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP- IJCNLP), p...
2019
-
[31]
Exploring the space of topic coherence measures,
M. R ¨oder, A. Both, and A. Hinneburg, “Exploring the space of topic coherence measures,” inProc. 8th ACM Int. Conf. Web Search Data Mining (WSDM), Feb. 2015, pp. 399–408
2015
-
[32]
Topic modeling with Wasserstein autoencoders,
F. Nan, R. Ding, R. Nallapati, and B. Xiang, “Topic modeling with Wasserstein autoencoders,” inProc. 57th Annu. Meeting Assoc. Comput. Linguistics (ACL), Jul. 2019, pp. 6345–6381
2019
-
[33]
Decoupling sparsity and smoothness in the Dirichlet variational autoencoder topic model,
S. Burkhardt and S. Kramer, “Decoupling sparsity and smoothness in the Dirichlet variational autoencoder topic model,”J. Mach. Learn. Res., vol. 20, no. 131, pp. 1–27, 2019
2019
-
[34]
Topic modeling in embedding spaces,
A. B. Dieng, F. J. Ruiz, and D. M. Blei, “Topic modeling in embedding spaces,”Trans. Assoc. Comput. Linguist., vol. 8, pp. 439–453, 2020
2020
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.