Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation

Anna Lieb; Eni Mustafaraj; Maneesh Arora

REVIEW 3 major objections 4 minor 18 references

Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Topic models built from GPT-4 actor descriptions, rather than raw headlines, group short documents into named actor categories such as governors, teachers, and parents.

desk verdict A practical prompt-based augmentation idea whose evaluation is confounded by the prompt itself: BERTopic is clustering GPT-4's actor descriptions, so the 'unsupervised discovery' claim does not hold. read the letter →

arxiv 2504.17445 v1 pith:WJSWF46A submitted 2025-04-24 cs.CL

Anna Lieb , Maneesh Arora , Eni Mustafaraj This is my paper · ORCID

classification cs.CL

keywords largelanguagemodelsGPT-4topicmodelingtextaugmentationBERTopiccontentanalysiscriticalracetheorypoliticalscience

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a standard unsupervised topic model becomes far more useful for targeted social-science questions if you first replace each short document with an LLM-written description of the entity the document is about. The case study is 11,704 news headlines about critical race theory, and the targeted question is who the news frames as the primary actor. When the augmented descriptions, not the raw headlines, are fed to BERTopic, the resulting topics are labeled by concrete actor categories such as governors, legislators, teachers, parents, attorneys general, and news media, whereas the raw-text baseline produces general themes like racism, legislation, and education with actors mixed together. The authors argue that this lets researchers answer domain-specific framing questions with minimal human guidance while keeping topic modeling unsupervised and reproducible.

What carries the argument

The central mechanism is a two-step augmentation pipeline. A fixed GPT-4 prompt asks for a brief description of the primary actor in each headline ('What type of actor is the primary actor in this headline? Briefly describe the primary actor...'), and those descriptions, rather than the raw headlines, become the input documents to BERTopic, which clusters document embeddings and reports representative keywords per topic. The prompt embeds the domain-specific research target, who is the salient actor, without naming expected actors, so the topic structure is shaped by the augmentation's added semantic context instead of by raw headline word co-occurrence.

What would settle it

Take a random sample of headlines, have independent coders label the primary actor, and compare those labels with GPT-4's descriptions; then rerun the BERTopic pipeline with a neutral prompt that never mentions actors. If the actor-specific topic structure disappears under the neutral prompt, or if the descriptions systematically disagree with human labels, the claimed advantage over raw-text topic modeling would be shown to be an artifact of the prompt rather than a property of augmented text.

Watch

Extended reading notes

Core claim

The paper's central claim is that unsupervised topic modeling using GPT-4 blurbs rather than unprocessed text creates highly interpretable categories that can be used to investigate domain-specific research questions with minimal human guidance. In the critical race theory case study, GPT-4 was prompted to briefly describe the primary actor in each headline, and BERTopic on those descriptions returned topics cleanly labeled by actor type: CRT ideology itself, school administration, teachers, governors, legislators, parents, news media, Republicans, Joe Biden, Florida, military, attorneys general, and Southern Baptists. The raw-headline baseline instead returned diffuse themes such as racial conflict, state-specific coverage, and values in the classroom, and it lumped school boards, a Supreme Court justice, parents, and teachers into one topic. The paper concludes that LLM-generated augmentations add semantic context and real-world knowledge to short documents and can expand the utility of existing unsupervised techniques while maintaining interpretability and reproducibility.

Load-bearing premise

The comparison assumes that GPT-4's brief descriptions of each headline's primary actor are accurate enough, and that the prompt's instruction to name the primary actor is not itself what creates the clean actor topics.

Editorial extensions

If this is right

If the claim holds, researchers can use topic models to test framing hypotheses directly, such as whether CRT coverage centers grassroots actors (students, parents, teachers) versus political elites (legislators, governors, pundits).
Short-text corpora like social media posts, slogans, and single-sentence survey responses become analyzable by topic models with a domain-specific target, without hand-labeled training data.
The augmentation step can be redirected to other targeted research questions by changing only the prompt, while preserving the unsupervised pipeline.
The approach reduces the burden of manual qualitative interpretation, since topics come pre-grouped by actor roles rather than by diffuse themes.
The procedure is reproducible with a fixed prompt and model, assuming the LLM output is logged or held constant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Testable extension: swapping GPT-4 for a smaller or open-weights model would show whether the clean actor-topic structure is tied to the augmentation concept or to GPT-4's particular encyclopedic knowledge.
Testable extension: changing the prompt to ask about a different target dimension, such as the policy target or the geographic level of the actors, would reveal whether the method is a general targeted-augmentation framework rather than an actor detector.
Quantitative follow-up: measuring inter-coder agreement between human labels and BERTopic's topic labels on a held-out sample would test whether 'highly interpretable' holds beyond the authors' qualitative reading.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper is a proof-of-concept that LLM-generated descriptions can replace raw short text as BERTopic input, and the idea is worth remembering. But the central comparison is not a fair test. The augmentation prompt explicitly asks 'What type of actor is the primary actor in this headline?' — so the resulting 'topics' are clusters of GPT-4's answers to a targeted actor question. The raw-headline baseline got no such prompt. That confound is load-bearing.

To its credit, the paper identifies a real pain point: topic models on short headlines produce mixed, hard-to-use topics for social-science constructs. The proposed fix is concrete and cheap: prompt GPT-4 to describe the construct, then cluster those descriptions with BERTopic. The GDELT case study uses real data, the Table 1 examples look accurate, and the authors are transparent about their prompt and about not having done a thorough augmentation-quality evaluation. The pipeline itself is not in the cited literature, as far as I can tell, so it is a genuine variant even if a simple combination.

The soft spots are substantial. The evaluation rests on the authors' own qualitative reading of KeyBERT words—no coherence metrics, no human agreement statistics, no gold-standard actor labels, no statistical tests. For a pilot that is fine, but the abstract's general claim outruns the evidence. More importantly, the comparison between Table 2 and Table 3 conflates augmentation with instruction. If you asked GPT-4 to describe 'the main policy topic' instead of the primary actor, you would likely get topical clusters too. The observed actor organization is partly forced by the prompt. A proper baseline would use a neutral prompt (e.g., 'describe this headline') or at least include an 'actor extraction plus clustering' baseline that is honestly labeled as such. The method might still be useful, but it should be described as prompted extraction followed by clustering, not as unsupervised topic modeling.

Minor points: BERTopic minimum topic size differs between models (100 vs. 90), which is not a big deal but adds noise. The large 'no assignment' buckets are not discussed.

My recommendation: this deserves a serious referee, not a desk reject, because the underlying question is worth settling: does text augmentation with LLMs actually improve topic models for targeted constructs beyond simply prompting the LLM to produce construct-specific text? The current evidence does not answer that. I would ask the authors to add controlled baselines, quantitative evaluation, and softer claims. If the paper comes to me as referee, I would engage—but I would not accept the current headline claim as supported.

Referee Report

3 major / 4 minor

Summary. The paper proposes an LLM-augmented topic modeling pipeline for short texts, applied to a case study of 11,704 news headlines about critical race theory. For each headline, GPT-4 is prompted to describe the primary actor, and BERTopic is then run on these generated descriptions rather than on the raw headlines. The authors report that the augmented model produces clean actor-specific topics (governors, legislators, teachers, parents, etc.) whereas a baseline BERTopic model on raw headlines yields broader, less targeted themes. They conclude that LLM-generated augmentation creates highly interpretable categories suitable for domain-specific social science questions with minimal human guidance.

Significance. If the claim were supported, the paper would offer a practical recipe for using LLM augmentation to steer unsupervised topic models toward theoretically relevant constructs, which is a genuine need in computational social science. The manuscript is transparent about its data source (GDELT), the augmentation prompt, and the full BERTopic output in Tables 2 and 3, and it explicitly discloses that the augmentation quality was only qualitatively reviewed. These are strengths: the procedure is concrete and easy to replicate or challenge. The paper also correctly identifies a real limitation of standard topic models for targeted research questions. However, the evidence base is a single case study with no quantitative evaluation of interpretability and no controlled comparison that isolates the effect of augmentation from the effect of prompt content. The central claim therefore rests on an uncontrolled confound, which limits the paper's contribution as it currently stands.

major comments (3)

[Methods, footnote 1; Tables 2–3] The central comparison is not a fair test of augmentation. The augmentation prompt explicitly asks, 'What type of actor is the primary actor in this headline? Briefly describe the primary actor,' so the GPT-4 output is already organized around actor identity. Running BERTopic on these descriptions and finding actor-centric topics is largely a consequence of the prompt, not an emergent property of the augmented text. The baseline in Table 3 receives raw headlines with no equivalent targeted instruction. Consequently, Tables 2 and 3 conflate (i) the value of adding semantic context with (ii) the effect of telling the model which construct to attend to. Even perfectly accurate GPT-4 descriptions would likely produce actor clusters because the prompt demands actor descriptions. This confound is load-bearing for the paper's main claim that LLM-generated augmentation, rather than prompt design, drives the improved interpretability.
[Results; Tables 2–3] The claim that the augmented topics are 'highly interpretable' is supported only by the authors' own qualitative interpretation of the BERTopic keyword lists. No inter-coder agreement, no blinded human evaluation, no coherence metrics (e.g., NPMI or topic coherence), and no statistical test are reported. The manuscript's main evidence is the visual contrast between 'cleanly grouped' actor topics in Table 2 and the mixed actor/theme topics in Table 3. Without a blinded evaluation, the perceived interpretability gap may reflect the authors' expectations, especially because the prompt was designed to produce actor categories. This is a load-bearing gap: the paper's headline finding is about interpretability, yet interpretability is never measured.
[Methods; Results; Abstract] The paper states that the approach works with 'minimal human guidance,' but the actual procedure includes prompt engineering, a qualitative review of a sample of GPT-4 outputs (conceded in the Methods), a rule-based exclusion of 2,132 documents, and manual labeling of all resulting topics. The human effort is not quantified or compared with that of the baseline or with semi-supervised approaches such as keyword-assisted topic models. Moreover, because the prompt itself encodes the research construct ('primary actor'), the human guidance is substantial and is concentrated at the very step that produces the observed topical structure. The 'minimal human guidance' claim therefore overstates what the paper demonstrates.

minor comments (4)

[Table 3 caption] In the caption, 'displated' should be 'displayed.'
[Table 2, 'No assignment' row] The 'No assignment' row combines two different categories: 'Outlier documents' and 'Rule-based exclusion from model (contains “does not reference” or “does not explicitly reference”).' Please clarify how the rule-based exclusion was implemented and why those documents were not simply treated as a distinct topic.
[References] Reference [16] is mangled in the text: 'Andreas R.T. Schuck Sophie Lecheler, Mario Keer and Regula H¨anggli' should have the author list formatted correctly (presumably Lecheler, Keer, Schuck, and Hänggli).
[Methods] The prompt instructs GPT-4 not to include the headline in the response, so the augmented documents do not contain the original headline text. The paper should justify this design choice and discuss the possibility that the original text carries information that is lost in the paraphrase.

Circularity Check

1 steps flagged · score 6.0 of 10

The claimed actor-topic discovery is built into the augmentation prompt: GPT-4 is explicitly asked to name the primary actor, and BERTopic clusters those answers.

self definitional [Methods, footnote 1; Results paragraph 1]
"We used the following prompt to extract actor information from each headline: What type of actor is the primary actor in this headline? Briefly describe the primary actor. If the headline doesn't reference an actor, say so. You don't need to include the headline in your response."

The input documents to BERTopic are GPT-4's answers to a prompt that explicitly demands the primary actor's type. Clustering these descriptions therefore cannot fail to organize documents by actor type, so topics such as governors, legislators, teachers, and parents are the direct output of the extraction prompt rather than an unsupervised discovery from the raw headlines. The paper's central finding—that augmented topic modeling 'identifies specific primary actors'—is equivalent to the prompt's request to identify the primary actor, making the result true by construction. The raw-headline baseline received no equivalent prompt, so the observed contrast in Tables 2 and 3 conflates the effect of augmentation with the effect of telling the model to attend to actors.

full rationale

The derivation chain is short: headlines are passed to GPT-4 with a prompt that explicitly asks 'What type of actor is the primary actor in this headline?'; the resulting descriptions are used as BERTopic input; the topics are then labeled as actor categories. The circularity appears exactly at the prompt step: because the input text already contains the answer to a targeted actor-identification question, clustering it into governor/legislator/teacher/parent groups is a re-organization of that prompted content, not an independent latent-structure discovery. The paper does not rely on self-citation, and the specific actor labels are not enumerated in the prompt, so there is some genuine empirical content in which actor types dominate and how cleanly they cluster. However, the headline claim that augmented topic modeling 'creates highly interpretable categories' of actors is partially forced by construction, and the comparison against the raw headline model is not controlled for the prompt effect. This is a partial circularity rather than a complete one, so the score is 6 rather than 8 or 10.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim relies on two untested domain assumptions: the accuracy of GPT-4 descriptions and the validity of keyword-based interpretability judgments. The prompt design is an ad hoc choice that influences the outcome. Two BERTopic hyperparameters are chosen by hand and differ between models, adding further unstated flexibility.

free parameters (2)

BERTopic minimum topic size (augmented model) = 100
Hyperparameter chosen to control topic granularity; not justified and differs from the baseline.
BERTopic minimum topic size (baseline model) = 90
Hyperparameter chosen for baseline; differs from the augmented model, complicating comparison.

assumptions (3)

domain assumption GPT-4-generated actor descriptions accurately reflect the primary actor in each headline and provide relevant real-world context.
The method's output depends entirely on the quality of LLM augmentation; the authors note in Methods that this 'may vary across applications' but do not evaluate it.
domain assumption KeyBERT keyword representations are a valid basis for assessing topic interpretability.
Interpretability is judged by the authors reading representative keywords and documents; no inter-rater reliability or external validation is provided.
ad hoc to paper The prompt design does not itself predetermine the actor categories beyond the general notion of 'primary actor.'
The prompt explicitly asks for the primary actor, which plausibly drives the actor-specific topic structure; this confounds the comparison with the raw-headline baseline.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation." pith.science (2026). https://pith.science/paper/WJSWF46A

@misc{pith2026250417445,
  author       = {Pith},
  title        = {Pith review of: Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJSWF46A}},
  note         = {Machine review of arXiv:2504.17445}
}

read the original abstract

Unsupervised machine learning techniques, such as topic modeling and clustering, are often used to identify latent patterns in unstructured text data in fields such as political science and sociology. These methods overcome common concerns about reproducibility and costliness involved in the labor-intensive process of human qualitative analysis. However, two major limitations of topic models are their interpretability and their practicality for answering targeted, domain-specific social science research questions. In this work, we investigate opportunities for using LLM-generated text augmentation to improve the usefulness of topic modeling output. We use a political science case study to evaluate our results in a domain-specific application, and find that topic modeling using GPT-4 augmentations creates highly interpretable categories that can be used to investigate domain-specific research questions with minimal human guidance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

[1]

Three gaps in computational text analysis methods for social sciences: A research agenda

Christian Baden, Christian Pipal, Martijn Schoonvelde, and Mariken AC G van der Velden. Three gaps in computational text analysis methods for social sciences: A research agenda. Communication Methods and Measures , 16(1):1--18, 2022

work page 2022
[2]

Druckman

Dennis Chong and James N. Druckman. Framing theory. Annual Review of Political Science , 10(1):103--126, 2007

work page 2007
[3]

Keyword-assisted topic models

Shusei Eshima, Kosuke Imai, and Tomoya Sasaki. Keyword-assisted topic models. American Journal of Political Science , 2020

work page 2020
[4]

Assessing topic model relevance: Evaluation and informative priors

Angela Fan, Finale Doshi-Velez, and Luke Miratrix. Assessing topic model relevance: Evaluation and informative priors. Statistical Analysis and Data Mining: The ASA Data Science Journal , 12(3):210--222, 2019

work page 2019
[5]

A bayesian hierarchical topic model for political texts: Measuring expressed agendas in senate press releases

Justin Grimmer. A bayesian hierarchical topic model for political texts: Measuring expressed agendas in senate press releases. Political Analysis , 18(1):1--35, 2010

work page 2010
[6]

Text as data: The promise and pitfalls of automatic content analysis methods for political texts

Justin Grimmer and Brandon M Stewart. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political analysis , 21(3):267--297, 2013

work page 2013
[7]

BERT opic: Neural topic modeling with a class-based TF-IDF procedure

Maarten Grootendorst. BERT opic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 , 2022

arXiv 2022
[8]

Is anyone responsible?: How television frames political issues

Shanto Iyengar. Is anyone responsible?: How television frames political issues . University of Chicago Press, 1994

work page 1994

Show all 18 references

[9]

Text classification algorithms: A survey

Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura Barnes, and Donald Brown. Text classification algorithms: A survey. Information , 10(4):150, 2019

2019
[10]

GDELT : Global data on events, location, and tone, 1979--2012

Kalev Leetaru and Philip A Schrodt. GDELT : Global data on events, location, and tone, 1979--2012. ISA Annual Convention , pages 1--49, 2013

1979
[11]

Text mining for social science--the state and the future of computational text analysis in sociology

Ana Macanovic. Text mining for social science--the state and the future of computational text analysis in sociology. Social Science Research , 108:102784, 2022

2022
[12]

Laura K. Nelson. Computational grounded theory: A methodological framework. Sociological Methods & Research , 49(1):3--42, 2020

2020
[13]

GPT-4 technical report, 2023

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, (...), and Barret Zoph. GPT-4 technical report, 2023

2023
[14]

Topic GPT : A prompt-based topic modeling framework

Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit Iyyer. Topic GPT : A prompt-based topic modeling framework. arXiv preprint arXiv:2311.01449 , 2023

2023 arXiv
[15]

Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! arXiv preprint arXiv:2004.14914 , 2020

Suzanna Sia, Ayush Dalmia, and Sabrina J Mielke. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! arXiv preprint arXiv:2004.14914 , 2020

2004 arXiv
[16]

Schuck Sophie Lecheler, Mario Keer and Regula Hänggli

Andreas R.T. Schuck Sophie Lecheler, Mario Keer and Regula Hänggli. The effects of repetitive news framing on political opinions over time. Communication Monographs , 82(3):339--358, 2015

2015
[17]

How a conservative activist invented the conflict over critical race theory

Benjamin Wallace-Wells. How a conservative activist invented the conflict over critical race theory. The New Yorker, June 2021

2021
[18]

Source-lda: Enhancing probabilistic topic models using prior knowledge sources

Justin Wood, Patrick Tan, Wei Wang, and Corey Arnold. Source-lda: Enhancing probabilistic topic models using prior knowledge sources. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE) , pages 411--422. IEEE, 2017

2017

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Three gaps in computational text analysis methods for social sciences: A research agenda

Christian Baden, Christian Pipal, Martijn Schoonvelde, and Mariken AC G van der Velden. Three gaps in computational text analysis methods for social sciences: A research agenda. Communication Methods and Measures , 16(1):1--18, 2022

work page 2022

[2] [2]

Druckman

Dennis Chong and James N. Druckman. Framing theory. Annual Review of Political Science , 10(1):103--126, 2007

work page 2007

[3] [3]

Keyword-assisted topic models

Shusei Eshima, Kosuke Imai, and Tomoya Sasaki. Keyword-assisted topic models. American Journal of Political Science , 2020

work page 2020

[4] [4]

Assessing topic model relevance: Evaluation and informative priors

Angela Fan, Finale Doshi-Velez, and Luke Miratrix. Assessing topic model relevance: Evaluation and informative priors. Statistical Analysis and Data Mining: The ASA Data Science Journal , 12(3):210--222, 2019

work page 2019

[5] [5]

A bayesian hierarchical topic model for political texts: Measuring expressed agendas in senate press releases

Justin Grimmer. A bayesian hierarchical topic model for political texts: Measuring expressed agendas in senate press releases. Political Analysis , 18(1):1--35, 2010

work page 2010

[6] [6]

Text as data: The promise and pitfalls of automatic content analysis methods for political texts

Justin Grimmer and Brandon M Stewart. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political analysis , 21(3):267--297, 2013

work page 2013

[7] [7]

BERT opic: Neural topic modeling with a class-based TF-IDF procedure

Maarten Grootendorst. BERT opic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 , 2022

arXiv 2022

[8] [8]

Is anyone responsible?: How television frames political issues

Shanto Iyengar. Is anyone responsible?: How television frames political issues . University of Chicago Press, 1994

work page 1994