Applying LLM and Topic Modelling in Psychotherapeutic Contexts

Alexander Vanin; Anastasia Panfilova; Vadim Bolshev

REVIEW 4 major objections 6 minor 5 references

Applying LLM and Topic Modelling in Psychotherapeutic Contexts

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims BERTopic can automatically extract stable, interpretable topics from therapist speech, and that these topics recur across classical and modern therapy styles.

desk verdict A detailed descriptive case study of BERTopic on therapist speech, but the cross-era topic stability claim is not supported as reported. read the letter →

arxiv 2412.17449 v1 pith:D2HH7277 submitted 2024-12-23 cs.LG cs.AI

Alexander Vanin , Vadim Bolshev , Anastasia Panfilova This is my paper · ORCID

classification cs.LGcs.AI

keywords psychotherapylanguagespeechtopicmodelingBERTopictherapistclientlargemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that BERTopic, a transformer-embedding topic-modeling tool, can automatically extract meaningful and interpretable topics from therapist remarks in recorded therapy sessions, and that the same core topics recur across classical and modern therapeutic styles. The study applies the pipeline to two corpora built from public recordings—19 sessions with classical therapists and 111 sessions with modern therapists—and, after combining unsupervised clustering with LLM-generated labels and expert refinement, ends with roughly four dozen interpretable topics per corpus. Twelve topic pairs, including fear, anger, work-related anxiety, relationship and friendship complexities, education, mother relationships, pain, aging, possibilities, self-criticism, goals, and session planning, show near-identical content across the two groups. If the claim holds, it matters because topic modeling offers a scalable and less labor-intensive alternative to manual coding of therapy process, with practical uses in supervision and therapist training.

What carries the argument

The central object is BERTopic, a neural topic-modeling pipeline that turns a collection of short texts into a small set of labeled themes. It works by converting each therapist utterance into a vector with a multilingual Sentence-Transformer model, projecting the vectors into a low-dimensional space with UMAP, grouping them with HDBSCAN, and scoring words within each cluster using c-TF-IDF, a class-based variant of TF-IDF. The paper augments this machinery with GPT-generated topic labels and expert-driven merging of similar clusters, guided by hierarchical dendrograms, LDAvis maps, and coherence scores. The load-bearing idea is that clusters in embedding space correspond to interpretable therapeutic speech acts, so the same topics should reappear when the pipeline is run on different therapist populations.

What would settle it

Have human annotators transcribe and attribute every utterance in a set of therapy recordings, run the same BERTopic pipeline on the human-attributed therapist speech, and compare the resulting topics with those produced from automatic diarization and transcription. If the topic sets diverge substantially, or if the automatic pipeline's therapist utterances turn out to include client speech, the reported cross-era stability would be an artifact of transcription error rather than a property of therapeutic language.

Watch

Extended reading notes

Core claim

The central claim is that BERTopic—an unsupervised pipeline that embeds each utterance, reduces dimensionality with UMAP, clusters with HDBSCAN, and summarizes clusters with class-based TF-IDF—can identify stable thematic structures in therapist speech. The evidence is the observed overlap between two independently modeled corpora: after expert merging the paper reports 43 and 46 topic clusters, and twelve topic pairs reach cosine similarity between 0.9 and 1.0, covering fear, anger, work anxiety, relationships, education, mother relationships, pain and bitterness, aging and childhood, possibilities, self-acceptance and self-criticism, goals, and session planning. The conclusion draws the general lesson that BERTopic's unsupervised learning makes it an effective tool for psychotherapy content analysis across different contexts.

Load-bearing premise

The load-bearing premise is that the YouTube recordings are a representative sample of classical and modern therapeutic practice and that automatic diarization and transcription correctly isolate the therapist's speech from the client's.

Editorial extensions

If this is right

Automatic topic modeling can replace much of the manual coding of therapy transcripts, lowering cost and reducing coder bias in psychotherapy process research.
The stable cross-era topics point to a common core of concerns—fear, anger, work, relationships, education, and goals—that therapist training programs could target directly.
LLM-generated labels combined with expert merging provide a reusable workflow for turning raw transcripts into interpretable topic structures without hand-labeling every utterance.
The same pipeline can be extended to client remarks, to the evolution of topics within a single session, and to real-time feedback tools that support clinical supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable consequence is that therapist speech from a third culture or era should yield the same twelve core topics; if it does not, the observed stability may be an artifact of the small classical sample.
Because the topic labels come from an LLM, the human-readable 'topics' are partly a product of the labeling prompt; varying the prompt or model and measuring label stability would separate genuine structure from LLM phrasing.
The near-identical topic pairs could reflect common therapist techniques, but they could equally reflect common client problems; linking topic prevalence to session outcome measures would disambiguate these readings.
The practical value of the method depends on transcription quality, so a robustness check that reruns the pipeline on manually transcribed and speaker-attributed sessions would show how much of the result survives better data.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper is worth a look as a descriptive case study. It applies BERTopic to therapist remarks from classical and modern therapy sessions, uses GPT to label topics, and has experts merge and interpret the clusters. The topic lists are detailed, with example quotes, and the qualitative results look sensible: both corpora yield natural themes like fear, anger, and relationships. That is a legitimate new application, even if the pipeline itself is standard BERTopic.

The problem is the central comparative claim. Section 5.3 reports twelve topic pairs with cosine similarity 0.9–1.0 as evidence that the same topics recur in classical and modern speech. But the authors already merged and pruned clusters separately in each corpus, using the same kind of distance information that later defines the similarity. With roughly two thousand pairwise similarities, a few high-matching pairs are expected by chance; no permutation baseline or multiple-comparison correction is given. The corpora are also wildly imbalanced—19 classical sessions vs 111 modern ones, 8,641 vs 4,058 utterances—and the diarization that separates therapist from client speech is never validated. So the conclusion that BERTopic is a "powerful tool" for psychotherapy content analysis is overstated; what the paper actually shows is that BERTopic can produce interpretable topics in each corpus.

There are also mechanical errors that need fixing. Section 4.3 says 43 and 46 clusters, while Sections 5.1 and 5.2 say 44 and 47. The modern "Others" cluster contains 4,117 texts, which exceeds the 4,058 utterances in the whole modern corpus. No data or code is provided, so nothing can be reproduced.

My overall take: this is a useful feasibility study with a transparent description of the workflow and plausible qualitative findings. The cross-era stability result is not established as reported. If the authors reframe it as an exploratory demonstration and fix the counts, release the data, and add a null model or at least an explicit caveat, it deserves publication. A serious editor should send it to review rather than desk reject, because the corpus and the detailed topic interpretation have some value for the therapy-process research community. I would not cite it in my own work until the counts and the comparison are fixed.

Referee Report

4 major / 6 minor

Summary. The paper applies BERTopic, a transformer-embedding-based topic modeling pipeline, to two corpora of therapist utterances collected from public YouTube recordings: 19 sessions by classical therapists (Rogers, Perls, Ellis; 8,641 utterances) and 111 sessions by modern therapists across many orientations (4,058 utterances). The pipeline uses multilingual sentence embeddings, UMAP, HDBSCAN, c-TF-IDF, GPT-generated topic labels, and multiple rounds of expert-guided cluster merging. The authors report 44 and 47 final topics for the classical and modern corpora, respectively, list them with keywords and example quotes, and identify 12 topic pairs with cosine similarity between 0.9 and 1.0, which they interpret as evidence that common therapeutic themes recur across classical and modern styles. The conclusion asserts that BERTopic's unsupervised capabilities make it a powerful tool for psychotherapy content analysis.

Significance. If the cross-era topic recurrence claim were rigorously established, the paper would be a useful demonstration that modern NLP topic modeling can extract interpretable, clinically meaningful themes from therapist speech and track continuity across therapeutic traditions. The detailed topic inventories themselves are a potentially valuable descriptive resource for psychotherapy researchers, and the paper is transparent about its reliance on expert judgment and LLM-assisted labeling. However, the central comparative claim is not yet supported: the topic pairs in §5.3 are selected post hoc from a large similarity matrix with no null model, the corpora are highly imbalanced and unrepresentative, and internal numerical inconsistencies cast doubt on the underlying data curation. The paper is best viewed as a pilot study whose methodological gaps must be closed before the stated conclusion can be accepted.

major comments (4)

[§5.3] The central claim that 12 topics recur across classical and modern therapists is not statistically supported. The pairs are selected from a similarity matrix of roughly 43×46–47×47 topic pairs (about 2,000 comparisons) without any permutation baseline, null model, or multiple-comparison correction. High maximum cosine similarities are expected even for unrelated topic sets of this size, so the observed 0.9–1.0 values do not by themselves demonstrate stability. The authors should report the full distribution of pairwise similarities, compare against a shuffled-topic or random-embedding baseline, and state the threshold and correction used.
[§4.3, §5.1, §5.2] The manuscript contains conflicting internal counts that must be resolved. §4.3 states that 43 and 46 clusters remained for classical and modern therapists, while §5.1 and §5.2 report 44 and 47 topics, respectively. In addition, §3.1 reports 4,058 modern utterances, but the modern 'Others' cluster in §5.2 has N=4,117, which exceeds the entire modern corpus. If the modeling unit is preprocessed sentences rather than raw utterances, the paper should say so explicitly and report the actual document counts. As written, these inconsistencies undermine confidence in the data-handling pipeline.
[§4.3] The expert-guided merging step is a load-bearing part of the analysis but is not reported in a reproducible way. Experts merged or removed clusters using cosine distance and semantic judgment, and the same embedding/distance machinery is then used in §5.3 to define cross-group similarity. This creates a risk that the merging decisions artificially inflate cross-group overlap by discarding idiosyncratic clusters and leaving only generic themes. The authors should at least report inter-rater agreement, provide a sensitivity analysis showing that the 12 pairs survive under alternative merge criteria or with automated merging, or compare the merged result against the unmerged topic sets.
[§3.1] The comparison between 'classical' and 'modern' therapists is based on unrepresentative and unmatched samples: 19 sessions by only three named therapists (including repeated appearances of the client Gloria) versus 111 sessions across many orientations, with no matching on client demographics, session length, recording format, or therapist identity. The general claim that topics are stable 'across classical and modern therapeutic styles' therefore exceeds what this dataset can support. The paper should reframe the conclusion to refer to these particular corpora and explicitly discuss how sampling imbalances could bias the observed topic overlap.

minor comments (6)

[§5.3] The text says 'we will highlight only those topics with a coherence level ranging from 0.9 to 1.0,' but the metric being described is cosine similarity, not the topic coherence defined in §4.3. The wording should be corrected.
[§4.2.4] The GPT prompt is quoted, but the specific model version and generation parameters (temperature, max tokens) are not given. Please specify the model and settings for reproducibility.
[References] The reference to Devlin and Chang is misformatted and contains an incorrect year '1810'; this should be Devlin et al. (2019), 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.'
[§5.1] The 'Others' cluster for classical therapists contains 3,827 of 8,641 utterances (44% of the corpus). Please discuss what is excluded in this category and how its size affects the interpretation of the remaining topics.
[Affiliations] The affiliation contains a typo, 'Laborarory,' which should be 'Laboratory.'
[Abstract] The abstract states the method is free of the biases of manual coding, but §4.3 introduces substantial expert judgment. Please soften or qualify this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the BERTopic topic extraction and cross-era cosine comparison are descriptive outputs, not predictions forced by fitted inputs or self-citations.

full rationale

The paper's central claim is descriptive rather than derivational: BERTopic topics are produced from external pretrained Sentence-Transformer embeddings and published libraries (UMAP, HDBSCAN, c-TF-IDF), and the cross-era comparison is a cosine-similarity measurement over the resulting topic representations. No equation in the paper defines the reported cross-group topic pairs in terms of the model inputs, and no parameter is fitted to the target comparison and then renamed as a prediction. The expert merging step in Section 4.3 is an interpretability procedure applied separately within each corpus; the paper does not state that cross-corpus similarity information was used to guide the merging, so the later cosine similarities are not forced by construction. There are also no load-bearing self-citations or imported uniqueness theorems; citations to BERTopic, Sentence-Transformer, UMAP, and HDBSCAN are external method references. The main weakness is statistical, not circular: the 12 topic pairs with cosine similarity 0.9-1.0 are selected from a large pairwise matrix without a null model or multiple-comparison correction, and the two corpora differ greatly in size and composition. That concern affects the evidentiary strength of the stability claim, but it does not make the derivation circular. Accordingly, no circular step is exhibited, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the accuracy of transcription and speaker attribution, the representativeness of a convenience YouTube sample, and the meaningfulness of clusters produced by a standard embedding/clustering pipeline. None of these are validated against external benchmarks, and the free parameters are hand-set with no sensitivity analysis.

free parameters (5)

UMAP hyperparameters = n_neighbors=15, n_components=5, min_dist=0, metric=cosine
Hand-chosen in §4.2.2; no sensitivity analysis, so topic structure may depend on these values.
HDBSCAN minimum cluster size = 40
Set in §4.2.3; determines number of topics and outlier assignment, with no robustness check.
Embedding model choice = paraphrase-multilingual-MiniLM-L12-v2
Selected in §4.2.1 based on 'higher efficiency... verified empirically' but no comparison data shown; embeddings directly determine clusters.
Cosine similarity reporting threshold = 0.9 to 1.0
In §5.3 only topics with similarity 0.9-1.0 are reported as shared; threshold choice changes the list of common topics.
Expert topic merge decisions
Multiple iterations of merging and removing clusters based on expert judgment (§4.3); this manual model-selection process affects all reported topics and coherence.

assumptions (6)

domain assumption Automated diarization and transcription of YouTube recordings accurately separate therapist speech from client speech.
The corpora of 8641 and 4058 utterances are built on this, §3.1; no accuracy metrics provided.
domain assumption The selected YouTube sessions are representative of 'classical' and 'modern' psychotherapy.
Sample is 19 recordings from three named classical therapists vs 111 modern sessions from many orientations, §3.1; sampling is convenience-based.
domain assumption Pretrained Sentence-Transformer embeddings capture the semantics of therapeutic dialogue.
Used to build vector space in §4.2.1; not fine-tuned or validated on therapy text.
domain assumption HDBSCAN clusters correspond to meaningful latent topics.
Clustering in §4.2.3 with min_cluster_size=40; no validation that clusters map to interpretable themes beyond expert review.
standard math c-TF-IDF and cosine similarity are appropriate measures of topic content and similarity.
Used in §4.2.4 and §5.3; standard tools, but the c-TF-IDF attribution to Kenton & Toutanova (2019) is incorrect.
domain assumption Mimno coherence is a valid quality measure for these short documents.
Used in §4.3 to assess and compare topic quality; no comparison to human-rated coherence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Applying LLM and Topic Modelling in Psychotherapeutic Contexts." pith.science (2026). https://pith.science/paper/D2HH7277

@misc{pith2026241217449,
  author       = {Pith},
  title        = {Pith review of: Applying LLM and Topic Modelling in Psychotherapeutic Contexts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D2HH7277}},
  note         = {Machine review of arXiv:2412.17449}
}

read the original abstract

This study explores the use of Large language models to analyze therapist remarks in a psychotherapeutic setting. The paper focuses on the application of BERTopic, a machine learning-based topic modeling tool, to the dialogue of two different groups of therapists (classical and modern), which makes it possible to identify and describe a set of topics that consistently emerge across these groups. The paper describes in detail the chosen algorithm for BERTopic, which included creating a vector space from a corpus of therapist remarks, reducing its dimensionality, clustering the space, and creating and optimizing topic representation. Along with the automatic topical modeling by the BERTopic, the research involved an expert assessment of the findings and manual topic structure optimization. The topic modeling results highlighted the most common and stable topics in therapists speech, offering insights into how language patterns in therapy develop and remain stable across different therapeutic styles. This work contributes to the growing field of machine learning in psychotherapy by demonstrating the potential of automated methods to improve both the practice and training of therapists. The study highlights the value of topic modeling as a tool for gaining a deeper understanding of therapeutic dialogue and offers new opportunities for improving therapeutic effectiveness and clinical supervision.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 2 canonical work pages

[1]

Angelov, D. (2020). Top2vec: Distributed representations of topics. ArXiv Preprint ArXiv:2008.09470. Blei, D. M., Ng, A. Y ., & Jordan, M. I. (2001). Latent Dirichlet Allocation. Advances in Neural Information Processing Systems,

arXiv 2020
[2]

J., & Ogles, B

Lambert, M. J., & Ogles, B. M. (2004). The effectiveness of psychotherapy. In Bergin and Garfield’ s handbook of psychotherapy and behavior change (5th Edition, pp. 139–193). Wiley. Ma, P., Zeng-Treitler, Q., & Nelson, S. J. (2021). Use of two topic modelin g methods to investigate covid vaccine hesitancy. Int. Conf. ICT Soc. Hum. Beings , 384, 221–226. M...

work page arXiv 2004
[14]

Let me tel l you about your mental health!

ClearNLP Constituent -to-Dependency Conversion . (n.d.). Retrieved December 16, 2024, from https://github.com/clir/clearnlp- guidelines/blob/master/md/components/dependency _conversion.md Devlin, J., & Chang, M. (1810). W, Lee K, Toutanova K (2018) Bert: pre -training of deep bidirectional transformers for language understanding. ArXiv Preprint ArXiv:1810...

arXiv 2018
[102]

https://doi.org/10.1037/A0022161 OntoNotes Release 5.0 - Linguistic Data Consortium . (n.d.). Retrieved December 16, 2024, from https://catalog.ldc.upenn.edu/LDC2013T19 Sievert, C., & Shirley, K. (2014). LDAvis: A method for visualizing and interpreting topics. Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces, 63...

arXiv 2014
[2022]

https://doi.org/10.1155/2022/7893775

work page doi:10.1155/2022/7893775 2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Angelov, D. (2020). Top2vec: Distributed representations of topics. ArXiv Preprint ArXiv:2008.09470. Blei, D. M., Ng, A. Y ., & Jordan, M. I. (2001). Latent Dirichlet Allocation. Advances in Neural Information Processing Systems,

arXiv 2020

[2] [2]

J., & Ogles, B

Lambert, M. J., & Ogles, B. M. (2004). The effectiveness of psychotherapy. In Bergin and Garfield’ s handbook of psychotherapy and behavior change (5th Edition, pp. 139–193). Wiley. Ma, P., Zeng-Treitler, Q., & Nelson, S. J. (2021). Use of two topic modelin g methods to investigate covid vaccine hesitancy. Int. Conf. ICT Soc. Hum. Beings , 384, 221–226. M...

work page arXiv 2004

[3] [14]

Let me tel l you about your mental health!

ClearNLP Constituent -to-Dependency Conversion . (n.d.). Retrieved December 16, 2024, from https://github.com/clir/clearnlp- guidelines/blob/master/md/components/dependency _conversion.md Devlin, J., & Chang, M. (1810). W, Lee K, Toutanova K (2018) Bert: pre -training of deep bidirectional transformers for language understanding. ArXiv Preprint ArXiv:1810...

arXiv 2018

[4] [102]

https://doi.org/10.1037/A0022161 OntoNotes Release 5.0 - Linguistic Data Consortium . (n.d.). Retrieved December 16, 2024, from https://catalog.ldc.upenn.edu/LDC2013T19 Sievert, C., & Shirley, K. (2014). LDAvis: A method for visualizing and interpreting topics. Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces, 63...

arXiv 2014

[5] [2022]

https://doi.org/10.1155/2022/7893775

work page doi:10.1155/2022/7893775 2022