From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis

Elham Aghakhani; George Demiris; Jina Huh-Yoo; Karla T. Washington; Lu Wang; Rezvaneh Rezapour

REVIEW 4 major objections 7 minor 15 references

From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that zero-shot GPT-4o, prompted with a codebook of five PST core strategies and four newly added facilitative strategies, can label therapist utterances in real-world problem-solving therapy transcripts at a weighted F1…

desk verdict Useful new codebook and realistic corpus, but the headline F1 is computed on an ambiguous evaluation unit and a single-expert gold standard; re-examine before trusting the downstream numbers. read the letter →

arxiv 2501.06101 v2 pith:62CPBSYU submitted 2025-01-10 cs.CL

Elham Aghakhani , Lu Wang , Karla T. Washington , George Demiris , Jina Huh-Yoo , Rezvaneh Rezapour This is my paper · ORCID

classification cs.CL

keywords problem-solvingtherapyLLMannotationGPT-4odialogueanalysistherapeuticdynamicsPSTstrategiesmentalhealthNLPtransformerfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that large language models can take over the fine-grained labeling of what therapists actually do in problem-solving therapy sessions, turning unstructured transcripts into structured data. The authors build an annotation framework from the five-step ADAPT model of PST and add a new set of four facilitative strategies, then prompt GPT-4o to label 14,417 real therapist utterances. GPT-4o reaches a weighted F1 of 0.76, a combined precision-and-recall score, against a 500-utterance human-annotated gold set, outperforming GPT-4, Llama-3.1, and Yi-1.5, and its labels are stable across runs (mean entropy 0.035). On top of the labels, the paper analyzes autonomy, self-disclosure, and metaphor use, finding that therapists shift from non-directive, metaphor-rich exploration early in therapy to directive, concrete implementation language later. If the labeling is trustworthy, the framework offers a scalable way to study and eventually guide PST delivery, including real-time feedback for therapists and fidelity checks for AI therapy tools.

What carries the argument

The load-bearing object is the annotation codebook: five PS Core strategies taken from the ADAPT model of problem-solving therapy, extended with four new Facilitative strategies that capture rapport-building, session logistics, engagement, and test review. The machinery is zero-shot prompting of GPT-4o with strategy definitions and few-shot examples at temperature zero, plus an entropy check over five repeated runs to confirm label stability (mean entropy 0.035). The same prompting approach is then reused for therapeutic dynamics, asking GPT-4o to classify autonomy, self-disclosure, question type, and metaphor with source and target domains, and LIWC is applied to the labeled utterances to connect strategies to linguistic features. The codebook carries the argument because F1 is measured strategy by strategy, and the downstream progression and dynamic analyses inherit the labels.

What would settle it

Take a fresh sample of, say, 500 utterances from the same corpus and have two PST experts with documented expertise annotate them independently, then reconcile disagreements to create an adjudicated gold set. If GPT-4o's weighted F1 against this independent gold drops well below 0.76, or if expert-expert agreement is substantially lower than GPT-4o's agreement with the original labels, the claim that GPT-4o reliably annotates PST strategies would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that automated annotation of PST is achievable with current LLMs. Using a codebook with five PS Core strategies from the ADAPT model (positive mindset, defining problems and goals, generating alternative solutions, outcome prediction and planning, trying out solution plan) and four novel Facilitative strategies (social courtesies, session management, therapeutic engagement, test review), the authors prompted GPT-4o at zero temperature to classify each therapist utterance. Without any conversational context, GPT-4o achieved a weighted F1 of 0.76, with per-strategy F1 scores from 0.68 to 0.92; adding two prior utterances lowered performance to 0.66 because the model sometimes labeled the context instead of the target utterance. The authors then used GPT-4o's labels to annotate all 14,417 utterances and found a clear progression across visits: early sessions emphasize positive mindset and problem definition, later visits shift to generating alternatives and planning, and the final visit emphasizes trying out the solution plan. They also prompted GPT-4o to label directive versus non-directive autonomy, immediate versus non-immediate self-disclosure, and metaphor use, and they report that non-directive language and metaphors dominate early stages while directive, concrete language grows later. Fine-tuned transformer models trained on 5,000 GPT-4o-labeled utterances reach competitive F1 scores (ModernBERT 0.80 for PS Core, DeBERTa 0.73 for Facilitators), supporting a distillation path to smaller, privacy-conscious models.

Load-bearing premise

The entire evaluation rests on the 500 therapist utterances that one domain expert and one researcher labeled by hand; that gold standard was not adjudicated or checked against a second independent expert, so any mistakes or biases in it are baked into the reported F1 and into every label GPT-4o subsequently assigned to the full corpus.

Editorial extensions

If this is right

The full corpus of 14,417 therapist utterances is now labeled with PS Core and Facilitative strategies, enabling quantitative study of how PST is actually delivered across visits.
Session-level strategy progression is observable: therapists start with mindset and problem definition, move to alternative solutions and planning, and emphasize trying out the solution plan in the final visit, with Test Review concentrated in the first visit.
Adding conversational context to the prompt currently hurts rather than helps GPT-4o (F1 0.66 vs 0.76), so future work needs better context-grounding rather than simply feeding more dialogue history.
Small fine-tuned models trained on LLM labels can reach competitive accuracy (ModernBERT 0.80 on PS Core; DeBERTa 0.73 on Facilitators), suggesting a path to local, privacy-preserving annotation of therapy data.
The framework can serve as an evaluation tool for LLM-assisted therapy systems, checking whether automated interventions actually apply PST strategies and maintain therapeutic alliance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same codebook-plus-prompt recipe could be adapted to other structured therapies such as CBT or interpersonal therapy, since the Facilitative strategies are largely therapy-agnostic; the paper does not test this.
The finding that self-disclosure appears in only 257 utterances and is mostly non-immediate suggests a quantitative fidelity marker: a therapist disclosing frequently or immediately may be deviating from typical PST practice, and this could be monitored automatically.
If context degrades performance, a promising testable extension is to give the model the preceding strategy label or a short summary of the session instead of raw previous utterances, which might improve coherence without the observed label-leakage errors.
A direct application the authors do not develop: run the same pipeline on transcripts from LLM-based mental health chatbots to measure how faithfully they follow PST, providing an audit mechanism for AI therapy products.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper has two contributions worth talking about. First, the new Facilitative Strategies codebook (four categories added to the standard PST Core) is a practical addition that other therapy-annotation work can reuse. Second, the corpus itself—240 real sessions with 14,417 therapist utterances—is a solid resource, even though the transcripts are not public. The comparison across four LLMs and four transformer models is thorough, and the finding that a fine-tuned encoder (ModernBERT, 0.8 PS Core F1) can beat a strong LLM on the distillation task is genuinely useful for privacy-conscious deployment.

The soft spot is precisely where the stress-test lands. The headline 0.76 weighted F1 is measured on 500 hand-annotated utterances, but Table 3 reports support counts that sum to 525. This isn't a typo; it reflects an ambiguous evaluation unit. If annotators could assign one PS Core and one Facilitator per utterance, then the flattened 10-class evaluation should have either 500 instances (one merged label per utterance) or 1,000 (the two dimensions evaluated separately). Getting 525 means some utterances are double-counted and some are not, so the weighted average is over an ill-defined set. Until the authors clarify exactly which labels went into the computation, the 0.76 cannot be interpreted as an utterance-level accuracy. That ambiguity also undermines the per-class precision and recall in Table 3 and propagates into the downstream session-progression and therapeutic-dynamics analyses, which use GPT-4o's full-corpus labels without adjusting for measurement error.

The gold standard is a second real concern. The annotation was done by one domain expert and one researcher, with Cohen's Kappa reported per class but no adjudication or resolution protocol described. The final labels are effectively one expert's judgments, so label noise is unmeasured and the F1 could be biased in either direction. This is not fatal—many annotation studies rely on a single expert—but it should be acknowledged and, ideally, a second independent expert should validate at least a subset.

On the positive side, the paper is candid about its limitations and does not overclaim clinical readiness. The Facilitative Strategies were developed on the same corpus, but the evaluation set is separate, so the circularity burden is mild. The paper does not release data or code, which limits reproducibility but is understandable for therapy data.

Who gets value from this? Researchers working in computational mental health, clinical NLP, or PST process research. The identified issues are fixable: clarify the evaluation unit, add adjudication details or a second expert, and state that downstream analyses inherit label noise. I would send it to peer review with a request for major revision; the core contribution is solid enough to deserve another round.

Referee Report

4 major / 7 minor

Summary. The paper presents an annotation framework for therapist utterances in problem-solving therapy (PST), combining five established PST Core Strategies with four newly proposed "Facilitative Strategies" (Social Courtesies, Session Management, Therapeutic Engagement, Test Review). The authors evaluate four LLMs (GPT-4, GPT-4o, Llama-3.1-70B, Yi-1.5-34B) and four transformer-based classifiers on 500 human-annotated therapist utterances from 240 real-world sessions, reporting that GPT-4o achieves the highest weighted F1 (0.76) and using GPT-4o to label all 14,417 in-scope utterances. On this labeled corpus they analyze strategy prevalence, therapeutic dynamics (autonomy, self-disclosure, metaphor), LIWC-based linguistic patterns, and strategy progression across the first three visits. They additionally fine-tune DeBERTa, MentalBERT, ModernBERT, and FLAN-T5 on 5,000 GPT-4o-labeled utterances and report competitive F1 scores.

Significance. If the headline result holds, the paper offers a useful, scalable pipeline for automating PST process annotations: the use of real-world transcripts (rather than social media or simulated data), the extension of the PST codebook with expert-informed Facilitative Strategies, and the explicit comparison of proprietary and open-weight LLMs and distilled transformer students are all valuable. The paper also ships a deterministic, entropy-checked prompting setup and evaluates on a human gold standard rather than in a self-referential loop. However, the strength of the claim is currently bounded by the small, unadjudicated gold standard, the ambiguous evaluation unit, the absence of confidence intervals, and the unvalidated use of GPT-4o labels for all downstream distributional and dynamics analyses. These issues are fixable and do not undermine the interest of the framework.

major comments (4)

[Section 3.1 / Table 3] The evaluation set is described as 500 therapist utterances, but the support column in Table 3 sums to 525 instances (PS Core: 39+66+36+33+18; Facilitators: 25+64+130+24; None: 90). Since the annotation instructions allow one PS Core and one Facilitator label per utterance, the unit of evaluation is ambiguous: the weighted F1 could be computed over merged utterance-level labels, over the two annotation dimensions separately, or over a flattened set that yields 525 labels. The paper must state the evaluation unit explicitly, report per-dimension and overall metrics consistently, and provide confidence intervals for the 0.76 figure. As written, the headline metric cannot be interpreted as utterance-level accuracy.
[Section 3.1, Data Annotation] The gold standard was produced by "one domain expert alongside one researcher," with per-class Cohen's Kappa, but the paper does not describe any adjudication process, a second independent expert, or how disagreements were resolved into the final labels. Inter-annotator agreement between these two coders does not, by itself, validate the final gold labels against a reference; if the final set reflects one expert's judgments, label noise is unmeasured and the reported F1 could be biased in either direction. The authors should report the adjudication protocol, ideally have a second expert review the disagreements, and quantify the robustness of the reported F1 to alternative gold-label resolutions.
[Sections 4.3-4.5 (Figures 2-7, Table 5)] All downstream results—strategy prevalence, autonomy/self-disclosure/metaphor analyses, and the visit-by-visit progression—are computed from GPT-4o labels on the full 14,417 utterances, and the only direct validation of those labels is the 500-utterance gold set. The paper reports no stratified evaluation of the full-corpus label distribution, no per-class error analysis beyond Table 3, and no uncertainty quantification. In particular, the closing sentence of Section 4.5—that the observed patterns "further validate the effectiveness of our classifier"—is circular when the patterns are derived from the classifier's own labels. Please validate full-corpus annotations on a held-out human-annotated sample (e.g., a stratified sample by strategy and visit) or explicitly reframe Sections 4.3-4.5 as exploratory model-generated findings.
[Section 3.3 / Appendix D.2] The therapeutic-dynamics labels used in Section 4.3—autonomy (directive/non-directive), self-disclosure (immediate/non-immediate), question type, and metaphor presence—are generated by GPT-4o with no human-annotated evaluation of these dimensions. Figures 2-5 and 10 and the percentage claims in Section 4.3 (e.g., 94.49% of directive utterances being closed-ended) therefore inherit unquantified model error. The authors should either evaluate the dynamics annotation on a small human-annotated sample or present these analyses as illustrative rather than as validated measurements.

minor comments (7)

[Abstract] The abstract states that GPT-4o achieved "the highest accuracy (0.76)"; all reported metrics are F1 scores, so this should read "highest weighted F1" to avoid conflating accuracy with F1.
[Section 3.1] The data provider is identified only as "[Anonymized collaborator]"; please name the source or provide a fuller data-use description so readers can assess provenance and external validity.
[Appendix C / Section 3.2] Appendix C lists the GPT-4o model as "gpt-4o-2024-05-13" while Section 3.2 states "gpt-4o-2024-08-06"; the model identifier should be consistent.
[Section 4.1] The statement that "all models showed lower F1 scores when provided with additional context" is not strictly supported by Table 1: GPT-4's Facilitators score is 0.65 in both conditions. Please qualify the claim to the affected settings.
[Section 3.1] The exclusion of utterances shorter than five words is justified by an assumption that longer utterances are more meaningful; please report a sensitivity analysis of the main prevalence and progression results to this threshold, since the threshold defines the entire analysis corpus.
[Table 2] Because the models were run at temperature 0, one would expect identical labels across five runs; the nonzero mean entropy values need a brief explanation (e.g., API nondeterminism) to be interpretable.
[Figure 10 / Appendix G] The co-occurrence matrix reports percentages within autonomy rows, but the caption does not state whether row or column normalization is used; please clarify the normalization and the denominator for the quoted percentages.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the 0.76 F1 is measured against a human-annotated gold standard, and no load-bearing derivation reduces to its own inputs.

full rationale

The paper's central claim—GPT-4o achieves a weighted F1 of 0.76 on PST strategy annotation—is an empirical evaluation against a human-annotated gold standard of 500 utterances, not a quantity derived from the model's own outputs or from a fitted parameter. The PS Core strategies come from the established ADAPT model (Demiris et al., 2019), and the newly introduced Facilitative Strategies are an expanded annotation scheme applied to the corpus; applying a codebook to the same corpus it was designed for is descriptive coding, not circular derivation. The transformer fine-tuning experiments use GPT-4o-generated labels as training data but are evaluated on the same independent human test set, so their reported F1 values are not forced by construction. The main reviewer concerns—a single domain expert plus one researcher with no described adjudication, and Table 3 support counts summing to 525 rather than 500—are validity and reporting issues rather than circularity: they affect the interpretability of the metric but do not make the metric equivalent to its inputs. Similarly, Section 4.5's statement that progression patterns 'further validate the effectiveness of our classifier' is an overclaim, since those patterns are summaries of the model's own labels, but it is not a load-bearing derivation of the headline F1. No self-citation chain, uniqueness argument, or fitted parameter renamed as a prediction is used to force any central result. Score 2 reflects minor methodological concerns; no specific circular step meets the quoted-evidence bar.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The reported F1 stands on a hand-chosen five-word minimum utterance length, an internally developed codebook, and a small gold standard annotated by one expert and one researcher. The distributional and strategic-progression findings further inherit any errors in GPT-4o labels, because only 500 of 14,417 utterances are human-validated.

free parameters (1)

Minimum utterance length threshold = 5 words
The authors excluded all therapist utterances shorter than five words to 'ensure high quality and relevance' (Section 3.1). This threshold is chosen by hand, not derived, and it removes 19,856 of 34,273 utterances, so it could affect the strategy distribution reported in Table 5.

assumptions (5)

domain assumption The five ADAPT PST Core Strategies are valid and sufficient for coding therapist behavior in these sessions.
Used as the basis of the codebook in Section 3.1 (Demiris et al., 2019); not empirically revalidated in this corpus.
ad hoc to paper The four novel Facilitative Strategies are valid, distinct, and complete enough to capture non-core therapist actions.
Created by the authors and domain experts 'upon further exploration of our data' (Section 3.1); no independent validation or inter-rater reliability on a separate corpus is provided.
domain assumption Human labels from one domain expert and one researcher constitute a reliable gold standard.
Section 3.1 Data Annotation; only two annotators, one not an expert, and no adjudication procedure is described.
ad hoc to paper Excluding utterances shorter than five words does not bias the evaluation or the distributional findings.
Section 3.1; 19,856 of 34,273 therapist utterances were removed, which may preferentially drop short acknowledgments and affect strategy prevalence.
domain assumption Transcriptions preserve the content needed for strategy classification despite the loss of tone and non-verbal cues.
Acknowledged as a limitation in Section 7; the analysis is textual only.

invented entities (1)

Facilitative Strategies (Social Courtesies, Session Management, Therapeutic Engagement, Test Review)
purpose: Added to the PST codebook to label therapist utterances that do not fit the five ADAPT core steps, enabling the reported distribution and dynamics analyses.
Defined by the authors and domain experts after inspecting the corpus (Section 3.1, Table 6); no external validation or evidence that these categories are stable across raters beyond the in-corpus Kappa.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis." pith.science (2026). https://pith.science/paper/62CPBSYU

@misc{pith2026250106101,
  author       = {Pith},
  title        = {Pith review of: From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/62CPBSYU}},
  note         = {Machine review of arXiv:2501.06101}
}

read the original abstract

Problem-solving therapy (PST) is a structured psychological approach that helps individuals manage stress and resolve personal issues by guiding them through problem identification, solution brainstorming, decision-making, and outcome evaluation. As mental health care increasingly adopts technologies like chatbots and large language models (LLMs), it is important to thoroughly understand how each session of PST is conducted before attempting to automate it. We developed a comprehensive framework for PST annotation using established PST Core Strategies and a set of novel Facilitative Strategies to analyze a corpus of real-world therapy transcripts to determine which strategies are most prevalent. Using various LLMs and transformer-based models, we found that GPT-4o outperformed all models, achieving the highest accuracy (0.76) in identifying all strategies. To gain deeper insights, we examined how strategies are applied by analyzing Therapeutic Dynamics (autonomy, self-disclosure, and metaphor), and linguistic patterns within our labeled data. Our research highlights LLMs' potential to automate therapy dialogue analysis, offering a scalable tool for mental health interventions. Our framework enhances PST by improving accessibility, effectiveness, and personalized support for therapists.

Figures

Figures reproduced from arXiv: 2501.06101 by the authors.

**Figure 2.** Distribution of Autonomy in PS Core 0 0.5 1 Therapeutic Engagement Social Courtesies Test Review Session Management Autonomy Directive Non-Directive Ratio(%) Facilitative Strategies [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 5.** Proportion of Metaphors in Facilitators visit1 visit2 visit3 0 0.2 0.4 0.6 0.8 1 PS Core Trying Out Solution Plan Outcome Prediction and Planning Generating Alternative Solutions Defining Problems and Goals Problem-solving Positive Mindset Visit Number Frequency of PS core labels [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

Figures from the paper (4 more)

**Figure 6.** Figure 6: Distribution of PS Core in three visits visit1 visit2 visit3 0 0.2 0.4 0.6 0.8 1 Facilitators Session Management Test Review Social Courtesies Therapeutic Engagement Visit Number Frequency of Facilitative labels [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 8.** Figure 8: Co-Occurrence of PS core Strategies with Selected LIWC Features [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: Co-Occurrence of Facilitative Strategies with Selected LIWC Features [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

**Figure 10.** Figure 10: Co-occurrance of Question Type and Auton [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

[1]

Directive

Autonomy-Supportive Language Output Labels: "Directive", "Non-Directive", or "N/A". Definitions: - Directive: - Focuses on the therapist’s goals, plans, or strategies for the client. - Therapist explicitly tells the client what to do or prescribes a solution. - Prioritizes achieving a specific outcome, often determined by the therapist. - May reflect the ...

work page
[2]

Immediate

Self-Disclosure Output Labels: "Immediate", "Nonimmediate", or "N/A". Definitions: Self-disclosure refers to instances when the therapist intentionaly shares personal information, experiences, or feelings with the client. - Immediate Self-Disclosure: Sharing personal thoughts, feelings, or reactions related to the ongoing therapeutic interaction. - Nonimm...

work page
[3]

In Machine Learning for Health (ML4H), pages 82–100

Llms accelerate annotation for medical infor- mation extraction. In Machine Learning for Health (ML4H), pages 82–100. PMLR. Keith Harrigian, Carlos Aguirre, and Mark Dredze

work page
[4]

yes, <reasoning for metaphor detection>

Metaphors: Output Labels: "yes, <reasoning for metaphor detection>" (for use of metaphor) or "no" (for no use of metaphor). s Definition: According to conceptual metaphor theory, metaphor facilitates a mapping of attributes or characteristics from source domain to 15 target domain. For example, the word “invested” in the sentence “I have invested a lot of...

work page
[5]

COMPASS: Computational Mapping of Patient-Therapist Alliance Strategies with Language Modeling

Compass: Computational mapping of patient- therapist alliance strategies with language modeling. arXiv preprint arXiv:2402.14701. Yilun Liu, Shimin Tao, Weibin Meng, Feiyu Yao, Xi- aofeng Zhao, and Hao Yang. 2024. Logprompt: Prompt engineering towards zero-shot and inter- pretable log analysis. In Proceedings of the 2024 IEEE/ACM 46th International Confer...

work page Pith review arXiv 2024
[9]

International Journal of Social Psychiatry, 69(8):1882–1889

Old dog, new tricks? exploring the potential functionalities of chatgpt in supporting educational methods in social psychiatry. International Journal of Social Psychiatry, 69(8):1882–1889. Ruijun Song and Nana Sun. 2019. The efficacy of psy- chological capital intervention (pci) for depression from the perspective of positive psychology: a pilot study. Fr...

work page 2019
[10]

medRxiv, pages 2023–08

Large language models (llms) and empathy-a systematic review. medRxiv, pages 2023–08. Xin Sun, Jos A. Bosch, Jan De Wit, and Emiel Krahmer

work page 2023
[11]

uh, they suggest trying to think of prob- lems as challenges rather than threats

Human-in-the-loop interaction for continu- ously improving generative model in conversational agent for behavioral intervention. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23 Companion, page 99–101, New York, NY , USA. Association for Computing Machinery. Yi Tang, Chia-Ming Chang, and Xi Yang. 2024. ...

arXiv 2024

Show all 15 references

[14]

Open-Ended

Open vs. Closed Questions Output Labels: "Open-Ended", "Closed-Ended", or "N/A". Definitions: - Open-Ended Questions: Questions that encourage detailed, thoughtful, and expansive responses. These questions often begin with "how," "what," "why," or "tell me about" and invite th...
[2001]

Journal of Counseling & Development, 79(3):269–274

Metaphor and change in counseling. Journal of Counseling & Development, 79(3):269–274. Zilin Ma, Yiyang Mei, and Zhaoyuan Su. 2023. Under- standing the benefits and challenges of using large 11 language model-based conversational agents for men- tal well-being support. In AMIA...

2023
[2013]

Psychology and Psychotherapy: Theory, Research and Practice, 86(3):294–314

How collaboration in therapy becomes thera- peutic: The therapeutic collaboration coding system. Psychology and Psychotherapy: Theory, Research and Practice, 86(3):294–314. Raj Sanjay Shah, Faye Holt, Shirley Anugrah Hayati, Aastha Agarwal, Yi-Chia Wang, Robert E. Kraut, and D...

2022 arXiv
[2020]

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen

Do models of mental health based on social media data generalize? In Findings of the associ- ation for computational linguistics: EMNLP 2020 , pages 3774–3788. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled at...

2020 arXiv
[2021]

In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1–7

Sad: A stress annotated dataset for recog- nizing everyday stressors in sms-like conversational systems. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1–7. Maitrey Mehta, Derek Caperton, Katherine Axford, Lau- ren Weitzman, David...

2021
[2023]

Proceedings of the National Academy of Sciences, 120(30):e2305016120

Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. 10 Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, et al
[2024]

Journal of Machine Learning Research, 25(70):1–53

Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022. Scientific and creative analogies in pretrained language mod- els. In Findings of the Associati...

2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Directive

Autonomy-Supportive Language Output Labels: "Directive", "Non-Directive", or "N/A". Definitions: - Directive: - Focuses on the therapist’s goals, plans, or strategies for the client. - Therapist explicitly tells the client what to do or prescribes a solution. - Prioritizes achieving a specific outcome, often determined by the therapist. - May reflect the ...

work page

[2] [2]

Immediate

Self-Disclosure Output Labels: "Immediate", "Nonimmediate", or "N/A". Definitions: Self-disclosure refers to instances when the therapist intentionaly shares personal information, experiences, or feelings with the client. - Immediate Self-Disclosure: Sharing personal thoughts, feelings, or reactions related to the ongoing therapeutic interaction. - Nonimm...

work page

[3] [3]

In Machine Learning for Health (ML4H), pages 82–100

Llms accelerate annotation for medical infor- mation extraction. In Machine Learning for Health (ML4H), pages 82–100. PMLR. Keith Harrigian, Carlos Aguirre, and Mark Dredze

work page

[4] [4]

yes, <reasoning for metaphor detection>

Metaphors: Output Labels: "yes, <reasoning for metaphor detection>" (for use of metaphor) or "no" (for no use of metaphor). s Definition: According to conceptual metaphor theory, metaphor facilitates a mapping of attributes or characteristics from source domain to 15 target domain. For example, the word “invested” in the sentence “I have invested a lot of...

work page

[5] [5]

COMPASS: Computational Mapping of Patient-Therapist Alliance Strategies with Language Modeling

Compass: Computational mapping of patient- therapist alliance strategies with language modeling. arXiv preprint arXiv:2402.14701. Yilun Liu, Shimin Tao, Weibin Meng, Feiyu Yao, Xi- aofeng Zhao, and Hao Yang. 2024. Logprompt: Prompt engineering towards zero-shot and inter- pretable log analysis. In Proceedings of the 2024 IEEE/ACM 46th International Confer...

work page Pith review arXiv 2024

[6] [9]

International Journal of Social Psychiatry, 69(8):1882–1889

Old dog, new tricks? exploring the potential functionalities of chatgpt in supporting educational methods in social psychiatry. International Journal of Social Psychiatry, 69(8):1882–1889. Ruijun Song and Nana Sun. 2019. The efficacy of psy- chological capital intervention (pci) for depression from the perspective of positive psychology: a pilot study. Fr...

work page 2019

[7] [10]

medRxiv, pages 2023–08

Large language models (llms) and empathy-a systematic review. medRxiv, pages 2023–08. Xin Sun, Jos A. Bosch, Jan De Wit, and Emiel Krahmer

work page 2023

[8] [11]

uh, they suggest trying to think of prob- lems as challenges rather than threats

Human-in-the-loop interaction for continu- ously improving generative model in conversational agent for behavioral intervention. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23 Companion, page 99–101, New York, NY , USA. Association for Computing Machinery. Yi Tang, Chia-Ming Chang, and Xi Yang. 2024. ...

arXiv 2024