{"id":"25d09c73-0295-48d9-b370-f5c9779c17c6","arxiv_id":"2501.06101","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-4o annotates problem-solving therapy strategies in real transcripts with 0.76 weighted F1, and the resulting labels reveal a shift from exploratory to implementation-focused strategies as sessions progress.","lead":"This paper tests whether large language models can automatically label therapist strategies in real problem-solving therapy sessions, finding GPT-4o reaches the best weighted F1 of 0.76. The authors add new 'facilitative' strategy categories and use the labels to show how therapists shift from exploratory to action-oriented techniques across sessions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.76 F1 rests on a 500-utterance gold standard with no independent adjudication, and Table 3's support counts sum to 525 rather than 500, so the headline metric and all downstream GPT-4o labels inherit an unquantified and possibly ill-defined evaluation unit.","rationale":"The reader identified the small single-expert gold standard with no adjudication as the weakest assumption, and I agree that this is the most load-bearing point: the entire empirical contribution and all downstream analyses depend on the trustworthiness of those 500 labels. My stress-test adds a precise internal inconsistency that strengthens the concern: Table 3's support counts sum to 525 while the text claims 500 utterances, and the annotation schema permits two labels per utterance, so the evaluation unit is undefined. This is not just a typo; it affects how the weighted F1 should be computed and interpreted. The concern is serious but not fatal in itself. If the authors release the gold labels, add a second independent expert's annotations with adjudication, and report corrected per-class F1 with confidence intervals, the central claim could be either confirmed or rejected. The reader's conditional verdict remains appropriate: the paper's contribution is plausible and the evaluation is meaningful, but the headline number and all downstream inferences are not yet sufficiently verified. I do not see a reason to move to accept or reject based on the current text alone; the right disposition is to require the described checks. I therefore leave the reader's verdict unchanged.","tokens_in":19552,"tokens_out":5922,"duration_ms":64123,"concrete_test":"Have a second PST domain expert, not otherwise involved in the study, independently annotate the same 500 utterances with the published codebook. Then adjudicate disagreements between the original labels and the new expert to construct a reference gold standard, and recompute GPT-4o's class-wise and weighted F1 against it, with bootstrap confidence intervals. In the same release, clarify whether the evaluation unit is utterances or label instances by reporting the total number of unique utterances and the total number of labels; if the correct protocol is multi-label, recompute F1 separately for PS Core and Facilitators rather than as a single 525-instance set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GPT-4o can annotate PST strategies well enough to support the paper's downstream analyses, and every downstream result is built on the 14,417 utterances labeled by that model. The only direct evidence for the model's quality is the weighted F1 of 0.76 on 500 hand-annotated utterances, so the integrity of that gold standard is load-bearing. Two specific problems make the metric hard to interpret. First, Section 3.1 says annotation was performed by 'one domain expert alongside one researcher,' with Cohen's Kappa reported per class, but no adjudication procedure, no second independent expert, and no description of how disagreements were resolved into the final gold labels. If the final labels are essentially one expert's judgments, label noise is unmeasured and the reported F1 could be biased in either direction. Second, Table 3 lists support counts that sum to 525 (PS Core: 39+66+36+33+18 = 192; Facilitators: 25+64+130+24 = 243; None: 90; total 525), yet the paper consistently says the evaluation set contains 500 utterances. The annotation instructions allow up to two labels per utterance (one PS Core, one Facilitator), so a flattened 10-class evaluation should contain either 500 instances (if one merged label per utterance) or 1,000 instances (if PS Core and Facilitator labels are evaluated separately), but not 525. Without knowing which unit the F1 is computed on, the reported 0.76 cannot be interpreted as an utterance-level accuracy, and the per-class weights in Table 3 are correspondingly ambiguous. Because the same GPT-4o labels are then used to quantify strategy frequencies (Table 5), strategic progression across visits (Figures 6-7), and correlations with autonomy, self-disclosure, and metaphor, any error in the gold standard or ambiguity in the evaluation unit propagates into all of those analyses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an annotation framework for therapist utterances in problem-solving therapy (PST), combining five established PST Core Strategies with four newly proposed \"Facilitative Strategies\" (Social Courtesies, Session Management, Therapeutic Engagement, Test Review). The authors evaluate four LLMs (GPT-4, GPT-4o, Llama-3.1-70B, Yi-1.5-34B) and four transformer-based classifiers on 500 human-annotated therapist utterances from 240 real-world sessions, reporting that GPT-4o achieves the highest weighted F1 (0.76) and using GPT-4o to label all 14,417 in-scope utterances. On this labeled corpus they analyze strategy prevalence, therapeutic dynamics (autonomy, self-disclosure, metaphor), LIWC-based linguistic patterns, and strategy progression across the first three visits. They additionally fine-tune DeBERTa, MentalBERT, ModernBERT, and FLAN-T5 on 5,000 GPT-4o-labeled utterances and report competitive F1 scores.","tokens_in":19923,"tokens_out":7246,"duration_ms":65392,"significance":"If the headline result holds, the paper offers a useful, scalable pipeline for automating PST process annotations: the use of real-world transcripts (rather than social media or simulated data), the extension of the PST codebook with expert-informed Facilitative Strategies, and the explicit comparison of proprietary and open-weight LLMs and distilled transformer students are all valuable. The paper also ships a deterministic, entropy-checked prompting setup and evaluates on a human gold standard rather than in a self-referential loop. However, the strength of the claim is currently bounded by the small, unadjudicated gold standard, the ambiguous evaluation unit, the absence of confidence intervals, and the unvalidated use of GPT-4o labels for all downstream distributional and dynamics analyses. These issues are fixable and do not undermine the interest of the framework.","major_comments":[{"comment":"The evaluation set is described as 500 therapist utterances, but the support column in Table 3 sums to 525 instances (PS Core: 39+66+36+33+18; Facilitators: 25+64+130+24; None: 90). Since the annotation instructions allow one PS Core and one Facilitator label per utterance, the unit of evaluation is ambiguous: the weighted F1 could be computed over merged utterance-level labels, over the two annotation dimensions separately, or over a flattened set that yields 525 labels. The paper must state the evaluation unit explicitly, report per-dimension and overall metrics consistently, and provide confidence intervals for the 0.76 figure. As written, the headline metric cannot be interpreted as utterance-level accuracy.","section":"Section 3.1 / Table 3"},{"comment":"The gold standard was produced by \"one domain expert alongside one researcher,\" with per-class Cohen's Kappa, but the paper does not describe any adjudication process, a second independent expert, or how disagreements were resolved into the final labels. Inter-annotator agreement between these two coders does not, by itself, validate the final gold labels against a reference; if the final set reflects one expert's judgments, label noise is unmeasured and the reported F1 could be biased in either direction. The authors should report the adjudication protocol, ideally have a second expert review the disagreements, and quantify the robustness of the reported F1 to alternative gold-label resolutions.","section":"Section 3.1, Data Annotation"},{"comment":"All downstream results—strategy prevalence, autonomy/self-disclosure/metaphor analyses, and the visit-by-visit progression—are computed from GPT-4o labels on the full 14,417 utterances, and the only direct validation of those labels is the 500-utterance gold set. The paper reports no stratified evaluation of the full-corpus label distribution, no per-class error analysis beyond Table 3, and no uncertainty quantification. In particular, the closing sentence of Section 4.5—that the observed patterns \"further validate the effectiveness of our classifier\"—is circular when the patterns are derived from the classifier's own labels. Please validate full-corpus annotations on a held-out human-annotated sample (e.g., a stratified sample by strategy and visit) or explicitly reframe Sections 4.3-4.5 as exploratory model-generated findings.","section":"Sections 4.3-4.5 (Figures 2-7, Table 5)"},{"comment":"The therapeutic-dynamics labels used in Section 4.3—autonomy (directive/non-directive), self-disclosure (immediate/non-immediate), question type, and metaphor presence—are generated by GPT-4o with no human-annotated evaluation of these dimensions. Figures 2-5 and 10 and the percentage claims in Section 4.3 (e.g., 94.49% of directive utterances being closed-ended) therefore inherit unquantified model error. The authors should either evaluate the dynamics annotation on a small human-annotated sample or present these analyses as illustrative rather than as validated measurements.","section":"Section 3.3 / Appendix D.2"}],"minor_comments":[{"comment":"The abstract states that GPT-4o achieved \"the highest accuracy (0.76)\"; all reported metrics are F1 scores, so this should read \"highest weighted F1\" to avoid conflating accuracy with F1.","section":"Abstract"},{"comment":"The data provider is identified only as \"[Anonymized collaborator]\"; please name the source or provide a fuller data-use description so readers can assess provenance and external validity.","section":"Section 3.1"},{"comment":"Appendix C lists the GPT-4o model as \"gpt-4o-2024-05-13\" while Section 3.2 states \"gpt-4o-2024-08-06\"; the model identifier should be consistent.","section":"Appendix C / Section 3.2"},{"comment":"The statement that \"all models showed lower F1 scores when provided with additional context\" is not strictly supported by Table 1: GPT-4's Facilitators score is 0.65 in both conditions. Please qualify the claim to the affected settings.","section":"Section 4.1"},{"comment":"The exclusion of utterances shorter than five words is justified by an assumption that longer utterances are more meaningful; please report a sensitivity analysis of the main prevalence and progression results to this threshold, since the threshold defines the entire analysis corpus.","section":"Section 3.1"},{"comment":"Because the models were run at temperature 0, one would expect identical labels across five runs; the nonzero mean entropy values need a brief explanation (e.g., API nondeterminism) to be interpretable.","section":"Table 2"},{"comment":"The co-occurrence matrix reports percentages within autonomy rows, but the caption does not state whether row or column normalization is used; please clarify the normalization and the denominator for the quoted percentages.","section":"Figure 10 / Appendix G"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a CL/NLP venue and the empirical core is plausible. My main reservation is that the evaluation evidence is thinner than the downstream claims require: the 500-utterance gold set is small, lacks adjudication, and the Table 3 support count is not aligned with the stated sample size. These issues are fixable in revision, and I would not reject on methodological disagreement alone. I would ask for a stronger validation section and a clearer definition of the evaluation unit before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has two contributions worth talking about. First, the new Facilitative Strategies codebook (four categories added to the standard PST Core) is a practical addition that other therapy-annotation work can reuse. Second, the corpus itself—240 real sessions with 14,417 therapist utterances—is a solid resource, even though the transcripts are not public. The comparison across four LLMs and four transformer models is thorough, and the finding that a fine-tuned encoder (ModernBERT, 0.8 PS Core F1) can beat a strong LLM on the distillation task is genuinely useful for privacy-conscious deployment.\n\nThe soft spot is precisely where the stress-test lands. The headline 0.76 weighted F1 is measured on 500 hand-annotated utterances, but Table 3 reports support counts that sum to 525. This isn't a typo; it reflects an ambiguous evaluation unit. If annotators could assign one PS Core and one Facilitator per utterance, then the flattened 10-class evaluation should have either 500 instances (one merged label per utterance) or 1,000 (the two dimensions evaluated separately). Getting 525 means some utterances are double-counted and some are not, so the weighted average is over an ill-defined set. Until the authors clarify exactly which labels went into the computation, the 0.76 cannot be interpreted as an utterance-level accuracy. That ambiguity also undermines the per-class precision and recall in Table 3 and propagates into the downstream session-progression and therapeutic-dynamics analyses, which use GPT-4o's full-corpus labels without adjusting for measurement error.\n\nThe gold standard is a second real concern. The annotation was done by one domain expert and one researcher, with Cohen's Kappa reported per class but no adjudication or resolution protocol described. The final labels are effectively one expert's judgments, so label noise is unmeasured and the F1 could be biased in either direction. This is not fatal—many annotation studies rely on a single expert—but it should be acknowledged and, ideally, a second independent expert should validate at least a subset.\n\nOn the positive side, the paper is candid about its limitations and does not overclaim clinical readiness. The Facilitative Strategies were developed on the same corpus, but the evaluation set is separate, so the circularity burden is mild. The paper does not release data or code, which limits reproducibility but is understandable for therapy data.\n\nWho gets value from this? Researchers working in computational mental health, clinical NLP, or PST process research. The identified issues are fixable: clarify the evaluation unit, add adjudication details or a second expert, and state that downstream analyses inherit label noise. I would send it to peer review with a request for major revision; the core contribution is solid enough to deserve another round.","headline":"Useful new codebook and realistic corpus, but the headline F1 is computed on an ambiguous evaluation unit and a single-expert gold standard; re-examine before trusting the downstream numbers.","tokens_in":20517,"tokens_out":3087,"would_cite":false,"duration_ms":32138,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that zero-shot GPT-4o, prompted with a codebook of five PST core strategies and four newly added facilitative strategies, can label therapist utterances in real-world problem-solving therapy transcripts at a weighted F1…","keywords":["problem-solving therapy","LLM annotation","GPT-4o","therapy dialogue analysis","therapeutic dynamics","PST strategies","mental health NLP","transformer fine-tuning"],"falsifier":"Take a fresh sample of, say, 500 utterances from the same corpus and have two PST experts with documented expertise annotate them independently, then reconcile disagreements to create an adjudicated gold set. If GPT-4o's weighted F1 against this independent gold drops well below 0.76, or if expert-expert agreement is substantially lower than GPT-4o's agreement with the original labels, the claim that GPT-4o reliably annotates PST strategies would be refuted.","tokens_in":19354,"feed_emoji":"🧠","tokens_out":5823,"duration_ms":51065,"temperature":0.7,"pith_summary":"The paper tries to show that large language models can take over the fine-grained labeling of what therapists actually do in problem-solving therapy sessions, turning unstructured transcripts into structured data. The authors build an annotation framework from the five-step ADAPT model of PST and add a new set of four facilitative strategies, then prompt GPT-4o to label 14,417 real therapist utterances. GPT-4o reaches a weighted F1 of 0.76, a combined precision-and-recall score, against a 500-utterance human-annotated gold set, outperforming GPT-4, Llama-3.1, and Yi-1.5, and its labels are stable across runs (mean entropy 0.035). On top of the labels, the paper analyzes autonomy, self-disclosure, and metaphor use, finding that therapists shift from non-directive, metaphor-rich exploration early in therapy to directive, concrete implementation language later. If the labeling is trustworthy, the framework offers a scalable way to study and eventually guide PST delivery, including real-time feedback for therapists and fidelity checks for AI therapy tools.","feed_headline":"GPT-4o tags problem-solving therapy strategies at 0.76 F1","feed_subtitle":"Zero-shot annotations of 14,417 real therapist utterances open scalable analysis of how PST is delivered.","key_machinery":"The load-bearing object is the annotation codebook: five PS Core strategies taken from the ADAPT model of problem-solving therapy, extended with four new Facilitative strategies that capture rapport-building, session logistics, engagement, and test review. The machinery is zero-shot prompting of GPT-4o with strategy definitions and few-shot examples at temperature zero, plus an entropy check over five repeated runs to confirm label stability (mean entropy 0.035). The same prompting approach is then reused for therapeutic dynamics, asking GPT-4o to classify autonomy, self-disclosure, question type, and metaphor with source and target domains, and LIWC is applied to the labeled utterances to connect strategies to linguistic features. The codebook carries the argument because F1 is measured strategy by strategy, and the downstream progression and dynamic analyses inherit the labels.","core_discovery":"The central claim is that automated annotation of PST is achievable with current LLMs. Using a codebook with five PS Core strategies from the ADAPT model (positive mindset, defining problems and goals, generating alternative solutions, outcome prediction and planning, trying out solution plan) and four novel Facilitative strategies (social courtesies, session management, therapeutic engagement, test review), the authors prompted GPT-4o at zero temperature to classify each therapist utterance. Without any conversational context, GPT-4o achieved a weighted F1 of 0.76, with per-strategy F1 scores from 0.68 to 0.92; adding two prior utterances lowered performance to 0.66 because the model sometimes labeled the context instead of the target utterance. The authors then used GPT-4o's labels to annotate all 14,417 utterances and found a clear progression across visits: early sessions emphasize positive mindset and problem definition, later visits shift to generating alternatives and planning, and the final visit emphasizes trying out the solution plan. They also prompted GPT-4o to label directive versus non-directive autonomy, immediate versus non-immediate self-disclosure, and metaphor use, and they report that non-directive language and metaphors dominate early stages while directive, concrete language grows later. Fine-tuned transformer models trained on 5,000 GPT-4o-labeled utterances reach competitive F1 scores (ModernBERT 0.80 for PS Core, DeBERTa 0.73 for Facilitators), supporting a distillation path to smaller, privacy-conscious models.","pith_inferences":["The same codebook-plus-prompt recipe could be adapted to other structured therapies such as CBT or interpersonal therapy, since the Facilitative strategies are largely therapy-agnostic; the paper does not test this.","The finding that self-disclosure appears in only 257 utterances and is mostly non-immediate suggests a quantitative fidelity marker: a therapist disclosing frequently or immediately may be deviating from typical PST practice, and this could be monitored automatically.","If context degrades performance, a promising testable extension is to give the model the preceding strategy label or a short summary of the session instead of raw previous utterances, which might improve coherence without the observed label-leakage errors.","A direct application the authors do not develop: run the same pipeline on transcripts from LLM-based mental health chatbots to measure how faithfully they follow PST, providing an audit mechanism for AI therapy products."],"forward_implications":["The full corpus of 14,417 therapist utterances is now labeled with PS Core and Facilitative strategies, enabling quantitative study of how PST is actually delivered across visits.","Session-level strategy progression is observable: therapists start with mindset and problem definition, move to alternative solutions and planning, and emphasize trying out the solution plan in the final visit, with Test Review concentrated in the first visit.","Adding conversational context to the prompt currently hurts rather than helps GPT-4o (F1 0.66 vs 0.76), so future work needs better context-grounding rather than simply feeding more dialogue history.","Small fine-tuned models trained on LLM labels can reach competitive accuracy (ModernBERT 0.80 on PS Core; DeBERTa 0.73 on Facilitators), suggesting a path to local, privacy-preserving annotation of therapy data.","The framework can serve as an evaluation tool for LLM-assisted therapy systems, checking whether automated interventions actually apply PST strategies and maintain therapeutic alliance."],"supporting_citations":[{"why":"Supplies the ADAPT five-step PST core strategies that form the PS Core category of the annotation codebook.","marker":"Demiris et al., 2019"},{"why":"The PST treatment manual whose client-centered problem-solving approach the authors use to interpret autonomy and strategy progression.","marker":"Nezu et al., 2012"},{"why":"One of the therapy-conversation codebooks consulted when developing the new Facilitative strategies.","marker":"Li et al., 2023a"},{"why":"Prior work using LLMs to classify counselor utterances, the direct predecessor this paper extends.","marker":"Lee et al., 2024"},{"why":"Scoping review documenting the scarcity of real-world, expert-validated therapy datasets, motivating the use of real transcripts.","marker":"Hua et al., 2024"},{"why":"Provides LIWC-22, the tool used for the linguistic-pattern analysis of labeled utterances.","marker":"Boyd et al., 2022"},{"why":"Conceptual Metaphor Theory scaffolding used in the prompts for metaphor detection with source and target domains.","marker":"Tian et al., 2024"},{"why":"Prior proxy for autonomy using open- versus closed-ended questions, which the paper refines into directive and non-directive categories.","marker":"Shah et al., 2022"},{"why":"Establishes the zero- and few-shot prompting paradigm that the LLM annotation approach relies on.","marker":"Brown et al., 2020"}],"fun_headline_variants":["GPT-4o maps therapy strategies from conversations","LLMs automate problem-solving therapy analysis","Zero-shot GPT-4o labels therapist dialogue at 0.76 F1","Study: GPT-4o tags PST strategies without fine-tuning","Automating PST annotation: GPT-4o scores 0.76 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation rests on the 500 therapist utterances that one domain expert and one researcher labeled by hand; that gold standard was not adjudicated or checked against a second independent expert, so any mistakes or biases in it are baked into the reported F1 and into every label GPT-4o subsequently assigned to the full corpus.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o maps therapy strategies from conversations","LLMs automate problem-solving therapy analysis","Zero-shot GPT-4o labels therapist dialogue at 0.76 F1","Study: GPT-4o tags PST strategies without fine-tuning","Automating PST annotation: GPT-4o scores 0.76 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1443,"prompt_tokens":1047,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":310}},"tokens_in":663,"tokens_out":396,"duration_ms":4257,"temperature":1.0,"reasoning_tokens":310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:35.614414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of, say, 500 utterances from the same corpus and have two PST experts with documented expertise annotate them independently, then reconcile disagreements to create an adjudicated gold set. If GPT-4o's weighted F1 against this independent gold drops well below 0.76, or if expert-expert agreement is substantially lower than GPT-4o's agreement with the original labels, the claim that GPT-4o reliably annotates PST strategies would be refuted.","supporting_citations":[],"review_version":1}