{"id":"7200fd79-e355-4cbb-a469-d2a87e217a0b","arxiv_id":"2507.20241","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A stage-planning LLM therapist, INT, and an Innovative Moment metric, IMA, improve narrative therapy dialogue quality over direct LLM role-playing in simulated and human evaluations.","lead":"This paper describes a large language model system, INT, that runs narrative therapy sessions by planning therapeutic stages and reflection depth, plus an evaluation metric, IMA, that tracks innovative moments in client language. It reports that 230 human users rated INT higher than standard LLM role-players on three of four therapy dimensions, with caveats about missing statistics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IMA salience denominator includes therapist words, so INT's deliberately shorter replies inflate its reported IM percentages; the 'nearly doubling' of Level-2 IMs may be a response-length artifact.","rationale":"The reader's weakest assumption is planning-state accuracy, which is a real issue: Ψ_S and Ψ_L are never validated against expert judgment, so the causal role of progression-aware planning is not established. However, it is not the most decisive problem for the headline. Even if planning states are noisy, the ablations and the human Likert ratings could still support part of the contribution. The more serious issue is that the IMA metric used to support the 'nearly doubling advanced IMs' claim is structurally confounded with therapist verbosity. Because the denominator in Eq. (5) is the full turn including therapist words, the metric rewards systems that say less. INT is explicitly prompted to be terse—'no more than three sentences' and one question—and Fig. 4 confirms much shorter responses than GPT-4o role-play. The back-of-envelope calculation shows the large percentage advantage shrinks to near parity when salience is computed over client words only. This is a structural property of the metric, not an accusation of bad faith. The human ratings of therapeutic dimensions are not subject to this artifact, so the global claim of better therapeutic quality and depth may survive; but the specific IMA-based headline claim would need reanalysis. I therefore keep the conditional verdict while adding the condition that IMA salience must be recomputed on a client-only denominator or with matched verbosity.","tokens_in":23422,"tokens_out":6796,"duration_ms":84113,"concrete_test":"Recompute Table 3 IMA salience with the denominator restricted to client words, Σ_t WordCount(C_t), and also report absolute IM word counts per session for every system. Additionally, run a verbosity-matched comparison where GPT-4o role-play replies are truncated to INT's average length. If INT's Level-2 advantage drops from about 1.8x to 1.3x or below, the 'nearly doubling' claim is primarily a response-length artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for INT's superiority is the IMA salience claim. Eq. (5) defines Salience(I_i) = Σ_t WordCount(C_t ∩ I_i) / Σ_t WordCount(C_t ∪ T_t), so the denominator includes therapist words. In the human evaluation (Table 3, Fig. 4a), INT's average therapist response is 66.1 words while GPT-4o's role-play is 113.0 words, with client turns of 38.8 vs 31.5 words. This gives approximate per-turn denominators of 104.9 vs 144.5. The reported Level-2 SUM is 19.408% for INT vs 10.826% for GPT-4o, about 1.8x. Scaling by those denominators yields roughly 20.4 vs 15.6 IM words per turn—only about 1.3x. Recomputing with a client-only denominator gives approximately 52.5% vs 49.5%, near parity. The paper neither reports absolute IM word counts nor client-only salience, so the 'nearly doubling' headline and the IMA-based demonstration that INT elicits advanced narrative transformation may be an artifact of therapist brevity rather than of client narrative change. Since IMA is a core contribution, this confound is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes INT, a planning-based LLM system that simulates narrative therapists by tracking therapeutic stages and reflection levels and using retrieval-augmented generation, together with IMA, an evaluation method that operationalizes the Innovative Moments Coding System to measure narrative transformation in client speech. The authors report automated experiments with 260 simulated clients and human experiments with 200 main plus 30 follow-up participants, claiming that INT outperforms standard role-playing LLMs across therapeutic dimensions and nearly doubles the elicitation of advanced (Level 2) innovative moments. They additionally fine-tune Qwen3-8B on an INT-generated NTConv dataset and report improvements over ESConv on automatic and human metrics.","tokens_in":23673,"tokens_out":5613,"duration_ms":64851,"significance":"If the central claims hold, this is a useful step toward theory-grounded, process-aware therapeutic dialogue systems and toward automatic, therapy-grounded evaluation. The strengths of the paper are the external grounding of IMA in the IMCS coding system, the use of human expert annotators with reported inter-rater reliability (Cohen's kappa > 0.75), the relatively large human study (230 participants), and the planned release of code and data. However, the headline quantitative claim that INT 'nearly doubles' advanced innovative moments is currently confounded by the definition of the salience metric, which includes therapist words in the denominator, and the automated evaluation is partly self-referential because GPT-4o plays the client, the therapist, and the supervisor. These issues make the magnitude of the reported improvements uncertain rather than definitively wrong, so the paper merits major revision.","major_comments":[{"comment":"The IMA salience denominator includes therapist words: Salience(I_i) = Σ WordCount(C_t ∩ I_i) / Σ WordCount(C_t ∪ T_t). Because INT produces deliberately shorter therapist turns (66.1 vs. 113.0 words per turn in Fig. 4a), the reported Level-2 SUM is inflated. For the human evaluation, INT's Level-2 SUM is 19.408% and GPT-4o's is 10.826%, a ratio of about 1.79, which underlies the 'nearly doubling' claim. Using the average turn lengths in Fig. 4(a), the approximate IM word counts per turn are 0.19408 × (38.8 + 66.1) ≈ 20.4 for INT and 0.10826 × (31.5 + 113.0) ≈ 15.6 for GPT-4o, a ratio of about 1.30; with a client-only denominator the salience figures become approximately 52.5% and 49.7%, near parity. The paper does not report absolute IM word counts or client-only salience, so the central demonstration of INT's superiority in eliciting advanced narrative markers is not yet supported. Please reanalyze the IMA results with client-only denominators or absolute counts, and temper the abstract claim accordingly.","section":"§4.1.2, §4.1.3, Table 2"},{"comment":"The automated evaluation is partly self-referential: GPT-4o simulates the client, GPT-4o implements INT's planning and response generation, GPT-4o acts as the counseling supervisor for dimension scoring, and GPT-4o performs IM annotation. Since INT is built on GPT-4o prompts, the automated comparison may reward responses that match GPT-4o's own stylistic preferences rather than genuine therapeutic quality. The paper reports no agreement study between the GPT-4o supervisor/annotator and human experts on the same automated dialogues. I recommend reporting human--LLM annotation agreement on a shared sample and varying the client and supervisor models to ensure the Table 2 conclusions are not an artifact of model self-preference.","section":"§4.2, Table 3"},{"comment":"The claim in the abstract and §4.2 that INT 'consistently outperforms standard LLMs in therapeutic quality and depth' is overstated: in the human evaluation, INT's Reassuring score is 3.09, below GPT-4o's 3.11 and Claude-3.7-sonnet's 3.08. The paper reports no error bars or significance tests for any of the human-rated dimensions, so the remaining differences, while directionally favorable to INT on Empowering, Transformative, and Reconnecting, are not shown to be statistically reliable. Please report per-participant variance, confidence intervals, and appropriate significance tests, or qualify the claims.","section":"§4.1.3, Table 3"},{"comment":"The human expert IM annotations cover only 10 randomly sampled sessions per system. The IMA percentages in Table 3 are therefore based on n = 10 per condition, and no per-session variance or confidence intervals are reported. Given the high variability of client narratives, a difference in Level-2 SUM of 19.4% versus 10.8% could be driven by a few unusual sessions. Please provide per-session distributions, bootstrap confidence intervals, or a mixed-effects analysis for the IMA comparisons.","section":"§3.2.2, Eqs. (1)-(2), Fig. 4(b)"},{"comment":"The accuracy of the stage planner Ψ_S and reflection-level planner Ψ_L is never measured against expert judgment. The paper reports stage and reflection-level distributions in Fig. 4(b), but not whether these inferred states match what a trained narrative therapist would choose. Because the ablations (w/o RAG and w/o RAGRL) change both the planning signal and the response-generation prompt, the observed improvement cannot be unambiguously attributed to progression-aware planning. I recommend evaluating planning accuracy on a labeled subset, or running an oracle-condition experiment in which the stage and reflection level are supplied by an expert or by a different model.","section":"§4.2"}],"minor_comments":[{"comment":"There is a missing space in 'reassuringclients' in the first paragraph; it should read 'reassuring clients'.","section":"Introduction, first paragraph"},{"comment":"The subscript in 'lt t' is malformed and should be a single subscripted variable, e.g., l_t, to avoid confusion with the turn index t.","section":"§3.2.2, Eq. (2)"},{"comment":"The sentence 'both evaluation approaches confirm INT's superiority in facilitating narrative transformation' is imprecise because INT's overall IM salience in Table 2 (35.064%) is lower than Doubao-1.5-pro's (42.110%); the claim should specify that INT shows the highest Level-2 salience in the automated setting.","section":"§4.2, Table 2"},{"comment":"The sentence describing the IM classification code book contains the typo 'IM typs' and should be 'IM types'.","section":"Appendix D"},{"comment":"The caption of Fig. 4(a) does not explain which color corresponds to which variant or which bar group corresponds to which metric; the figure is difficult to read without that legend.","section":"Figure 4(a)"},{"comment":"The human evaluation comparing NTConv◦ and ESConv◦ reports win/loss/tie counts but no significance test; a simple sign test or bootstrap would strengthen the claim that NTConv◦ is preferred.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely useful framework and a clinically grounded evaluation idea, but the headline result depends on a metric denominator that penalizes longer therapist turns, and the automated evaluation is self-referential. The authors should be asked to reanalyze with client-only salience and to report significance tests and planning accuracy. If the near-parity result under client-only salience holds, the paper should be reframed as a framework and dataset contribution rather than a demonstration of transformative superiority; I would not reject outright because the framework, the NTConv dataset, and the human study are valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is real: nobody has formalized White and Epston's narrative therapy stages into an LLM planning loop, and nobody has turned the Innovative Moments Coding System into an automatic salience metric. The stage/reflection formalism and the INT pipeline are a genuine integration of clinical theory with prompt-based agent design. I also give them credit for the human annotation: trained coders, Cohen's kappa above 0.75, explicit co-occurrence rules, and a 30-person within-subject ablation. The fine-tuning application is a bonus, not the core.\n\nBut the central quantitative evidence is soft. The IMA salience denominator in Eq. (5) includes both client and therapist words. INT replies are much shorter than GPT-4o role-play (66 vs 113 words per turn in Figure 4a). So the \"nearly doubling\" of Level-2 IMs is largely a response-length artifact. If you recompute with absolute IM counts or a client-only denominator, the gap shrinks to about 1.3x, and the client-only percentages are near parity. The paper never reports absolute IM word counts or client-only salience, so the flagship claim is not yet supported. That is a load-bearing flaw, not a rounding error.\n\nOther soft spots are less severe but real: no significance tests or error bars; the automated evaluation uses GPT-4o as client, therapist, and supervisor, so it is partially self-referential; expert annotation covers only 10 sessions per system; the planning-state accuracy is never measured against expert judgment; and \"consistently outperforms\" is contradicted by Table 3, where INT's Reassuring score (3.09) is below GPT-4o's (3.11). The Limitation section is honest about some of this, but the abstract overstates.\n\nWho is this for? People working on LLM psychotherapy simulation and process-oriented evaluation. They will want to read the INT formalism and the IMA adaptation, then re-run the numbers with client-only salience. The paper deserves a serious referee: it is novel, clearly written, and the hypothesis is testable. But it should be sent back for major revision, not accepted. I would require: absolute IM counts and client-only salience, inferential statistics on the human dimensions, validation of the state tracker, and artifacts actually released.","headline":"Novel theory-grounded LLM therapy framework, but the IMA salience metric is confounded by therapist response length and the paper overstates its main quantitative claim.","tokens_in":24209,"tokens_out":2129,"would_cite":true,"duration_ms":27254,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that an LLM can act as a narrative therapist by explicitly planning therapeutic stages and reflection levels, and that doing so produces measurably more transformative dialogue than generic role-playing.","keywords":["narrative therapy","interactive narrative therapist","innovative moments","large language models","therapeutic stage planning","retrieval-augmented generation","therapeutic dialogue evaluation","mental health support"],"falsifier":"Have two expert narrative therapists annotate the intended therapeutic stage and reflection depth for every turn of the collected dialogues, then compare their labels with INT's planned states; if agreement is near chance, and if an ablation that replaces the planned state with a random state while keeping the same retrieved exemplars matches full INT's IM salience and dimension scores, the central claim fails.","tokens_in":23231,"feed_emoji":"🧠","tokens_out":6618,"duration_ms":74078,"temperature":0.7,"pith_summary":"This paper sets out to show that an LLM can simulate a narrative therapist the way the therapy is actually practiced: moving through ordered stages—trust building, problem externalization, re-authoring, and re-membering—and deliberately adjusting how deeply it probes at each step. The authors claim this staged, progression-aware approach makes simulated therapy measurably more effective than asking a generic LLM to role-play a therapist. They support the claim with evaluations over 260 simulated clients and 230 human participants, and with a new assessment tool, IMA, that counts 'innovative moments'—client statements that break with a problem-saturated story—and tracks them across a session. If correct, the work matters because it turns an empirically grounded theory of change into a computational protocol that can be deployed, evaluated, and improved without requiring a human therapist in the loop.","feed_headline":"Planned therapy stages nearly double clients' narrative shifts","feed_subtitle":"A stage-aware LLM therapist beats role-play on four therapeutic dimensions and tracks progress across sessions.","key_machinery":"The load-bearing object is the planning space $\\Phi=(\\mathcal{S},\\mathcal{L})$, a formal encoding of narrative therapy's structure: $\\mathcal{S}$ is the set of four therapeutic stages and $\\mathcal{L}$ is the set of reflection depths available inside each stage. Two prompt-defined functions, $\\Psi_S$ and $\\Psi_L$, read the dialogue history and the current client utterance to pick the next stage and the next reflection level, and a third function $\\Psi_T$ generates the response conditioned on that state plus $k=5$ retrieved expert exemplars. The companion evaluation machinery is the IMA classifier $\\Gamma$, which labels each client utterance with one or more of six innovative-moment types (Action, Reflection, and Protest, at two levels), and the salience formula of Equation (5), which converts those labels into a session-level measure of narrative change. The machinery works together to test the paper's hypothesis: that state planning produces the depth and pacing that role-playing instructions alone do not.","core_discovery":"The central claim is that explicit planning of the therapeutic state is what separates a transformative LLM therapist from a merely supportive one. INT maintains a planning space of four stages and multiple reflection levels per stage; before each reply it uses prompt-driven functions to choose the next stage and reflection depth, then retrieves expert-style exemplars for that state and generates a short, conversational response. IMA operationalizes the clinical construct of innovative moments by classifying each client utterance into six types at two levels and computing their salience as the fraction of dialogue words that carry narrative change. The paper's evidence shows INT beating direct role-playing baselines on the four narrative-therapy dimensions in human evaluation, reaching the highest overall IM salience (29.7% versus a next-best 21.9%) and the highest Level 2 markers, and the same pattern holds in the automated simulated-client study. The authors also report that removing either retrieval augmentation or reflection-level planning degrades performance, with the full system advancing further into the re-authoring and re-membering stages than its ablations.","pith_inferences":["An unstated corollary is that the same planner could be turned into a supervision display for human therapists, showing where a session sits in stage and reflection depth; the paper does not test this, but its planning space is exactly the information such a tool would need.","The IMA trajectory—Level 1 moments early, Level 2 moments late—could be used as a training signal: a therapist optimized to elicit Level 2 innovative moments would, if the paper's theory is right, produce better therapeutic outcomes than one optimized for empathy alone.","Because the stage and reflection labels come from the same model class that generates the responses, the causal role of planning is not fully separated from retrieval and style; a cleaner test would freeze the planner and vary only the response generator, or vary the planner while holding the responder fixed."],"forward_implications":["A process-aware therapy simulator can be built by prompting a general LLM with a stage-and-reflection planner, without task-specific fine-tuning for the therapist role.","The IMA salience metric offers a way to compare therapy systems on clinically meaningful narrative change rather than on static empathy ratings or surface overlap metrics.","Because INT can synthesize support conversations, the resulting NTConv dialogues can be used to fine-tune smaller models; the paper reports that a model trained on them outperforms one trained on the original ESConv dialogues on both automatic and human metrics.","The observed Level 1-to-Level 2 innovative-moment shift across turns provides process-level evidence that structured planning induces the kind of narrative progression seen in successful therapy, not merely compliant client agreement."],"supporting_citations":[{"why":"Grounds narrative therapy itself: problem-saturated stories, externalization, and re-authoring as the therapeutic target that INT formalizes.","marker":"White and Epston, 1990"},{"why":"Supplies the staged 'maps of narrative practice' and the idea of reflection depth that INT's planning space encodes.","marker":"White, 2007"},{"why":"Provides the Innovative Moments Coding System and protocol that IMA's six IM types and salience calculation adapt.","marker":"Gonçalves et al., 2011"},{"why":"Supplies empirical evidence that progression from early to advanced innovative moments corresponds to therapeutic change, motivating the Level 1/Level 2 distinction.","marker":"Gonçalves et al., 2012"},{"why":"Used to support the claim that Level 1-to-Level 2 IM shifts mark successful narrative change, grounding the interpretation of INT's observed trajectories.","marker":"Montesano et al., 2017"},{"why":"Supplies the ESConv dataset of real seeker profiles and conversations from which the simulated clients and the NTConv experiment are built.","marker":"Liu et al., 2021"},{"why":"Documents the GPT-4o model used as the planner, responder, and evaluator; the paper selects it after preliminary studies.","marker":"Hurst et al., 2024"}],"fun_headline_variants":["AI therapist plans stages to boost client narrative shifts","Stage-aware LLM therapist outdoes role-play in therapy depth","Interactive narrative therapist tracks innovative moments","Planned therapy stages improve client narrative change"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the stage and reflection level the planner picks are accurate and informative; the paper never measures those labels against expert judgment, so the measured advantage could in principle come from the retrieved exemplars or the response style rather than from progression-aware planning.","fun_headline_variants_meta":{"raw":{"variants":["AI therapist plans stages to boost client narrative shifts","Stage-aware LLM therapist outdoes role-play in therapy depth","Interactive narrative therapist tracks innovative moments","Planned therapy stages improve client narrative change"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1564,"prompt_tokens":918,"completion_tokens":646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":588}},"tokens_in":534,"tokens_out":646,"duration_ms":9188,"temperature":1.0,"reasoning_tokens":588,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:41:32.885477+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two expert narrative therapists annotate the intended therapeutic stage and reflection depth for every turn of the collected dialogues, then compare their labels with INT's planned states; if agreement is near chance, and if an ablation that replaces the planned state with a random state while keeping the same retrieved exemplars matches full INT's IM salience and dimension scores, the central claim fails.","supporting_citations":[],"review_version":1}