{"id":"a241a688-17f7-4894-abe0-efc28cd708a3","arxiv_id":"2412.16674","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new Chinese dataset and framework for mixed-type, spatiotemporal-aware psychological counseling dialogues, with experiments suggesting goal prompts and location/time stamps improve responses.","lead":"The authors built STAMPsy, a Chinese dataset of over 5,000 simulated psychological counseling conversations that mix five dialogue types and add time, location, and weather awareness. They also fine-tuned a dialogue model, Self-STAMPsy, and report that its module-based prompts improve response quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"STAMPsy's golden dialogues were generated with the same goal/stamp conditioning that Self-STAMPsy consumes, so ablations may reward pipeline replication rather than counseling quality; external validation is needed.","rationale":"The reader identified LLM-generated data as the weakest assumption; I partially agree but sharpen it. The issue is not only transfer to real counseling; it is internal validity. The generation prompt and the model input share the same conditioning variables, so the test set is not neutral. Expert revision mitigates but does not remove the bias, because the experts were revising dialogues that already contained these signals. A control-set experiment would distinguish genuine utility from pipeline mimicry. The reader's CONDITIONAL verdict remains appropriate; no change needed if the condition is reframed as requiring this control experiment. If the control experiment fails, the verdict would move toward REJECT for the central claim, while the dataset could still be a resource.","tokens_in":23702,"tokens_out":7489,"duration_ms":70543,"concrete_test":"Generate a matched control test set from the same original source texts, using the identical generation pipeline but with the spatiotemporal-stamp and helping-skill modules removed from the system prompt (all other instructions intact), then have experts revise it exactly as before. Re-run the full ablation suite on this control set; if the Table 6 deltas (e.g., BLEU-1 42.48 vs 40.06, ROUGE-L 44.75 vs 38.09, empathy 1.82 vs 1.31) do not reproduce, the reported effectiveness of goal prompts and spatiotemporal states is an artifact of the generation procedure. Also verify that no source case contributes dialogues to both train and test; if it does, repeat with case-level splitting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 'Dataset Collection' states that the authors 'conclude a multipart system prompt shaped by instructions based on the CCM and expand single-turn text to multi-turn dialogues,' and Appendix F shows this prompt embeds the helper personality, helping-skill descriptions, and tables connecting time/weather/location to counseling responses. The golden responses in STAMPsy are therefore produced under exactly the kind of goal and spatiotemporal conditioning that Self-STAMPsy injects at inference (egi and gSTi). Because the test split is drawn from the same corpus, the comparisons in Table 6 between Self-STAMPsy and '-w/o Helping Skills' / '-w/o STi' primarily measure how well the model recreates the conditioning used to write the golden responses. The manual and automatic metrics cannot distinguish 'this signal helps a counselor respond' from 'this signal was present when the gold response was authored.' Expert revision checked consistency and correctness but did not independently validate that the spatiotemporal/goal conditioning is causally useful; it only confirms the generated dialogues are internally coherent. Additionally, source texts are divided into multiple meetings, and no case-level train/test split is described, so the same original case can appear in both training and test sets, further inflating absolute scores. The central claim therefore rests on an evaluation that is entangled with the data-generation pipeline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces STAMPsy, a Chinese dataset of 5,006 mixed-type psychological counseling dialogues with helping-skill annotations and spatiotemporal stamps, and proposes Self-STAMPsy, an iterative self-feedback framework that combines helping-skill selection (via BERT), spatiotemporal state processing, adaptive graph-based retrieval, and LoRA fine-tuning of Qwen2. The authors evaluate on STAMPsy and a small external GHSC transcript, reporting that Self-STAMPsy outperforms baselines on automatic metrics and manual ratings by psychotherapists, and ablation results indicate that removing the helping-skill prompt, spatiotemporal stamp, or self-feedback module degrades performance. The paper claims that clarifying dialogue goals in advance and utilizing spatiotemporal states are effective for psychological counseling dialogue generation.","tokens_in":23997,"tokens_out":5826,"duration_ms":45257,"significance":"If the dataset is trustworthy and the evaluation sound, the paper makes a useful contribution: it is the first mixed-type psychological counseling dataset with spatiotemporal awareness, it ships with a public repository, and the task formulation (goal clarification before counseling) is practical and under-explored. The expert revision and quality audit (0.91 quality score, Kappa 0.84) are commendable. However, the evaluation is currently entangled with the data-generation pipeline, and the quantitative evidence for the central claim is statistically thin. The core idea deserves to be published, but only after the evaluation is made independent and rigorous.","major_comments":[{"comment":"The golden responses in STAMPsy are generated by an LLM using a multipart system prompt that embeds the helping-skill descriptions and the spatiotemporal-state tables (time of day, weather, location) that Self-STAMPsy later injects as e_gi and g_ST_i. The test split is drawn from the same corpus, so the ablations '-w/o Helping Skills' and '-w/o ST_i' in Table 6 primarily measure whether the model replicating the data-creation conditioning produces text closer to the gold, not whether those signals improve counseling quality. Expert revision checked internal consistency and correctness but did not independently validate the causal usefulness of the goal/stamp conditioning. The central claim that clarifying goals and using spatiotemporal states are effective therefore needs an evaluation not entangled with the data-generation pipeline, such as a human study on real or independently written counseling responses.","section":"Dataset Collection and Appendix F"},{"comment":"No significance tests or confidence intervals are reported for any automatic or manual metric. Several decisive comparisons are small: the GHSC gap between Self-STAMPsy (70.91) and GPT4 (69.09) is 1.8 points, and the manual Rel. and Help. scores are identical (1.85) or differ by 0.02–0.05 on a 0–2 scale. The manual-evaluation description states only that 'three out of four evaluators are consistent on the results of at least 910 conversations in the test set,' without the number of scored conversations, per-item variance, or inter-annotator agreement for the five indicators. Without this information, the reported improvements cannot be distinguished from noise.","section":"Experiments, Table 6"},{"comment":"The authors state that they 'divide long cases into several meetings manually' but do not describe a case-level train/test split. If meetings from the same original case appear in both training and test sets, the model has already seen the client's profile and history, inflating absolute scores and reducing the validity of the benchmark. The paper should group all meetings derived from the same source case into the same split and report results on that split.","section":"Dataset Collection / Data Source"},{"comment":"The GHSC evaluation uses a single helper-client transcript from Hill's system to measure helping-skill understanding, and the annotation scheme is the same Hill taxonomy used to annotate STAMPsy. A single transcript, annotated under the same framework, is too small and too dependent on the STAMPsy protocol to support the general conclusion that 'current LLMs are unable to accurately clarify goals' (Table 6, GHSC column). Expanding GHSC to multiple transcripts with independent expert annotations would strengthen the claim.","section":"Evaluation / GHSC"}],"minor_comments":[{"comment":"The 'Total 5330' appears to be the sum of the Direct Guidance subtypes (89+1128+2210+1777+137=5341), not the total of all goal types; the table caption should clarify what the numbers count and reconcile them with Table 3's 'Avg. # of goals per dialogue 16.51' (the five goal-type counts sum to 25,672, which would imply a different average).","section":"Table 2"},{"comment":"The sentence 'Following PsyCoun(Hu et al. 2024)' appears to refer to 'PsycoLLM' in the references; check the citation.","section":"Experiments"},{"comment":"Figure 4 is not referenced in the text; add a reference such as '(see Figure 4)'.","section":"Methodology"},{"comment":"There are typos: 'SpatioTmeporal' in the Table 6 caption, 'environemnt' in the case study, and a duplicated sentence in the Case Study paragraph about MindChat and EmoLLM.","section":"Case Study and Table 6"},{"comment":"The manual evaluation guideline says 'score 0 (bad): not fluent and the answer is not in line with the question all'; fix the grammar. Also, the data quality audit reports a Kappa for the 1/0 quality evaluation but not for the fine-grained helping-skill and goal annotations; consider reporting that agreement.","section":"Appendix I"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv preprint (cs.AI) that appears to target an NLP/dialogue venue. Given the dataset release, it is a plausible dataset/benchmark paper, but the current evaluation does not support the abstract's causal claims. I recommend requiring an independent human evaluation or an external real-world corpus to break the circularity between data generation and the model's conditioning signals. Also, the single-transcript GHSC is too weak to serve as external validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the STAMPsy dataset is a real contribution: 5,006 Chinese mixed-type counseling dialogues with helping-skill and spatiotemporal annotations, expert-revised, with a reported kappa of 0.84. That is worth having. Second, the central claim—that clarifying dialogue goals and injecting spatiotemporal states improves responses—is not actually supported by the experiments as designed. The golden responses were generated by LLMs prompted with exactly those goal and spatiotemporal instructions, then revised by experts. So the ablations that remove those signals may just be measuring how well the model replicates the data-generation pipeline, not whether those signals make counseling better. This is a load-bearing flaw for the effectiveness claim, though it does not sink the dataset.\n\nWhat the paper does well: it builds a benchmark with several baselines, runs ablations, and reports both automatic and manual metrics. The iterative self-feedback framework is a reasonable addition. The paper is also honest about limitations and does not overstate clinical applicability.\n\nWhere the soft spots are: no significance tests or confidence intervals in Table 6, and several manual differences between Self-STAMPsy and GPT-4 are tiny (Relevance 1.85 vs 1.85, Informativeness 1.72 vs 1.70). The external GHSC benchmark is a single transcript and uses the same Hill taxonomy as the annotation, so it is not an independent check. The paper does not describe a case-level train/test split; since long cases are divided into multiple meetings, the same original case could appear in both training and test sets, which would inflate scores. These are fixable in revision.\n\nThe stress-test note about the data-generation entanglement is on point. The authors say the multipart system prompt embeds helping skills and spatiotemporal tables, so the gold responses are conditioned on the same signals the model is asked to predict. Expert revision checked consistency, not whether the conditioning is causally useful. That means the paper's abstract claim overshoots what the evidence shows.\n\nWho this is for: researchers building counseling dialogue datasets or models will get value from the resource and the framework. It deserves a serious referee, but the effectiveness claim needs external validation—e.g., human pairwise comparisons on real or independently collected counseling dialogues, or at least a split where the gold responses are written without the conditioning. I would send it to review with that expectation.","headline":"Useful dataset, but the effectiveness claim is entangled with how the data were generated; needs external validation.","tokens_in":24558,"tokens_out":2640,"would_cite":true,"duration_ms":23845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that counseling chatbots improve when they choose the therapist's next helping skill and account for the client's time and place before replying.","keywords":["psychological counseling dialogue","mixed-type dialogue","spatiotemporal awareness","helping skills","self-feedback","large language models","Chinese dialogue dataset","retrieval-augmented generation"],"falsifier":"A concrete test would be to label a small corpus of real anonymized online counseling sessions with the same helping-skills and spatiotemporal annotations and to rerun Self-STAMPsy against the strongest baselines under the paper's evaluation protocol; if the gap narrows or reverses, the central claim is not transferable.","tokens_in":23495,"feed_emoji":"🧠","tokens_out":9035,"duration_ms":71902,"temperature":0.7,"pith_summary":"The paper argues that online psychological-counseling chatbots should handle several dialogue types in one session and should know what they are doing and where and when the client is. To make this concrete, it introduces STAMPsy, a Chinese dataset of 5,006 counseling dialogues and 61,832 utterances in which every turn is annotated with the counselor's helping skill, the client's behavior, and a spatiotemporal stamp recording time, place, and weather. It then builds Self-STAMPsy, a framework that predicts the next helping skill before generating each reply, extracts the spatiotemporal state, retrieves relevant knowledge, and refines responses through self-reflection. The reported experiments on this benchmark show that the full model outperforms several fine-tuned LLM baselines on automatic metrics and on clinician-rated relevance, helpfulness, and empathy, and that removing the goal or spatiotemporal modules degrades performance. If these results transfer, goal clarification and world-awareness are practical design choices for AI counseling rather than optional extras.","feed_headline":"Chatbots counsel better when they know the goal and the setting","feed_subtitle":"A new benchmark and model show that predicting the therapist's helping skill and spatiotemporal stamp improves responses.","key_machinery":"The load-bearing mechanism is the helping-skill sequence. Each counselor utterance is labeled with one of eight helping skills drawn from counseling psychology — open questions, restatements, feeling reflection, interpretation, information giving, direct guidance, immediacy, and self-disclosure — and the system predicts the next skill before writing the response, converting it into a natural-language instruction such as 'The therapist will then design a therapy.' The second mechanism is the spatiotemporal stamp: each dialogue is linked to a quadruple recording domain, slot, value, and time or place or weather stamp, and the model generates an emotional-state consequence from that stamp. These two signals are fed, together with adaptively retrieved knowledge-graph triples and an iterative self-feedback reflection, into a fine-tuned dialogue-generation LLM. The framework is called Self-STAMPsy, and the benchmark, STAMPsy, is the dataset plus the three sub-tasks used to test it.","core_discovery":"The central claim is that mixed-type counseling dialogues can be generated more faithfully when the model treats each counselor turn as an explicit helping-skill goal embedded in a spatiotemporal context. The authors report that a model that first predicts the helping skill and then uses the spatiotemporal stamp outperforms fine-tuned general-purpose and psychology-oriented baselines on automatic metrics and on clinician-rated relevance, informativeness, human-likeness, helpfulness, and empathy. Ablations show quality drops when the helping-skill prompt, the spatiotemporal stamp, or the self-feedback module is removed, with the empathy rating substantially lower when spatiotemporal information is omitted. The discovery is thus a mechanism-level result: clarifying dialogue goals in advance and stamping dialogues with time and place are effective interventions, not just dataset artifacts.","pith_inferences":["Inference: the helping-skill prediction component may be useful on its own as a training or supervision aid for counselors, independent of the response generator, because it operates on transcripts and produces a checkable skill label.","Inference: the spatiotemporal quadruple mechanism could absorb other nonverbal context cues the paper lists but does not implement, such as paralanguage or body movement, if such inputs were available from a webcam or wearable device.","Inference: because the dataset is LLM-generated from casebooks and public question data and then expert-revised, a decisive test of transfer is whether the same gap over baselines appears on real, anonymized counseling transcripts; the paper itself notes that real consultation records are difficult to obtain.","Inference: the paper's own case study notes that the model sometimes asks several questions in one turn; goal clarification without a pacing constraint may add pressure, so a turn-level question-count control would be a natural extension."],"forward_implications":["Counseling systems can be designed as mixed-type conversations, so one model moves between diagnosis, information giving, empathetic support, recommendations, and question answering within a single session.","Clarifying the counselor's goal before each reply keeps conversations anchored to the client's emotional concerns, rather than drifting into generic educational or medical advice.","Spatiotemporal awareness lets a system tailor suggestions to the client's actual situation — for example, daylight exposure and a rich breakfast for early-morning awakening, versus pre-sleep relaxation for late-night insomnia.","The ablation results indicate that each of the four modules, helping-skill selection, spatiotemporal processing, adaptive retrieval, and iterative self-feedback, contributes to the reported quality; removing any one lowers performance.","A public Chinese benchmark now exists for comparing mixed-type psychological-counseling dialogue systems, with goal and spatiotemporal annotations as part of the standard evaluation."],"supporting_citations":[{"why":"Supplies the helping-skills taxonomy and the assumption that counseling proceeds through an organized skill sequence, which the annotation and the goal-prediction task rely on.","marker":"Hill 2009"},{"why":"Establishes time, environment, and spatial effects as nonverbal communication cues in counseling, the empirical basis for spatiotemporal stamps.","marker":"Sherry, Paula, and Cynthia 2009"},{"why":"Provides the 9-Box Case Conceptualization Model used to structure personal information and to derive question templates for dialogue expansion.","marker":"Meichenbaum 2009"},{"why":"PsyQA contributes the Chinese online mental-health support conversations and helping-skills material used as a data source and baseline.","marker":"Sun et al. 2021"},{"why":"ESConv supplies the emotional-support dialogue techniques and dataset lineage that the empathetic components of STAMPsy build on.","marker":"Liu et al. 2021a"},{"why":"MidMed is the medical mixed-type dialogue predecessor that establishes how to combine diagnosis, recommendation, QA, and empathetic turns, which STAMPsy adapts to psychology.","marker":"Shi et al. 2023"},{"why":"Provides the multi-type dialogue construction approach and the data-quality evaluation procedure used for STAMPsy.","marker":"Liu et al. 2020"},{"why":"Connects open-domain dialogue with dynamic spatiotemporal-aware knowledge, the direct inspiration for stamping dialogues with spatiotemporal state.","marker":"Zhou et al. 2022"},{"why":"Self-RAG supplies the learn-to-retrieve-and-critique mechanism that the iterative self-feedback module is inspired by.","marker":"Asai et al. 2023"},{"why":"CPsyCoun is the psychological-counseling benchmark used to select baseline models and evaluation practice for the experiments.","marker":"Zhang et al. 2024"}],"fun_headline_variants":["Predicting the goal and setting makes counseling chatbots more effective","Spatiotemporal stamps help counseling AI pick the right response","STAMPsy: new benchmark for mixed-type counseling dialogue","Why clarifying goals and adding place-time boosts counseling bots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that expert-revised, LLM-generated dialogues from casebooks and public QA data resemble real counseling closely enough that quality gains measured on this benchmark transfer to actual online clients.","fun_headline_variants_meta":{"raw":{"variants":["Predicting the goal and setting makes counseling chatbots more effective","Spatiotemporal stamps help counseling AI pick the right response","STAMPsy: new benchmark for mixed-type counseling dialogue","Why clarifying goals and adding place-time boosts counseling bots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1849,"prompt_tokens":937,"completion_tokens":912,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":844}},"tokens_in":553,"tokens_out":912,"duration_ms":8288,"temperature":1.0,"reasoning_tokens":844,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:21:33.297517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to label a small corpus of real anonymized online counseling sessions with the same helping-skills and spatiotemporal annotations and to rerun Self-STAMPsy against the strongest baselines under the paper's evaluation protocol; if the gap narrows or reverses, the central claim is not transferable.","supporting_citations":[],"review_version":1}