{"id":"8deafb61-5b0c-4c75-8996-df987593bac9","arxiv_id":"2501.15260","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"UPSD4 improves unobtrusive symptom probing and diagnostic accuracy in simulated depression diagnosis dialogues by steering LLMs with psychological questioning strategies.","lead":"A new depression-diagnosis chatbot framework, UPSD4, uses indirect questioning strategies drawn from psychology to reduce stigma when probing sensitive symptoms. In simulated user evaluations it outperforms direct-questioning baselines, but real-patient effects remain untested.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"With-stigma simulator's explicit refusal rule in Table 13 likely predetermines the Dx Rate gap; real-user validation is still missing.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the with-stigma simulator's rule-based secretiveness may predetermine the advantage of indirect probing. My stress-test confirms this and locates the specific prompt text in Table 13, noting two additional aggravating factors: the simulator and the system share the same GPT-4 prior, and the human evaluation does not test real disclosure behavior—only comparative dialogue quality. The paper does have genuine independent support: the human evaluation with trained annotators and a good inter-annotator agreement, plus the ablation (UPSD4 w/o strat) showing strategy-specific gains even within the simulated setting. These make the method internally consistent, so the appropriate verdict remains CONDITIONAL rather than REJECT. The central real-world claim, however, hinges on the simulator's behavior matching real stigma-affected users, which is not demonstrated. The proposed test—removing the explicit refusal instruction—would directly probe whether the rule is responsible for the Diagnosis Rate gap. If the gap persists, concern weakens; if it collapses, the paper's headline claim needs substantial revision. This is a falsifiable, concrete check, so I recommend keeping the reader's CONDITIONAL verdict rather than escalating to REJECT or UNVERDICTED.","tokens_in":19427,"tokens_out":3475,"duration_ms":33010,"concrete_test":"Remove the explicit 'secretive' and 'refuse' instructions from the with-stigma simulator prompt (Table 13), keeping only the stigma profiles and the Depression Stigma Scale context, and re-run the Table 2 comparison. If the UPSD4 advantage in Diagnosis Rate over Vanilla GPT-4 (currently 59.85% vs. 38.64%) narrows by more than 10 percentage points, the rule is likely responsible for the headline result. As a further grounding check, gather refusal judgments from human participants with high Depression Stigma Scale scores on a sample of 50 system responses per condition and compare their refusal rates to the simulator's; if humans do not show the same pattern, the simulation does not support the real-world claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that UPSD4 enhances depression diagnosis in a stigma-free way rests on the with-stigma user simulator behaving like a real person with depression stigma. Appendix C.4, Table 13 instructs the simulator to 'be *secretive* to respond when you think the psychologist's response will cause *stigma*' and to '*refuse* to answer the question when you think the psychologist's response will cause *stigma*.' Because the simulator is GPT-4, the same model family as the UPSD4 backbone and the GPT-4-based judge, its refusal decisions share the system's own prior about what is stigmatizing. Direct questions such as 'Have you had thoughts about harming yourself?' are almost trivially labeled as stigma-causing, so the improvement in Diagnosis Rate (Table 2: 59.85% vs. 38.64% against Vanilla GPT-4) may reflect rule-following behavior rather than real-world disclosure dynamics. The human evaluation in Table 4, while valuable (five psychology-trained annotators, Fleiss' kappa 0.731), validates dialogue quality, not whether real stigma-affected users would actually disclose more to indirect probing. The Limitations section itself notes that results can be influenced by prompts, but this is not explored for the user simulator. Thus the real-world conclusion is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UPSD4, a two-module framework for depression-diagnosis-oriented dialogue that aims to probe symptoms unobtrusively to mitigate stigma. The Unobtrusive Probing Module (UPM) selects among psychologically grounded questioning strategies (discreet probing, empathy, flow management) to guide response generation, while the Conversational Diagnosis Module (CDM) performs slot-filling against ICD-11 criteria and issues a diagnosis once all slots are filled. The method is evaluated on the D4 dataset using GPT-4-based user simulators with and without stigma beliefs, with automatic GPT-4 judgments of dialogue quality, a human evaluation by psychology-trained annotators, and diagnostic accuracy metrics. The central reported result is that UPSD4 outperforms baselines in dialogue unobtrusiveness and in diagnosis rate, particularly for the with-stigma simulator.","tokens_in":19682,"tokens_out":4052,"duration_ms":38274,"significance":"If the empirical claims held, UPSD4 would be a practically valuable contribution to mental-health dialogue systems: it offers a concrete, theory-grounded strategy taxonomy, a two-stage strategy selection mechanism, and a reproducible evaluation protocol (temperature 0, seed 42, fully specified prompts). The human evaluation with five psychology-trained annotators and Fleiss' kappa of 0.731 is a genuine strength, as is the careful reporting of prompts and simulator details. However, the central diagnostic claim rests on the behavior of a user simulator that is explicitly instructed to refuse questions it perceives as stigma-inducing, and on GPT-4 judging the output of a GPT-4-based system, so the significance for real-world deployment is not yet established.","major_comments":[{"comment":"The with-stigma user simulator is instructed to 'be secretive to respond when you think the psychologist’s response will cause stigma' and to 'refuse to answer the question when you think the psychologist’s response will cause stigma.' Since the simulator is GPT-4, the same model family as the UPSD4 backbone and the automatic judge, this rule likely determines the Diagnosis Rate gap: direct probes such as 'Have you had thoughts about harming yourself?' are almost trivially identified as stigma-causing, triggering refusal, while the indirect UPSD4 probes avoid the trigger. This is compounded by the Dx Rate definition in Appendix C.3, which counts a diagnosis as unsuccessful whenever any slot is refused or unfilled. The paper should provide evidence that this refusal behavior is a faithful model of real human stigma responses, for example by varying the refusal rule in the simulator, comparing simulator disclosures to human responses, or validating with human participants who actually answer the questions.","section":"Section 5.1, Appendix C.1, Table 13"},{"comment":"The automatic unobtrusiveness scores (Discreetness, Empathy, Coherence, Fluency) are produced by GPT-4, the same model family that generates the system responses and simulates the users. The G-Eval-style self-evaluation may therefore reflect GPT-4's own preferences about what counts as discreet rather than human perception. The human evaluation in Section 5.3 and Table 4 provides complementary evidence, but it only compares dialogue quality between methods; it does not measure whether real stigma-affected users would disclose more to indirect probing. Please add a human evaluation of disclosure behavior, or at least a judge-model swap and an analysis of judge bias, to support the claim that UPSD4 'enhances depression diagnosis' through better disclosure.","section":"Section 5.1, Evaluation Metrics"},{"comment":"The reported improvement of 7.06% in Accuracy and 8.66% in Diagnosis Rate for with-stigma users is computed relative to the best baseline, but the baselines (Vanilla GPT-4, Vanilla Qwen2) are not given any unobtrusive-probing instruction, so they are likely to ask direct questions. Combined with the simulator's hard-coded refusal rule, this makes the improvement partly an artifact of the evaluation setup rather than of UPSD4's questioning skills. The authors should either relax the refusal rule, measure the sensitivity of the Dx Rate to the refusal probability, or otherwise demonstrate that the advantage persists under less contrived simulator conditions.","section":"Table 2, With-Stigma rows"}],"minor_comments":[{"comment":"The notation 'Ht = {(uu_i, us_i)}' appears to be a typographical error; it should likely be 'u^u_i' and 'u^s_i' to distinguish user and system utterances.","section":"Section 3, Task Formalization"},{"comment":"The figure contains the typo 'Dpression Mood' and the abbreviation 'USPD4' in the right panel; the rest of the paper uses 'UPSD4'.","section":"Figure 3"},{"comment":"The column headers 'Disc', 'Empth', 'Cohr', and 'Fluen' are not expanded in the caption; please define them or use the full metric names for accessibility.","section":"Table 2"},{"comment":"The statement that D4 is 'the only currently available benchmark specifically designed for depression disorder diagnosis dialogue' should include a qualification such as 'to the best of our knowledge' or a citation, especially since the authors also cite newer related datasets.","section":"Section 5.1, Dataset section"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clean method and transparent reporting, but the empirical core is currently vulnerable to the criticism that the with-stigma simulator's refusal rule predetermines the advantage of indirect probing. If the authors can add a sensitivity analysis or a real-user validation, the contribution could be solid; otherwise, the diagnostic-rate claims should be substantially softened to 'in this simulated setting.' The human evaluation, while well executed, does not rescue the diagnosis-rate claim because it measures dialogue quality, not disclosure behavior."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper proposes UPSD4, a two-module system that uses psychology-derived probing strategies (discreet questioning, flow management, empathy) to elicit depression symptoms without asking blunt questions. That's a real and reasonable contribution. But the central evidence that it improves diagnosis with stigma-affected users rests on a simulator that is explicitly instructed to be secretive or refuse when it thinks the psychologist's response will cause stigma. That instruction almost guarantees the indirect approach will look better.\n\nWhat's genuinely new: the strategy taxonomy is borrowed from established survey research, yes, but applying it as a prompt-selection method for LLM-based diagnosis is new in this literature. The architecture (UPM deciding what to ask and how, CDM slot-filling ICD-11 criteria) is clean and reproducible. The human evaluation is decent—five psychology-trained annotators, 240 paired dialogues, kappa 0.731—and it does show that judges prefer UPSD4's dialogue quality. That's a real signal, independent of the simulator.\n\nThe soft spots are in the evaluation loop. The with-stigma simulator prompt (Table 13) is hard-coded to refuse when it perceives stigma. Direct questions—'have you had thoughts of harming yourself?'—trivially trigger that rule, so the Dx Rate gap (59.85% vs 38.64%) is at least partly an artifact of instruction-following, not human-like disclosure. On top of that, the automatic scores come from GPT-4, the same model family that generates the responses and the simulator's reactions; it can't be called an independent judge. The human evaluators rated the dialogue, not whether a genuinely stigma-affected person would open up more to indirect probes. The limitations section acknowledges that prompts influence results, but that point isn't applied to the simulator itself.\n\nNone of this makes the paper unserious. The strategy decomposition and the generation behavior are worth having, and the authors are honest about using simulators. But the phrase 'enhancing depression diagnosis' is not yet supported for real users; it's supported for a GPT-4 simulator told to distrust direct questions.\n\nFor whom: people building mental-health chatbots and anyone working on LLM evaluation should read it. I'd send it to review, with major revision required. The authors should either run a small real-user study or change the simulator to ground refusal behavior in empirical stigma research—and use a different judge, or at least a judge that isn't the generator.\n\nI'd bring it to a reading group; the evaluator-simulator circularity is a good teaching case.","headline":"A sensible strategy-based probing system for depression dialogue, but its diagnostic advantage is baked into a user simulator that is told to refuse direct questions.","tokens_in":20170,"tokens_out":3253,"would_cite":true,"duration_ms":29988,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a dialogue system can reduce stigma-driven nondisclosure in depression screening by probing symptoms indirectly, and that its UPSD4 method shows measurable diagnostic gains in simulated conversations.","keywords":["depression diagnosis dialogue","stigma","unobtrusive probing","large language models","empathetic response generation","ICD-11","slot filling","user simulation"],"falsifier":"Run UPSD4 against direct questioning in a controlled study with consenting adults who score high on the Depression Stigma Scale, measuring how many of the nine ICD-11 symptoms they disclose and whether a full diagnosis is reached; if disclosure is equal under both questioning styles, the claimed mechanism does not generalize.","tokens_in":19264,"feed_emoji":"🩺","tokens_out":7330,"duration_ms":60475,"temperature":0.7,"pith_summary":"Stigma keeps people from admitting depression symptoms, and this paper argues that automated diagnosis will stall unless dialogue systems learn to ask indirectly. To that end it introduces UPSD4, a two-module system that couples unobtrusive probing strategies with an ICD-11-based slot-filling diagnosis module. Evaluated against user simulators built from a depression-diagnosis dialogue benchmark, UPSD4 outperforms existing chatbots on dialogue quality and on diagnostic accuracy and completion, with the largest gains when the simulated user holds stigma beliefs. The takeaway is that how a screening system asks may matter as much as what it asks.","feed_headline":"Indirect probing lifts depression diagnosis accuracy by 7%","feed_subtitle":"In simulated chats, a two-module chatbot that probes symptoms tactfully gets more disclosure from users with stigma.","key_machinery":"The load-bearing component is the strategy set itself: nine fine-grained strategies organised under Discreet Probing, Empathy, and Flow Management, selected through a two-stage coarse-then-fine prompt because LLMs choose more reliably from small candidate sets. Each strategy translates a known survey- or psychotherapy-research technique into an instruction for response generation, and the selection loop is driven by slot states from the Conversational Diagnosis Module, so topic choice and questioning style are jointly determined. The same nine ICD-11 slots also serve as the diagnostic output structure, making the dialogue record directly interpretable.","core_discovery":"The paper's central claim is that unobtrusive probing is a learnable, promptable behavior, and that encoding it as explicit dialogue strategies is what makes an LLM-based depression screener work under stigma. UPSD4's Unobtrusive Probing Module selects a coarse strategy (Discreet Probing, Empathy, or Flow Management) and then a fine-grained one (e.g., Loading Question, Nominative Technique, Forgiving Question, Clarification) before generating each response. The Conversational Diagnosis Module tracks nine ICD-11 depression criteria as slots, updates them from the dialogue, chooses the next criterion to probe, and produces a diagnosis when all slots are filled. On the D4 benchmark with a with-stigma user simulator, the paper reports average improvements of 7.06% in Accuracy and 8.66% in Diagnosis Rate over the best baseline, and a shift toward gentler questioning strategies when the user shows reluctance.","pith_inferences":["If the same pattern holds with real patients, the strategy taxonomy could be turned into a training curriculum for human interviewers, since it turns survey-response psychology into concrete phrasings.","The method should transfer to other stigmatized health conditions, such as substance use, suicidal ideation, or HIV status, where direct questioning is known to suppress disclosure.","The measured margin is against a simulator whose secrecy is explicitly rule-prompted, so the size of the real-world benefit remains an open empirical question rather than a settled result."],"forward_implications":["Prompt-level strategy sets can change whether an LLM asks a sensitive question bluntly or indirectly; the paper's ablation shows that simply instructing the model to be unobtrusive, without concrete strategies, produces weaker results.","Because every method in the comparison shares the same diagnosis module, the reported gains in accuracy and diagnosis rate are attributed to the dialogue itself, meaning questioning style directly affects screening completeness.","The system shifts from Loading Questions toward Forgiving Questions when interlocutors show reluctance, which is evidence that an LLM can adapt its probing style to perceived stigma sensitivity.","The nine ICD-11 slots used for diagnosis are reusable scaffolding for other LLM-based mental-health screeners, independent of the specific probing prompts."],"supporting_citations":[{"why":"Supplies the D4 dataset and the CPT baseline that all methods are compared against.","marker":"Yao et al., 2022"},{"why":"Supplies the Depression Stigma Scale used to validate that the with-stigma simulator scores high on stigma beliefs.","marker":"Roeloffs et al., 2003"},{"why":"Supplies ICD-11, the diagnostic criteria from which the nine symptom slots are drawn.","marker":"The World Health Organization, 2024"},{"why":"Provides the unobtrusive-measurement concept that motivates indirect probing.","marker":"Webb et al., 1999"},{"why":"Grounds the sensitive-topic questioning techniques, including the loading question and nominative technique, in survey methodology.","marker":"Lee, 1993"},{"why":"Supplies evidence on sensitive-question response effects that the forgiving-question strategy relies on.","marker":"Tourangeau and Yan, 2007"},{"why":"Supplies the Helping Skills Theory from which the empathy strategies are summarized.","marker":"Hill, 2020"},{"why":"Provides the conceptualization of stigma used to construct the stigma profiles for the with-stigma user simulator.","marker":"Link and Phelan, 2001"}],"fun_headline_variants":["Soft questions, sharp results: AI's tactful probing lifts depression diagnosis 7%","How to ask without asking: AI's indirect probes improve depression screening","Gentle chat AI wins honesty: 7% better depression diagnosis via subtle probes","Unobtrusive AI probing: the key to overcoming stigma in depression detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The main result depends on the assumption that the with-stigma user simulator's rule-based tendency to become secretive and refuse when asked directly behaves like real people with depression stigma.","fun_headline_variants_meta":{"raw":{"variants":["Soft questions, sharp results: AI's tactful probing lifts depression diagnosis 7%","How to ask without asking: AI's indirect probes improve depression screening","Gentle chat AI wins honesty: 7% better depression diagnosis via subtle probes","Unobtrusive AI probing: the key to overcoming stigma in depression detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000456,"raw_usage":{"total_tokens":2255,"prompt_tokens":877,"completion_tokens":1378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":1294}},"tokens_in":493,"tokens_out":1378,"duration_ms":12729,"temperature":1.0,"reasoning_tokens":1294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:26:41.075130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run UPSD4 against direct questioning in a controlled study with consenting adults who score high on the Depression Stigma Scale, measuring how many of the nine ICD-11 symptoms they disclose and whether a full diagnosis is reached; if disclosure is equal under both questioning styles, the claimed mechanism does not generalize.","supporting_citations":[],"review_version":1}