Pith. sign in

REVIEW 4 major objections 5 minor 58 references

DS@GT at eRisk 2025: From prompts to predictions, benchmarking early depression detection with conversational agent based assessments and temporal attention models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single engineered system prompt, reused across four LLMs, yields internally consistent BDI-II severity rankings from naturalistic conversations, with a label-encoded regression fit of R² = 0.91.

desk verdict An honest, useful eRisk working note whose central reliability claim is overreach: the R²=0.91 regression just compares two outputs of the same LLM under a prompt that defines the category bands. read the letter →

arxiv 2507.10958 v1 pith:RDKEURDZ submitted 2025-07-15 cs.CL

Anthony Miyaguchi , David Guecha , Yuwen Chiu , Sidharth Gaur This is my paper
classification cs.CL
keywords conversationaldepressiondetectionBDI-IIlargelanguagemodelspromptengineeringtemporalattentionmentalhealthscreeningsocialmediaearlyLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The working note's main claim comes from the conversational pilot task: one carefully engineered system prompt, reused without modification across four different LLMs, turns a roughly twenty-turn naturalistic conversation into a structured assessment aligned with the Beck Depression Inventory II (BDI-II), without ever asking the persona directly about depression. Ground-truth labels were unavailable, so the authors validate through internal consistency, reporting a linear regression between the models' label-encoded severity categories and their reported BDI-II totals with R² = 0.91. They interpret this as evidence that the LLM-based classification is reliable and that its severity categories are clinically meaningful. A secondary social-media task, where a LightGBM with temporal attention over MentalRoBERTa embeddings beat a voting classifier on early ranking metrics, is presented as exploratory and below expectations. If the pilot claim holds, LLM conversations could offer a low-cost screening step that is aligned with an established clinical instrument.

What carries the argument

The central object is the evaluator-agent system prompt, reproduced in the appendix, with its embedded BDI-II reference. The prompt defines an empathetic, non-directive interviewer, prohibits direct questions about depression, walks the model through the 21 BDI-II symptom items with 0–3 anchor descriptions, and demands a per-turn JSON object containing the output message, next-step reasoning, assessment state, total BDI-II score, severity classification, confidence, and per-item scores. The numerical device that carries the argument is the label-encoding regression: the seven severity labels are mapped to levels 0–6 and fit against the reported BDI-II totals, yielding BDI Score = 9.218 × Classification − 9.549 with R² = 0.91. That single identity converts qualitative agreement among models into the paper's quantitative reliability claim.

What would settle it

Have clinicians or the task organizers score the same personas' transcripts with the BDI-II and compare those ground-truth totals to the LLMs' reported totals; if classification accuracy against ground truth is near chance or the correlation (not the internal regression) is near zero, the reliability claim fails. A more direct check is to recompute the label-encoding regression separately for each model and on held-out conversations to see whether the R² = 0.91 persists when classification and total score are not produced by the same prompt instance.

Watch

Extended reading notes

Core claim

The paper's central claim is that prompt engineering alone—one carefully specified system prompt containing BDI-II symptom anchors, an interaction state machine, and a JSON reporting schema—can make LLMs produce what it calls 'plausible depression assessments' of simulated personas. Because ground-truth labels were unavailable, the evidence is internal: across the four models, the label-encoded classification levels and total BDI-II scores obey BDI Score = 9.218 × Classification − 9.549 with R² = 0.91 (p < 0.001). The authors take this fit to confirm both the reliability of the classification system and the clinical significance of the severity distinctions, since each category step corresponds to roughly 9.2 BDI points. They also report that models agree closely on symptoms like loss of libido, suicidal thoughts, and punishment feelings but diverge on appetite changes and agitation, and that the models are poor at summing their own item-level scores, so totals must be computed externally.

Load-bearing premise

The load-bearing premise is that the LLM's self-reported JSON fields—per-item scores, total BDI-II score, and severity classification—actually reflect the persona's depressive state rather than the model's own internally consistent story; the paper itself says it cannot independently verify these item scores because the LLM's reasoning is opaque.

Editorial extensions

If this is right

  • A single reusable prompt can make multiple LLMs produce BDI-II-aligned severity estimates, so a screening conversation would not need to be re-engineered for each model.
  • Models stabilize their confidence and severity estimates roughly by turn 10–15, suggesting a short structured interview could suffice for a provisional assessment.
  • The slope of about 9.2 BDI points per severity category implies that the models' categories are separated by clinically meaningful intervals, if the regression extends beyond this dataset.
  • Because the models' own totals are often arithmetically wrong (correct on 23–61% of summaries), a deployed system should compute BDI-II totals in code from per-item scores rather than trusting the LLM's sum.
  • Symptoms with low cross-model agreement, such as appetite changes and agitation, should be routed to human review, while high-agreement signals like suicidal thoughts can be tracked automatically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The R² = 0.91 is a statement about internal consistency, not about validity against real depression; a clinician-rated ground-truth comparison is needed before these scores could support screening decisions.
  • Because classification and total score come from the same model on the same conversation, the regression may partly reflect the prompt's own anchoring; a sharper test would compute the fit with the classification from one model and the total score from another, or with held-out conversations.
  • The cross-model agreement hierarchy suggests a division of labour: high-agreement BDI-II items could serve as stable automatic markers, while low-agreement items (appetite, agitation, worthlessness-appearance) could be specifically targeted for better prompting or human oversight.
  • A natural extension would be to use the LLM's per-turn reasoning traces as a free-text supervision signal for smaller models, transferring the conversational assessment to settings where large proprietary models are unavailable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports the DS@GT team's participation in two eRisk 2025 tasks. For Task 2, the authors present a voting classifier over engineered features and a LightGBM classifier over MentalRoBERTa embeddings augmented with a hand-designed temporal attention mechanism; both runs achieve F1 = 0.20 in decision-based evaluation, with the embedding-based run performing better on ranking metrics after the first writing. For the Pilot Task, the authors use a single system prompt (Appendix A) with several LLMs acting as conversational assessors that conduct BDI-II-based interviews of simulated personas and emit structured JSON after every turn. Since individual ground-truth labels were not available to the team, the evaluation relies on cross-model agreement and internal consistency; the best run achieves DCHR = 0.50, ADODL = 0.89 and ASHR = 0.27. The central evidence for reliability is a linear regression (Eq. (1)) between the same-turn, same-model 'classification_suggestion' and 'total_bdi_score' fields, yielding R² = 0.91, which the conclusions interpret as confirming both reliability and clinical significance.

Significance. If the pilot claim were supported, the prompt-engineering pipeline would provide a low-cost screening instrument aligned with BDI-II without direct depression questions. The paper's concrete strengths are the full, publicly available system prompt in Appendix A, the honest reporting that ground-truth labels were unavailable at development time, and the explicit acknowledgment of the low arithmetic summation accuracy in Table 13 and of the fact that self-reported confidence is not a true measure of evaluation state. These reporting practices are valuable. However, the central reliability argument is currently circular, so the actual contribution is a reproducible prompt design and an exploratory consistency analysis rather than a validated screening tool.

major comments (4)
  1. [§3.5.1, Eq. (1), Appendix A] The R² = 0.91 regression is computed from two variables emitted by the same LLM in the same JSON turn, and the system prompt in Appendix A hard-codes BDI score ranges for each classification category ("0-10 Normal, 11-16 Mild, 17-20 Borderline, 21-30 Moderate, 31-40 Severe, 41+ Extreme"). The slope of 9.218 and the predicted category means (approximately 8.9, 18.1, 27.3, 36.5, 45.8) are close to the midpoints of those prompt-defined bands, making the regression plausibly a reflection of the prompt's own scoring anchors rather than of independent reliability or clinically meaningful severity differences. The conclusion in §4 that this "confirms both the reliability of the LLM-based classification system and the clinical significance of severity distinctions" is therefore unsupported. The authors should either reframe Eq. (1) explicitly as a prompt-internal consistency check or supply a null-model calibration, for example by sampling total_bdi_score uniformly within the prompt-defined band for each encoded label and showing that the observed R² exceeds the null distribution.
  2. [§3.3.2, Table 13] The paper states that item scores "cannot be independently verified" and reports that LLMs correctly sum the 21 item scores into total_bdi_score only 23–61% of the time (Table 13). Since total_bdi_score is the dependent variable in Eq. (1) and also feeds the official ADODL metric, the manuscript must clarify whether the official totals were the LLM-emitted values or were recomputed from item scores by the authors, and must discuss how the low arithmetic accuracy affects the reliability of the internal-consistency analysis and of the reported ADODL results. Without this clarification, the central analysis is built on a quantity whose accuracy the paper itself calls into question.
  3. [§2.4.2] The temporal attention mechanism rests on a hand-specified sparse vector with weights [0.9, 0.7, 0.8, 0.6, 0.7] assigned to embedding indices 15, 42, 127, 256 and 512, with no evidence that these MentalRoBERTa dimensions correspond to depression indicators, and on a linear recency weighting from 0.1 to 1.0. No ablation is provided comparing this mechanism with a simple mean-pooling baseline or with the voting classifier, and Table 2 shows identical decision-based results for both Task 2 runs. The contribution of the custom attention mechanism is therefore not demonstrated; the authors should either provide an ablation or explicitly frame the mechanism as an untested design choice.
  4. [§3.2, §3.3, Table 7] The paper says that "the pilot task does not provide ground-truth annotations" and that "traditional accuracy, precision, and recall cannot be calculated," yet Table 7 reports official DCHR, ADODL and ASHR scores that are computed by the organizers against ground-truth BDI-II severity. The manuscript should distinguish between (i) organizer-provided aggregate ground-truth-based scores obtained on the leaderboard and (ii) the team's post-hoc internal-consistency analysis. This distinction is essential for interpreting what the reported metrics mean and for assessing the pilot claim.
minor comments (5)
  1. [Abstract vs. §3.4.1] The abstract says "second on the official leaderboard," while Section 3.4.1 says run 1 "ranked third overall with respect to ADODL"; please make these statements consistent.
  2. [§3.4.2, Tables 7 and 8] The text says "We submitted four runs, but only two were included in the statistics by the organizers," but Table 7 lists four DS-GT runs and Table 8 reports two; please clarify which runs were scored and which were excluded from the run statistics.
  3. [§2.4.2] The object described as a "predefined attention matrix of dimension 768" is actually a vector of 768 values; the terminology should be corrected to avoid confusion.
  4. [§3.3.1 and Appendix A] The example JSON in Section 3.3.1 uses the field 'output_message', while the system prompt in Appendix A refers to generating the 'persona_response' text; these names should be aligned in the prompt description and the example.
  5. [§3.4.2] The word "flucuate" should be "fluctuate".

Circularity Check

1 steps flagged · score 6.0 of 10

The R²=0.91 regression used to 'confirm reliability and clinical significance' is a self-consistency check between two outputs of the same LLM under a prompt that already supplies the BDI score bands for each category.

  1. self definitional [Section 3.5.1, Eq. (1); Section 4; Appendix A ('BDI Reference Material')]
    "Linear regression analysis reveals a strong relationship between classification level and BDI-II score (R2 = 0.91, p <0.001): BDI Score = 9.218× Classification− 9.549 (1)... This dual finding confirms both the reliability of the classification system and the clinical significance of the severity distinctions made by the LLM agents. ... BDI Score Ranges: 0-10 Normal, 11-16 Mild, 17-20 Borderline, 21-30 Moderate, 31-40 Severe, 41+ Extreme."

    The two regressed variables are produced by the same LLM in the same JSON evaluation block: classification_suggestion and total_bdi_score. The system prompt hands the model the exact numeric bands for each category, so following the prompt makes the category a coarse binning of the same score being regressed. The near-one R2 and the slope of 9.218 per category (about one BDI band width) are artifacts of those anchors, not evidence that the assessments track depressive severity. Section 3.3.2 says item scores 'cannot be independently verified,' and Table 13 shows correct arithmetic summation only 23-61% of the time, so total_bdi_score is not an independent criterion. Claiming that Eq. (1) confirms reliability or clinical significance equates self-consistency with external validation.

full rationale

The official leaderboard results (DCHR, ADODL, ASHR) are computed against organizer-provided ground truth and are not circular. Task 2 is a standard supervised classification pipeline with no reduction of the reported metrics to its inputs. The circularity is confined to the pilot-task interpretation. There, the paper explicitly states that ground truth is unavailable and that evaluation relies on internal consistency and cross-model agreement; the R2 = 0.91 regression in Eq. (1) is precisely such an internal-consistency calculation. Because the system prompt (Appendix A) defines the BDI score ranges corresponding to each classification label and the same model generates both classification_suggestion and total_bdi_score, the strong linear fit largely re-expresses the prompt's own scoring bands. The paper's conclusion that this 'confirms both the reliability of the LLM-based classification system and the clinical significance of severity distinctions' therefore goes beyond what the analysis can support: it treats a within-model correlation as if it were external validation. The paper's own caveats about unverifiable item scores and poor score-summation accuracy (Table 13) reinforce that the regression is not an independent check. No self-citation chain is load-bearing here; the issue is the self-definitional status of the pilot's central evidence.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest on the organizers' ground truth, the representativeness of LLM personas, and the reliability of LLM-generated self-assessments. It also introduces hand-chosen attention parameters and uses a self-referential regression as evidence. No new physical or formal entities are introduced; the ad hoc attention matrix is a parameter, not an entity.

free parameters (4)
  • Content attention weights = [0.9, 0.7, 0.8, 0.6, 0.7] at embedding indices 15, 42, 127, 256, 512
    Hand-selected in Section 2.4.2 to highlight 'Depression indicators'; no data-driven selection or ablation is provided, and the claim that these dimensions encode depression is unsupported.
  • Temporal weighting schedule = linear weights from 0.1 (earliest) to 1.0 (most recent)
    Chosen by hand in Section 2.4.2 to favor recent posts; no empirical comparison of alternative schedules is presented.
  • Linear regression coefficients in Eq. (1) = slope 9.218, intercept -9.549
    Fitted to the LLM's own classification levels and BDI scores, then used as evidence of consistency; this is a fit, not a prediction.
  • LightGBM hyperparameters = 5000 n_estimators, learning_rate 0.01, max_depth 7, scale_pos_weight 8.23
    Reported in Section 2.4.2; scale_pos_weight follows the class ratio, while the other values are tuning choices that affect the Task 2 results.
assumptions (5)
  • domain assumption BDI-II total score and categories are a valid operationalization of depression severity for the personas.
    The pilot task defines ground truth by BDI-II-derived labels; the paper relies on this without questioning whether LLM personas exhibit clinically valid depression.
  • domain assumption The 12 LLM personas created by the organizers accurately simulate real user writings and depressive states.
    Used throughout the pilot task; if personas are not representative, the assessments and symptom analyses do not transfer to real users. See Section 3.2.
  • ad hoc to paper MentalRoBERTa embeddings contain interpretable dimensions whose fixed indices can be weighted to emphasize depression indicators.
    Section 2.4.2 assigns weights to specific hidden dimensions with no probe, interpretability analysis, or citation; this is an ad hoc assumption introduced for this paper.
  • ad hoc to paper Cross-model agreement and internal consistency are meaningful substitutes for accuracy when ground truth is absent.
    Section 3.3 states ground-truth labels are unavailable, so evaluation focuses on consistency; the leap from consistency to reliability appears in Sections 3.5 and 4.
  • standard math Standard statistical tools (least squares, R2) apply to the LLM-generated ordinal labels.
    The R2 regression in Eq. (1) treats label-encoded categories as numeric; this is standard practice but assumes interval-level properties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DS@GT at eRisk 2025: From prompts to predictions, benchmarking early depression detection with conversational agent based assessments and temporal attention models." pith.science (2026). https://pith.science/paper/RDKEURDZ

@misc{pith2026250710958,
  author       = {Pith},
  title        = {Pith review of: DS@GT at eRisk 2025: From prompts to predictions, benchmarking early depression detection with conversational agent based assessments and temporal attention models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDKEURDZ}},
  note         = {Machine review of arXiv:2507.10958}
}
read the original abstract

This Working Note summarizes the participation of the DS@GT team in two eRisk 2025 challenges. For the Pilot Task on conversational depression detection with large language-models (LLMs), we adopted a prompt-engineering strategy in which diverse LLMs conducted BDI-II-based assessments and produced structured JSON outputs. Because ground-truth labels were unavailable, we evaluated cross-model agreement and internal consistency. Our prompt design methodology aligned model outputs with BDI-II criteria and enabled the analysis of conversational cues that influenced the prediction of symptoms. Our best submission, second on the official leaderboard, achieved DCHR = 0.50, ADODL = 0.89, and ASHR = 0.27.

Figures

Figures reproduced from arXiv: 2507.10958 by the authors.

Figure 1
Figure 1. Box-plot comparison of posting behaviours across three metrics [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Comparison of average confidence over time. Note that averages ignore null values and thus flucuate [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Comparison of BDI scores by model and agent. Null values are filled from the last valid round [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Relationship between classification level (mapped numerically) and final BDI-II score. A strong linear correlation (𝑅2 = 0.91) confirms consistency between classification and severity estimation. We next analyzed the key_symptoms field, which encodes which of the 21 BD…
Figure 5
Figure 5. Figure 5: Grouped bar chart comparing the four most frequently-mentioned BDI-II symptoms identified at assessment turn 20. To quantify inter-model agreement, we computed the standard deviation of BDI-II item scores across four language models (Claude-3.7-sonnet, GPT-4o, Gemini-2…
Figure 6
Figure 6. Figure 6: Model agreement rates across BDI-II symptoms. Lower standard deviation indicates higher consensus across models [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Polar plot of average BDI-II item scores across models. Each axis (q1–q21) corresponds to a BDI-II symptom item. The radial score represents average symptom severity assigned by each model. Models converge on some core items but diverge significantly on others. Finally…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 52 canonical work pages

  1. [1]

    Coppersmith, R

    G. Coppersmith, R. Leary, E. Whyne, T. Wood, Quantifying mental health signals in twitter, in: Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, 2015, pp. 51–60

    work page 2015

  2. [2]

    D. E. Losada, F. Crestani, J. Parapar, Overview of erisk: Early risk detection on the internet, in: International Conference of the CLEF Association, Springer, Cham, 2017, pp. 346–360

    work page 2017

  3. [3]

    Parapar, A

    J. Parapar, A. Perez, X. Wang, F. Crestani, Overview of erisk 2025: Early risk prediction on the internet (extended overview), in: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2025), Madrid, Spain, 9-12 September, 2025, volume To be published of CEUR Workshop Proceedings, CEUR-WS.org, 2025

    work page 2025

  4. [4]

    Parapar, A

    J. Parapar, A. Perez, X. Wang, F. Crestani, Overview of erisk 2025: Early risk prediction on the internet, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction - 16th International Conference of the CLEF Association, CLEF 2025, Madrid, Spain, September 9-12, 2025, Proceedings, Part II, volume To be published of Lecture Notes in Comput...

    work page 2025

  5. [5]

    Parapar, P

    J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, Overview of eRisk 2023: Early Risk Prediction on the Internet, in: A. Arampatzis, E. Kanoulas, T. Tsikrika, S. Vrochidis, A. Giachanou, D. Li, M. Aliannejadi, M. Vlachos, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction, Springer Nature Switzerland...

    work page 2023

  6. [6]

    Parapar, P

    J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, Overview of eRisk 2024: Early Risk Prediction on the Internet, in: L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, G. M. Di Nunzio, L. Soulier, P. Galuščáková, A. García Seco de Herrera, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction, Springer Natu...

    doi:10.1007/978-3-031-71908-0_4 2024

  7. [7]

    Barachanou, F

    A. Barachanou, F. Tsalakanidou, S. Papadopoulos, REBECCA at eRisk 2024: Search for Symptoms of Depression Using Sentence Embeddings and Prompt-Based Filtering (2024)

    work page 2024

  8. [8]

    Martinez-Romo, L

    J. Martinez-Romo, L. Araujo, X. Larrayoz, M. Oronoz, A. Pérez, OBSER-MENH at eRisk 2023: Deep Learning-Based Approaches for Symptom Detection in Depression and Early Identification of Pathological Gambling Indicators (2023)

    work page 2023

Show all 58 references
  1. [9]

    F. A. Sakib, A. A. Choudhury, O. Uzuner, MASON-NLP at eRisk 2023: Deep Learning-Based Detection of Depression Symptoms from Social Media Texts, 2023. URL: http://arxiv.org/abs/2310. 10941. doi:10.48550/arXiv.2310.10941, arXiv:2310.10941 [cs]

    2023 doi arXiv

  2. [10]

    A. P. Bacuñana, I. S. Bedmar, APB-UC3M at eRisk 2024: Natural Language Processing and Deep Learning for the Early Detection of Mental Disorders (2024)

    2024

  3. [11]

    Hutto, E

    C. Hutto, E. Gilbert, VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text, Proceedings of the International AAAI Conference on Web and Social Media 8 (2014) 216–225. URL: https://ojs.aaai.org/index.php/ICWSM/article/view/14550. doi:10.1609/icwsm....

    2014 doi

  4. [12]

    URL: https://huggingface.co/mental/ mental-roberta-base

    mental/mental-roberta-base · Hugging Face, 2021. URL: https://huggingface.co/mental/ mental-roberta-base

    2021

  5. [13]

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, T.-Y. Liu, LightGBM: A Highly Efficient Gradient Boosting Decision Tree, in: Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017. URL: https://proceedings.neurips.cc/paper_files/...

    2017

  6. [14]

    Weizenbaum, ELIZA—a computer program for the study of natural language communication between man and machine, Commun

    J. Weizenbaum, ELIZA—a computer program for the study of natural language communication between man and machine, Commun. ACM 9 (1966) 36–45. URL: https://dl.acm.org/doi/10.1145/ 365153.365168. doi:10.1145/365153.365168

    1966

  7. [15]

    R. Wang, S. Milani, J. C. Chiu, J. Zhi, S. M. Eack, T. Labrum, S. M. Murphy, N. Jones, K. Hardy, H. Shen, F. Fang, Z. Z. Chen, PATIENT-Ψ: Using Large Language Models to Simulate Patients for Training Mental Health Professionals, 2024. URL: http://arxiv.org/abs/2405.19660. doi:...

    2024 doi arXiv

  8. [16]

    S. Chen, M. Wu, K. Q. Zhu, K. Lan, Z. Zhang, L. Cui, LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluation, 2023. URL: http://arxiv.org/abs/2305.13614. doi:10.48550/arXiv.2305.13614, arXiv:2305.13614 [cs]

    2023 doi arXiv

  9. [17]

    Kaywan, K

    P. Kaywan, K. Ahmed, A. Ibaida, Y. Miao, B. Gu, Early detection of depression using a conver- sational AI bot: A non-clinical trial, PLOS ONE 18 (2023) e0279743. URL: https://journals.plos. org/plosone/article?id=10.1371/journal.pone.0279743. doi:10.1371/journal.pone.0279743, ...

    2023 doi

  10. [18]

    K. K. Fitzpatrick, A. Darcy, M. Vierhile, Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial, JMIR Mental Health 4 (2017) e7785. URL: https://mental....

    2017 doi

  11. [19]

    Ferrario, J

    A. Ferrario, J. Sedlakova, M. Trachsel, The Role of Humanization and Robustness of Large Language Models in Conversational Artificial Intelligence for Individuals With Depression: A Critical Analysis, JMIR Mental Health 11 (2024) e56569. URL: https://mental.jmir.org/2024/1/e56...

    2024 doi

  12. [20]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, et al., Language models are few-shot learners, in: Advances in Neural Information Processing Systems, volume 33, 2020

    2020

  13. [21]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language Models Are Unsupervised Multitask Learners, Technical Report, OpenAI, 2019

    2019

  14. [22]

    Conversational Depression Detection via LLMs

    PACE, Partnership for an Advanced Computing Environment (PACE), 2017. URL: http://www. pace.gatech.edu. A. System Prompt The system prompt for the LLM-based agent in unrendered markdown. The initial system prompt was generated using Gemini 2.5-pro via http://gemini.google.com ...

    2017

  15. [23]

    Build rapport

    **Empathetic Conversational Partner:** Engage naturally, supportively, and in detail with an LLM persona simulating a human user. Build rapport

  16. [24]

    **Systematic Information Gatherer:** Proactively and smoothly guide the conversation to touch upon various life domains relevant to the Beck Depression Inventory (BDI) criteria (provided below for reference) within approximately 20 turns

  17. [25]

    **Objective Assessor:** Conduct a rigorous, structured assessment to determine if the persona exhibits signs consistent with depression, based *only* on conversational inference. Your approach should mirror the objectivity and methodical rigor expected in professional psycholo...

  18. [26]

    This is a strict task rule

    **No Direct Questions about Depression:** You MUST NOT ask the persona directly if they are depressed, feeling down, sad, or about their mental health status, symptoms, or diagnosis. This is a strict task rule

  19. [27]

    Avoid projecting or over-interpreting

    **Inference Only:** Your assessment must be based purely on objective inferences drawn from the persona’s language, tone, expressed thoughts, feelings, and behaviors as revealed *naturally* during the conversation. Avoid projecting or over-interpreting

  20. [28]

    **Initiate Conversation:** You are responsible for starting the conversation

  21. [29]

    Your responses can be detailed and natural, not necessarily short

    **Maintain Empathy:** Despite the need for rigor and proactive questioning, your conversational tone MUST remain empathetic, supportive, curious, and non-judgmental at all times. Your responses can be detailed and natural, not necessarily short. **Interaction Protocol:**

  22. [30]

    Hi there, how have things been going for you lately?

    **Initiation:** Begin with a gentle, open-ended question to establish rapport (e.g., "Hi there, how have things been going for you lately?", "Hello, hope you’re having an okay week. What’s been on your mind?")

  23. [31]

    Respond thoughtfully to their messages

    **Empathetic Engagement:** Use active listening (reflecting, clarifying, summarizing) and validate the persona’s feelings appropriately. Respond thoughtfully to their messages

  24. [32]

    What kinds of things have you been finding enjoyable recently?

    **Proactive & Structured Information Gathering:** * During the ’Gathering’ and ’Consolidating’ states, systematically aim to touch upon different BDI-relevant domains (e.g., mood, outlook, self-perception, interests/anhedonia, energy/sleep/appetite, social interactions, concen...

  25. [33]

    Do not rapid-fire questions

    **Avoid Interrogation:** Balance information gathering with rapport building. Do not rapid-fire questions. Allow the conversation to flow naturally. **BDI Reference Material (For Grounding Assessment - DO NOT ASK THESE DIRECTLY):** BDI Score Ranges: 0-10 Normal, 11-16 Mild, 17...

  26. [34]

    (1) I feel sad

    **Sadness:** (0) I do not feel sad. (1) I feel sad. (2) I am sad all the time and I can’ t snap out of it. (3) I am so sad and unhappy that I can’t stand it

  27. [35]

    (1) I feel discouraged about the future

    **Pessimism:** (0) I am not particularly discouraged about the future. (1) I feel discouraged about the future. (2) I feel I have nothing to look forward to. (3) I feel the future is hopeless and that things cannot improve

  28. [36]

    (1) I feel I have failed more than the average person

    **Past Failure:** (0) I do not feel like a failure. (1) I feel I have failed more than the average person. (2) As I look back on my life, all I can see is a lot of failures. (3) I feel I am a complete failure as a person

  29. [37]

    (1) I don’t enjoy things the way I used to

    **Loss of Pleasure (Anhedonia):** (0) I get as much satisfaction out of things as I used to. (1) I don’t enjoy things the way I used to. (2) I don’t get real satisfaction out of anything anymore. (3) I am dissatisfied or bored with everything

  30. [38]

    (1) I feel guilty a good part of the time

    **Guilty Feelings:** (0) I don’t feel particularly guilty. (1) I feel guilty a good part of the time. (2) I feel quite guilty most of the time. (3) I feel guilty all of the time

  31. [39]

    (1) I feel I may be punished

    **Punishment Feelings:** (0) I don’t feel I am being punished. (1) I feel I may be punished. (2) I expect to be punished. (3) I feel I am being punished

  32. [40]

    (1) I am disappointed in myself

    **Self-Dislike:** (0) I don’t feel disappointed in myself. (1) I am disappointed in myself. (2) I am disgusted with myself. (3) I hate myself

  33. [41]

    (1) I am critical of myself for my weaknesses or mistakes

    **Self-Criticalness:** (0) I don’t feel I am any worse than anybody else. (1) I am critical of myself for my weaknesses or mistakes. (2) I blame myself all the time for my faults. (3) I blame myself for everything bad that happens

  34. [42]

    (1) I have thoughts of killing myself, but I would not carry them out

    **Suicidal Thoughts or Wishes:** (0) I don’t have any thoughts of killing myself. (1) I have thoughts of killing myself, but I would not carry them out. (2) I would like to kill myself. (3) I would kill myself if I had the chance. *(Assess ONLY if spontaneously mentioned/impli...

  35. [43]

    (1) I cry more now than I used to

    **Crying:** (0) I don’t cry any more than usual. (1) I cry more now than I used to. (2) I cry all the time now. (3) I used to be able to cry, but now I can’t cry even though I want to

  36. [44]

    (1) I am slightly more irritated now than usual

    **Agitation/Irritation:** (0) I am no more irritated by things than I ever was. (1) I am slightly more irritated now than usual. (2) I am quite annoyed or irritated a good deal of the time. (3) I feel irritated all the time

  37. [45]

    (1) I am less interested in other people than I used to be

    **Loss of Interest in Others:** (0) I have not lost interest in other people. (1) I am less interested in other people than I used to be. (2) I have lost most of my interest in other people. (3) I have lost all of my interest in other people

  38. [46]

    (1) I put off making decisions more than I used to

    **Indecisiveness:** (0) I make decisions about as well as I ever could. (1) I put off making decisions more than I used to. (2) I have greater difficulty in making decisions more than I used to. (3) I can’t make decisions at all anymore

  39. [47]

    (1) I am worried that I am looking old or unattractive

    **Worthlessness (Body Image):** (0) I don’t feel that I look any worse than I used to. (1) I am worried that I am looking old or unattractive. (2) I feel there are permanent changes in my appearance that make me look unattractive. (3) I believe that I look ugly

  40. [48]

    (1) It takes an extra effort to get started at doing something

    **Loss of Energy (Work Difficulty):** (0) I can work about as well as before. (1) It takes an extra effort to get started at doing something. (2) I have to push myself very hard to do anything. (3) I can’t do any work at all

  41. [49]

    (1) I don’t sleep as well as I used to

    **Changes in Sleeping Pattern:** (0) I can sleep as well as usual. (1) I don’t sleep as well as I used to. (2) I wake up 1-2 hours earlier than usual and find it hard to get back to sleep. (3) I wake up several hours earlier than I used to and cannot get back to sleep

  42. [50]

    (1) I get tired more easily than I used to

    **Tiredness or Fatigue:** (0) I don’t get more tired than usual. (1) I get tired more easily than I used to. (2) I get tired from doing almost anything. (3) I am too tired to do anything

  43. [51]

    (1) My appetite is not as good as it used to be

    **Changes in Appetite:** (0) My appetite is no worse than usual. (1) My appetite is not as good as it used to be. (2) My appetite is much worse now. (3) I have no appetite at all anymore

  44. [52]

    (1) I have lost more than five pounds

    **Weight Loss:** (0) I haven’t lost much weight, if any, lately. (1) I have lost more than five pounds. (2) I have lost more than ten pounds. (3) I have lost more than fifteen pounds. *(Assess ONLY if spontaneously mentioned/implied or strongly indicated by other cues like app...

  45. [53]

    (1) I am worried about physical problems like aches, pains, upset stomach, or constipation

    **Somatic Preoccupation (Health Worries):** (0) I am no more worried about my health than usual. (1) I am worried about physical problems like aches, pains, upset stomach, or constipation. (2) I am very worried about physical problems and it’s hard to think of much else. (3) I...

  46. [54]

    (1) I am less interested in sex than I used to be

    **Loss of Libido:** (0) I have not noticed any recent change in my interest in sex. (1) I am less interested in sex than I used to be. (2) I have almost no interest in sex. (3) I have lost interest in sex completely. *(Assess ONLY if spontaneously mentioned/ implied. DO NOT PR...

  47. [55]

    ‘input_message ‘: (String) The actual text message the persona responds with, or "start"

  48. [56]

    ‘output_message ‘: (String) The actual text message you generated to send to the persona

  49. [57]

    ‘next_step_reasoning ‘: (String) Your explicit reasoning for choosing this specific ‘ output_message ‘. Explain how it connects to the assessment goals (e.g., exploring a BDI domain, clarifying a previous point, building rapport) based on the current conversation context and a...

  50. [58]

    Initializing

    ‘evaluation‘: (Object) Your internal BDI-based evaluation, containing: * ‘assessment_turn ‘: (Integer) Current turn number. * ‘assessment_state ‘: (String Enum: "Initializing", "Gathering", "Consolidating", " Concluding", "Finalized") Current phase. * ‘assessment_complete ‘: (...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.