Pith. sign in

REVIEW 4 major objections 5 minor 12 references

Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read This paper claims that guiding LLMs through standardized psychological questionnaires with adaptive retrieval lets them match or outperform supervised state-of-the-art depression screening on Reddit benchmarks with no training data.

desk verdict Questionnaire-guided RAG for LLM mental health screening is a genuine idea, but the SOTA claim is not yet supported: too much configuration search on 20 users and the BDI/BDI-II cutoff inconsistency blur the headline numbers. read the letter →

arxiv 2501.00982 v2 pith:JKGAMAA7 submitted 2025-01-02 cs.CL cs.AI

classification cs.CLcs.AI
keywords mentalhealthscreeningadaptiveretrieval-augmentedgenerationzero-shotLLMclassificationBDI-IIsocialmediadepressiondetectionpsychologicalquestionnaireseRiskbenchmarkinterpretableassessment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models can act as psychological assessors if their predictions are channeled through standardized questionnaires instead of being asked directly for a diagnosis. The authors build a retrieval-augmented pipeline that, for each item of the Beck Depression Inventory-II, retrieves the Reddit posts most semantically similar to the item's four response choices and asks an LLM to score the item from 0 to 3. On the eRisk 2019 and 2020 severity-assessment benchmarks this unsupervised procedure matches or outperforms the best supervised systems, and the same questionnaire-guided design beats directly prompting the LLM to screen for depression. The paper also reports proof-of-concept extensions to self-harm, anorexia, and pathological gambling questionnaires. If true, this would make interpretable, zero-training-data mental health screening from social media feasible at scale.

What carries the argument

The central mechanism is the item-query retrieval loop. Each questionnaire item is turned into four embedding queries, one per response choice, and for each choice the ABIDE algorithm selects an adaptive neighborhood size k* so that exactly the posts whose semantics stay coherent around that choice are retrieved from the user's post history. These retrieved posts are then fed to the LLM in a zero-shot prompt (direct or chain-of-thought) that asks for the item score in 0-3, and the scores are summed into the BDI-II total and mapped to depression severity. The questionnaire itself is the load-bearing intermediate representation: it decomposes the diagnosis into clinically validated items, and the adaptive neighborhood replaces a fixed retrieval threshold with a per-item, per-user choice of how many posts to use.

What would settle it

Build a small evaluation set where clinicians label, for a sample of users and BDI-II items, which posts support which 0-3 score, and measure whether the posts retrieved by embedding similarity to each choice match those labels substantially better than chance. Alternatively, ablate the retrieval: feed each LLM the same number of randomly selected posts instead of the adaptively retrieved ones; if the questionnaire scores do not drop, the retrieval stage is not what carries the result.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that structuring LLM assessment through clinically validated questionnaire items, with adaptive retrieval of relevant social media posts for each item, lets the LLM fill out the BDI-II accurately enough to match or beat state-of-the-art supervised models on the eRisk 2019 and 2020 Reddit benchmarks, using no training data at all. The best configurations reach 55.00% depression-category hit rate (DCHR) on eRisk 2019 and 52.86% with a small ensemble on eRisk 2020, and improve item-level closeness (ACR) in both collections. The paper further claims that questionnaire-guided screening outperforms direct prompting of the LLM for depression detection, and that the same pipeline transfers to self-harm, anorexia, and pathological gambling questionnaires, where it achieves the highest F1 score among compared systems on the anorexia and gambling early-detection tasks. The overall assertion is that standardized questionnaires provide the missing structured bridge between unstructured text and clinical categories, making LLM predictions more accurate and more interpretable.

Load-bearing premise

The whole pipeline rests on the assumption that semantic similarity between a response choice's text and a user's post is reliable evidence of which symptom level the user would endorse, and that the eRisk self-reported BDI-II answers are trustworthy ground truth; if those two links break, the item scores have no basis.

Editorial extensions

If this is right

  • If the central claim holds, supervised classifiers are not required for solid BDI-II severity screening on Reddit-style text; an unsupervised LLM pipeline can match them.
  • Questionnaire-guided prompting becomes a cheap alternative to direct LLM diagnosis, with item-level traceability for why a score was given.
  • The same adaptive-retrieval template can be pointed at new clinical instruments without training examples, potentially shortening the path from a new questionnaire to a new screening system.
  • Retrieval quality is load-bearing: the ablation showing aRAG beats feeding the LLM all posts implies post selection is a major source of the gain, not just LLM size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not test is validating the item-choice retrieval against human relevance judgments; if the neighborhoods align with clinician-endorsed evidence, the method could support evidence-traceable screening reports.
  • Since the benchmark ground truth is self-reported BDI-II, the reported gains inherit the instrument's measurement noise; comparing against clinician-administered structured interviews would be a stricter test and might change the ranking.
  • The framework's reliance on phrase-level choice texts means questionnaires with abstract or situational items may not embed well; testing with such instruments would map the boundary of the approach.
  • The method's zero-training property makes it attractive for resource-limited screening, but population skew in who posts on social media is a likely confound for generalizing beyond the benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes aRAG, an unsupervised two-step framework for mental health screening from Reddit post histories. For each BDI-II item, dense retrieval with an adaptive neighborhood size selects a user's most relevant posts using the item's four response choices as queries; a zero-shot LLM then scores the item, and item scores are summed and mapped to depression severity. The authors report that this approach matches or outperforms state-of-the-art supervised models on the eRisk 2019 and 2020 depression severity tasks, that questionnaire-guided prompting beats direct LLM screening, and that the approach extends to self-harm, anorexia, and pathological gambling questionnaires. The paper also includes ablation studies on the retrieval strategy, prompting strategy, and the role of the adaptive k* parameter.

Significance. If the empirical claims hold, the paper offers a valuable new direction: an interpretable, training-free pipeline that links social-media language to a validated psychometric instrument, with code released for reproducibility. The systematic sweep over six LLMs, two prompting strategies, and ten retrievers is a useful resource for the community. However, the central 'matches or outperforms SOTA' claim is currently not statistically supported due to configuration selection on a 20-user set and an incorrect treatment of one evaluation metric, so the significance is contingent on a re-analysis.

major comments (4)
  1. [Section 5.1, Table 4] The paper states that for all considered metrics the higher the value the better, but ADODL (Average Difference between Overall Depression Levels) is an error measure for which lower values are better. Under the correct direction, the reported ADODL values in Table 4 (e.g., 85.56% for Qwen 2.5 70B + Direct on eRisk 2019 versus 81.03% for CAMH) indicate that the proposed method is worse than the baseline on this metric, contradicting the claim that the approach outperforms benchmarks across all metrics. All ADODL comparisons and the associated conclusions in Sections 5.1 and 6 need to be re-evaluated.
  2. [Section 5.1, Table 4 and Figure 3] The configurations reported in Table 4 were selected as the best among 120 combinations (6 LLMs × 2 prompting strategies × 10 retrievers) evaluated on the eRisk 2019 severity dataset, which contains only 20 users (Table 2). With n=20, the selected best configurations are likely overfit to idiosyncrasies of individual users, and a single-user shift changes DCHR by 5 percentage points. The eRisk 2020 results inherit this selection, and no confidence intervals, bootstrap, or significance tests are provided for any SOTA comparison in Table 4. The claimed advantages (e.g., 52.86% vs 50.00% DCHR on eRisk 2020, or 55% vs 45% on eRisk 2019, which is 2 users) are within plausible noise, so the central claim of matching or outperforming SOTA is not established.
  3. [Section 5.1, Table 5] The headline DCHR results in Table 4 use the eRisk challenge's BDI cutoff ranges, not the BDI-II ranges that the questionnaire actually uses. Table 5 shows that switching to the correct BDI-II cutoffs changes DCHR substantially for some configurations (e.g., Claude + CoT + MiniLM-L12 on eRisk 2019 drops from 50% to 35%). Because the configurations were selected using the BDI cutoffs and the BDI-II ensemble DCHR is not reported, the paper's claim about 'when completing the BDI-II' is not supported as stated; results under the correct cutoffs must be reported and the selection procedure re-done if necessary.
  4. [Section 4.2, Figure 1] The entire pipeline rests on the premise that embedding the four BDI-II response choices as retrieval queries returns Reddit posts whose semantic similarity to those choice texts is reliable evidence of the user's true item score. This mapping is not validated against relevance judgments, retrieval gold standards, or any qualitative analysis of the retrieved posts. Without such validation, the interpretability and mechanism of the approach remain unverified; end-to-end accuracy alone does not establish that the item-choice-to-post mapping is functioning as intended.
minor comments (5)
  1. [Section 5.1] The sentence 'For all the metrics considered, the higher the value, the better' is incorrect for ADODL, as noted in the major comments; this should be corrected to avoid reader confusion.
  2. [Section 5.2] The text says 'our approach do not need any training data and requires no training data'—the repetition should be removed and the grammar corrected.
  3. [Acknowledgements] The phrase 'partly been founded by' should read 'partly been funded by'.
  4. [Section E.1] The no-retrieval baseline is described as inputting 'all posts that fit within each LLM's context window based on timestamp order'; since not all posts are used, the truncation criterion should be stated precisely for reproducibility.
  5. [Section 5.3] The paper says 'we removed the word depression and related terms' but does not specify which terms were removed; listing the exact terms would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: item scores are generated zero-shot from label-free adaptive retrieval; BDI-II labels are never inputs to the pipeline.

full rationale

The claimed derivation chain is not circular. For each BDI-II item, the method retrieves the user's own Reddit posts using the four response-choice texts as queries (Section 4.2, Figure 1) and then asks an LLM to select the most appropriate score; no ground-truth questionnaire score enters the retrieval step, the ABIDE k* selection, or the LLM prompt. The adaptive neighborhood k* is estimated from post and query embeddings only (Appendix B), and Section E.2 ablates it against fixed k values, so the adaptive component is supported within the paper rather than assumed solely by citation. The self-citations to ABIDE (Noia et al., 2024) and ABIDE-ZS (Ravenda et al., 2025) identify a method and an estimator, but they do not import the paper's conclusion; the reported gains over SOTA are empirical comparisons to external eRisk baselines. Selecting the best of 120 configurations on the 20-user eRisk 2019 set and then reporting eRisk 2020 numbers is a model-selection and statistical-power concern, not circularity, because no parameter is fitted to the 2020 labels and the 2019 numbers are maxima rather than independent predictions. The BDI-II cutoffs are instrument-defined and applied after scoring, not derived from the model outputs. The central claims therefore have independent empirical content; the main weaknesses are statistical, not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric free parameters are fit; the adaptive k* comes from ABIDE, and cutoffs come from the questionnaires. The method does rest on domain assumptions about query-to-post semantic transfer, psychometric cutoffs applying to LLM-generated scores, and self-reported ground truth. No new entities are introduced.

assumptions (4)
  • domain assumption Embeddings of BDI-II response choices, used as queries, retrieve Reddit posts whose semantic similarity is sufficient evidence for the user's item score.
    Stated in Section 4.2 and Figure 1; no relevance judgments or retrieval gold standard validate this mapping.
  • domain assumption Summing LLM-generated item scores and applying BDI or BDI-II cutoffs yields valid depression severity categories.
    Used in Section 5.1; authors note in Limitations that cutoffs may not transfer to social-media screening.
  • domain assumption eRisk self-reported BDI-II responses and binary diagnosis labels are trustworthy ground truth.
    All evaluations use these labels (Section 4.1); there is no independent clinical verification.
  • domain assumption ABIDE's adaptive neighborhood size k* is an appropriate number of posts to retrieve for each item.
    The method relies on ABIDE from (Noia et al., 2024), cited in Appendix B; no retrieval-specific validation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice." pith.science (2026). https://pith.science/paper/JKGAMAA7

@misc{pith2026250100982,
  author       = {Pith},
  title        = {Pith review of: Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JKGAMAA7}},
  note         = {Machine review of arXiv:2501.00982}
}
read the original abstract

In psychological practices, standardized questionnaires serve as essential tools for assessing mental health through structured, clinically-validated questions (i.e., items). While social media platforms offer rich data for mental health screening, computational approaches often bypass these established clinical assessment tools in favor of black-box classification. We propose a novel questionnaire-guided screening framework that bridges psychological practice and computational methods through adaptive Retrieval-Augmented Generation (\textit{aRAG}). Our approach links unstructured social media content and standardized clinical assessments by retrieving relevant posts for each questionnaire item and using Large Language Models (LLMs) to complete validated psychological instruments. Our findings demonstrate two key advantages of questionnaire-guided screening: First, when completing the Beck Depression Inventory-II (BDI-II), our approach matches or outperforms state-of-the-art performance on Reddit-based benchmarks without requiring training data. Second, we show that guiding LLMs through standardized questionnaires can yield superior results compared to directly prompting them for depression screening, while also providing a more interpretable assessment by linking model outputs to clinically validated diagnostic criteria. Additionally, we show, as a proof-of-concept, how our questionnaire-based methodology can be extended to other mental conditions' screening, highlighting the promising role of LLMs as psychological assessors.

Figures

Figures reproduced from arXiv: 2501.00982 by the authors.

Figure 1
Figure 1. Pipeline of the main steps of our architecture. For each user, embeddings are created for each post and for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Prompt templates based on different prompt [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Results associated with the use of different combinations of LLMs, prompting strategies, and retrieval [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance of our approach compared to the best approach in each of the two eRisk editions for the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the performance of the gpt-4o-mini + Direct Prompt combination with MiniLM-L12-v3 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison between aRAG and no-retrieval approaches for closed-source LLMs across [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Distribution of adaptive RAG scores condi [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

  1. [2]

    Contextualized construct representation: lever- aging psychometric scales to advance theory-driven text analysis. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generate...

  2. [8]

    Overview of erisk 2019 early risk prediction on the internet. In Experimental IR Meets Multilin- guality, Multimodality, and Interaction: 10th Inter- national Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, September 9–12, 2019, Proceedings 10, pages 340–357. Springer. David E Losada, Fabio Crestani, and Javier Parapar

  3. [9]

    In European conference on information re- trieval, pages 557–563

    erisk 2020: Self-harm and depression chal- lenges. In European conference on information re- trieval, pages 557–563. Springer. Alba María Mármol-Romero, Salud María Jiménez Zafra, Flor Miriam Plaza del Arco, M Dolores Molina-González, María Teresa Martín Valdivia, and Arturo Montejo-Ráez. 2022. Sinai at erisk@ clef 2022: Approaching early detection of gam...

  4. [10]

    In CLEF (Working Notes)

    Early mental health risk assessment through writing styles, topics and neural models. In CLEF (Working Notes). Elham Mohammadi, Hessam Amini, and Leila Kosseim

  5. [11]

    Measuring the Sever- ity of depression

    Quick and (maybe not so) easy detection of anorexia in social media posts. In CLEF (Working Notes). John F Morgan, Fiona Reid, and J Hubert Lacey. 1999. The scoff questionnaire: assessment of a new screen- ing tool for eating disorders. Bmj, 319(7223):1467– 1468. John A Naslund, Kelly A Aschbrenner, Lisa A Marsch, and Stephen J Bartels. 2016. The future o...

  6. [12]

    golden” sentences (738 in total) as queries to iden- tify semantically similar “silver

    methods obtained their own datasets to per- form standard ML classifiers using engineered fea- tures as linguistic markers. We also refer to recent works (Pérez et al., 2022b, 2023). The two approaches aim to estimate de- pression severity from Reddit posts using BDI- II symptom-based classifiers. While the first ap- proach (Pérez et al., 2022b) uses word...

  7. [321]

    Francesco Denti, Diego Doimo, Alessandro Laio, and Antonietta Mira

    Pearson. Francesco Denti, Diego Doimo, Alessandro Laio, and Antonietta Mira. 2022. The generalized ratios intrinsic dimension estimator. Scientific Reports , 12(1):20005. Fahed Elourajini and Esma Aïmeur. 2022. Aws-ep: A multi-task prediction approach for mbti/big5 person- ality tests. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW)...

  8. [2017]

    erisk 2017: Clef lab on early risk predic- tion on the internet: experimental foundations. In Experimental IR Meets Multilinguality, Multimodal- ity, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Dublin, Ire- land, September 11–14, 2017, Proceedings 8, pages 346–360. Springer. David E Losada, Fabio Crestani, and Javier Parapar

Show all 12 references
  1. [2019]

    Psychology and Psychotherapy: Theory, Research and Practice, 92(2):277–297

    The digital revolution and its impact on mental health care. Psychology and Psychotherapy: Theory, Research and Practice, 92(2):277–297. Ana-Maria Bucur, Adrian Cosma, and Liviu P Dinu

  2. [2020]

    Scientific reports, 10(1):11846

    A deep learning model for detecting mental illness from user content on social media. Scientific reports, 10(1):11846. Shane Latimer, Tanya Covic, Steven R Cumming, and Alan Tennant. 2009. Psychometric analysis of the self-harm inventory using rasch modelling. BMC psychiatry, ...

  3. [2022]

    arXiv preprint arXiv:2204.13569

    Life is not always depressing: Exploring the happy moments of people diagnosed with depression. arXiv preprint arXiv:2204.13569. Sergio Gastón Burdisso, Marcelo Errecalde, and Manuel Montes-y Gómez. 2019. Unsl at erisk 2019: a uni- fied approach for anorexia, self-harm and dep...

  4. [2023]

    IEEE Intelligent Systems, 38(2):15–23

    Will affective computing emerge from foun- dation models and general artificial intelligence? a first evaluation of chatgpt. IEEE Intelligent Systems, 38(2):15–23. Mohammad Atari, Ali Omrani, and Morteza Dehghani

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.