{"id":"c59ee42e-b3cd-42e8-9ba4-2c871b560a6d","arxiv_id":"2501.00982","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An unsupervised adaptive-RAG pipeline that retrieves a user's Reddit posts for each BDI-II item and has an LLM score the item matches or outperforms supervised benchmarks on eRisk depression and other mental-health screening tasks.","lead":"The paper tests whether large language models can fill out standardized psychological questionnaires, like the BDI-II, using a person's Reddit posts retrieved for each question. It reports that this questionnaire-guided approach matches or beats supervised systems on eRisk benchmarks without any training data, and produces more interpretable screening.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is not statistically grounded: best configurations are selected on 20 eRisk 2019 users, and the reported outperformance margins on eRisk 2020 fall within likely noise in the absence of confidence intervals.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing weakness is not primarily the unvalidated semantic mapping; it is the statistical fragility of the SOTA comparison. The reader already flags missing confidence intervals and selection bias in the rationale, and the request for a separate validation split is directly relevant, so my stress-test does not move the verdict. It sharpens the condition: before the headline claim can be accepted, the authors must show that the reported margins are not artifacts of selecting the best among 120 configurations on 20 users, and that the correct BDI-II reparametrization does not overturn the comparison. The semantic-mapping concern remains important for interpretability, but even an imperfect retrieval mapping could yield correct scores if the LLM compensates; the empirical claim fails only if the evaluation is not robust.","tokens_in":18213,"tokens_out":7820,"duration_ms":71706,"concrete_test":"Bootstrap the eRisk 2020 user set 1,000 times; for each bootstrap sample, recompute DCHR and AHR for the exact Table 4 aRAG configurations and for the strongest SOTA baseline (Pérez et al. 2023 Recall for DCHR). Report 95% percentile intervals and paired difference intervals. If the SOTA values lie inside the aRAG intervals, or the paired difference includes zero, the 'outperforms SOTA' claim is not supported. Additionally, repeat the configuration-selection procedure on two random halves of eRisk 2019 and check whether the same configurations are selected; if not, the reported configurations are unstable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 evaluates 120 combinations (6 LLMs × 2 prompt strategies × 10 retrievers) on the eRisk 2019 severity set, which contains only 20 users (Table 2). The configurations shown in Table 4 are those that scored best on these 20 users, and the SOTA comparison on eRisk 2020 inherits this selection. With n=20, the selected 'best' models are almost certainly overfit to idiosyncrasies of individual users; a single user's category shift changes DCHR by 5 percentage points on 2019. No confidence intervals, bootstrap, or significance tests are reported for any Table 4 comparison. The claimed advantages are small: eRisk 2020 DCHR 52.86% (ensemble) vs 50.00% (Pérez et al. 2023) is 2.86 points on 70 users; AHR 41.90% vs 38.97% is 2.93 points; eRisk 2019 DCHR 55% vs 45% is 2 users out of 20. Figure 3 shows extreme variance across configurations, so reporting only the best per metric is selective. Table 5 further shows that switching from eRisk's BDI cutoffs to the correct BDI-II cutoffs changes DCHR by more than 15 points for some configurations (e.g., Claude+CoT+MiniLM-L12: 35% to 50% on 2019), yet the BDI-II ensemble DCHR is not reported. Thus the central claim that aRAG 'matches or outperforms SOTA' is not established; the observed margins may be selection artifacts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes aRAG, an unsupervised two-step framework for mental health screening from Reddit post histories. For each BDI-II item, dense retrieval with an adaptive neighborhood size selects a user's most relevant posts using the item's four response choices as queries; a zero-shot LLM then scores the item, and item scores are summed and mapped to depression severity. The authors report that this approach matches or outperforms state-of-the-art supervised models on the eRisk 2019 and 2020 depression severity tasks, that questionnaire-guided prompting beats direct LLM screening, and that the approach extends to self-harm, anorexia, and pathological gambling questionnaires. The paper also includes ablation studies on the retrieval strategy, prompting strategy, and the role of the adaptive k* parameter.","tokens_in":18573,"tokens_out":4271,"duration_ms":37290,"significance":"If the empirical claims hold, the paper offers a valuable new direction: an interpretable, training-free pipeline that links social-media language to a validated psychometric instrument, with code released for reproducibility. The systematic sweep over six LLMs, two prompting strategies, and ten retrievers is a useful resource for the community. However, the central 'matches or outperforms SOTA' claim is currently not statistically supported due to configuration selection on a 20-user set and an incorrect treatment of one evaluation metric, so the significance is contingent on a re-analysis.","major_comments":[{"comment":"The paper states that for all considered metrics the higher the value the better, but ADODL (Average Difference between Overall Depression Levels) is an error measure for which lower values are better. Under the correct direction, the reported ADODL values in Table 4 (e.g., 85.56% for Qwen 2.5 70B + Direct on eRisk 2019 versus 81.03% for CAMH) indicate that the proposed method is worse than the baseline on this metric, contradicting the claim that the approach outperforms benchmarks across all metrics. All ADODL comparisons and the associated conclusions in Sections 5.1 and 6 need to be re-evaluated.","section":"Section 5.1, Table 4"},{"comment":"The configurations reported in Table 4 were selected as the best among 120 combinations (6 LLMs × 2 prompting strategies × 10 retrievers) evaluated on the eRisk 2019 severity dataset, which contains only 20 users (Table 2). With n=20, the selected best configurations are likely overfit to idiosyncrasies of individual users, and a single-user shift changes DCHR by 5 percentage points. The eRisk 2020 results inherit this selection, and no confidence intervals, bootstrap, or significance tests are provided for any SOTA comparison in Table 4. The claimed advantages (e.g., 52.86% vs 50.00% DCHR on eRisk 2020, or 55% vs 45% on eRisk 2019, which is 2 users) are within plausible noise, so the central claim of matching or outperforming SOTA is not established.","section":"Section 5.1, Table 4 and Figure 3"},{"comment":"The headline DCHR results in Table 4 use the eRisk challenge's BDI cutoff ranges, not the BDI-II ranges that the questionnaire actually uses. Table 5 shows that switching to the correct BDI-II cutoffs changes DCHR substantially for some configurations (e.g., Claude + CoT + MiniLM-L12 on eRisk 2019 drops from 50% to 35%). Because the configurations were selected using the BDI cutoffs and the BDI-II ensemble DCHR is not reported, the paper's claim about 'when completing the BDI-II' is not supported as stated; results under the correct cutoffs must be reported and the selection procedure re-done if necessary.","section":"Section 5.1, Table 5"},{"comment":"The entire pipeline rests on the premise that embedding the four BDI-II response choices as retrieval queries returns Reddit posts whose semantic similarity to those choice texts is reliable evidence of the user's true item score. This mapping is not validated against relevance judgments, retrieval gold standards, or any qualitative analysis of the retrieved posts. Without such validation, the interpretability and mechanism of the approach remain unverified; end-to-end accuracy alone does not establish that the item-choice-to-post mapping is functioning as intended.","section":"Section 4.2, Figure 1"}],"minor_comments":[{"comment":"The sentence 'For all the metrics considered, the higher the value, the better' is incorrect for ADODL, as noted in the major comments; this should be corrected to avoid reader confusion.","section":"Section 5.1"},{"comment":"The text says 'our approach do not need any training data and requires no training data'—the repetition should be removed and the grammar corrected.","section":"Section 5.2"},{"comment":"The phrase 'partly been founded by' should read 'partly been funded by'.","section":"Acknowledgements"},{"comment":"The no-retrieval baseline is described as inputting 'all posts that fit within each LLM's context window based on timestamp order'; since not all posts are used, the truncation criterion should be stated precisely for reproducibility.","section":"Section E.1"},{"comment":"The paper says 'we removed the word depression and related terms' but does not specify which terms were removed; listing the exact terms would improve reproducibility.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The ADODL direction error is a serious factual issue that likely inverts several of the reported comparisons. The configuration-selection problem is also severe: with 120 configurations on 20 users, the reported SOTA results are probably optimistic, and the absence of any uncertainty quantification for Table 4 makes the central claim difficult to evaluate. The BDI-II cutoffs issue further weakens the headline claim. The paper's core idea is novel and worth pursuing, but the empirical analysis needs a thorough rework; I would recommend major revision rather than rejection. The journal should ensure that the authors address the multiple-testing issue, report BDI-II-corrected results, and correct or qualify the ADODL interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe core idea is worth your time: use adaptive dense retrieval to pull the Reddit posts most relevant to each BDI-II item choice, then have a zero-shot LLM score each item, sum to a total, and map to severity. That's a genuinely new way to make LLM-based screening interpretable and training-free. The authors also run a broad grid of LLMs, retrievers, and prompts, ship code, and include an ablation showing retrieval beats feeding raw posts. The extension to self-harm, anorexia, and gambling is a reasonable proof-of-concept, even if those results are more suggestive than definitive.\n\nWhere it falls down is the evaluation. The best configurations are selected from 120 combinations evaluated on the same 20 users of eRisk 2019, then carried to 2020. With n=20, a single user's category shift changes DCHR by 5 points; the reported margins on 2020 are 2-3 points on 70 users. No confidence intervals, bootstrap, or significance tests accompany the SOTA comparisons. That selection effect is not a minor detail—it's the load-bearing part of the 'matches or outperforms SOTA' claim. The BDI-II cutoff issue makes it worse: Table 5 shows that switching from the eRisk BDI cutoffs to the proper BDI-II ranges moves DCHR by 15+ points for some configurations in both directions, yet the headline tables use the eRisk parameterization and the ensemble's BDI-II DCHR is never reported. At minimum, the authors need to report both, consistently, and explain why one should trust the eRisk cutoffs at all.\n\nThe conceptual premise—that semantic similarity between item choices and posts indicates the user's true score—is plausible but untested against relevance judgments. That's a softer concern; the zero-shot retrieval with adaptive k is a sensible operational choice.\n\nWho is this for? People working on social-media mental health screening and anyone thinking about how to make LLM psychometric assessments interpretable. It also works as a teaching example of how small-N configuration search can inflate results. I'd send it to review, but ask for a preregistered configuration plan or a proper validation split, confidence intervals, and a consistent treatment of BDI-II cutoffs before the SOTA claim is accepted.\n\nBest,\n[You]","headline":"Questionnaire-guided RAG for LLM mental health screening is a genuine idea, but the SOTA claim is not yet supported: too much configuration search on 20 users and the BDI/BDI-II cutoff inconsistency blur the headline numbers.","tokens_in":19064,"tokens_out":2518,"would_cite":true,"duration_ms":22931,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that guiding LLMs through standardized psychological questionnaires with adaptive retrieval lets them match or outperform supervised state-of-the-art depression screening on Reddit benchmarks with no training data.","keywords":["mental health screening","adaptive retrieval-augmented generation","zero-shot LLM classification","BDI-II","social media depression detection","psychological questionnaires","eRisk benchmark","interpretable assessment"],"falsifier":"Build a small evaluation set where clinicians label, for a sample of users and BDI-II items, which posts support which 0-3 score, and measure whether the posts retrieved by embedding similarity to each choice match those labels substantially better than chance. Alternatively, ablate the retrieval: feed each LLM the same number of randomly selected posts instead of the adaptively retrieved ones; if the questionnaire scores do not drop, the retrieval stage is not what carries the result.","tokens_in":18059,"feed_emoji":"🧠","tokens_out":6983,"duration_ms":60249,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models can act as psychological assessors if their predictions are channeled through standardized questionnaires instead of being asked directly for a diagnosis. The authors build a retrieval-augmented pipeline that, for each item of the Beck Depression Inventory-II, retrieves the Reddit posts most semantically similar to the item's four response choices and asks an LLM to score the item from 0 to 3. On the eRisk 2019 and 2020 severity-assessment benchmarks this unsupervised procedure matches or outperforms the best supervised systems, and the same questionnaire-guided design beats directly prompting the LLM to screen for depression. The paper also reports proof-of-concept extensions to self-harm, anorexia, and pathological gambling questionnaires. If true, this would make interpretable, zero-training-data mental health screening from social media feasible at scale.","feed_headline":"Questionnaire-guided LLMs match or beat trained screeners","feed_subtitle":"An adaptive retrieval pipeline lets LLMs score BDI-II items from Reddit posts with no training data.","key_machinery":"The central mechanism is the item-query retrieval loop. Each questionnaire item is turned into four embedding queries, one per response choice, and for each choice the ABIDE algorithm selects an adaptive neighborhood size k* so that exactly the posts whose semantics stay coherent around that choice are retrieved from the user's post history. These retrieved posts are then fed to the LLM in a zero-shot prompt (direct or chain-of-thought) that asks for the item score in 0-3, and the scores are summed into the BDI-II total and mapped to depression severity. The questionnaire itself is the load-bearing intermediate representation: it decomposes the diagnosis into clinically validated items, and the adaptive neighborhood replaces a fixed retrieval threshold with a per-item, per-user choice of how many posts to use.","core_discovery":"On its own terms, the paper's central claim is that structuring LLM assessment through clinically validated questionnaire items, with adaptive retrieval of relevant social media posts for each item, lets the LLM fill out the BDI-II accurately enough to match or beat state-of-the-art supervised models on the eRisk 2019 and 2020 Reddit benchmarks, using no training data at all. The best configurations reach 55.00% depression-category hit rate (DCHR) on eRisk 2019 and 52.86% with a small ensemble on eRisk 2020, and improve item-level closeness (ACR) in both collections. The paper further claims that questionnaire-guided screening outperforms direct prompting of the LLM for depression detection, and that the same pipeline transfers to self-harm, anorexia, and pathological gambling questionnaires, where it achieves the highest F1 score among compared systems on the anorexia and gambling early-detection tasks. The overall assertion is that standardized questionnaires provide the missing structured bridge between unstructured text and clinical categories, making LLM predictions more accurate and more interpretable.","pith_inferences":["A natural extension the authors do not test is validating the item-choice retrieval against human relevance judgments; if the neighborhoods align with clinician-endorsed evidence, the method could support evidence-traceable screening reports.","Since the benchmark ground truth is self-reported BDI-II, the reported gains inherit the instrument's measurement noise; comparing against clinician-administered structured interviews would be a stricter test and might change the ranking.","The framework's reliance on phrase-level choice texts means questionnaires with abstract or situational items may not embed well; testing with such instruments would map the boundary of the approach.","The method's zero-training property makes it attractive for resource-limited screening, but population skew in who posts on social media is a likely confound for generalizing beyond the benchmark."],"forward_implications":["If the central claim holds, supervised classifiers are not required for solid BDI-II severity screening on Reddit-style text; an unsupervised LLM pipeline can match them.","Questionnaire-guided prompting becomes a cheap alternative to direct LLM diagnosis, with item-level traceability for why a score was given.","The same adaptive-retrieval template can be pointed at new clinical instruments without training examples, potentially shortening the path from a new questionnaire to a new screening system.","Retrieval quality is load-bearing: the ablation showing aRAG beats feeding the LLM all posts implies post selection is a major source of the gain, not just LLM size."],"supporting_citations":[{"why":"Supplies the BDI-II instrument and the severity cut-off scores that define the questionnaire's output categories.","marker":"Beck, 1996"},{"why":"Provides the eRisk 2019 collection with ground-truth BDI-II responses and the official benchmark metrics the paper reports.","marker":"Losada et al., 2019"},{"why":"Provides the eRisk 2020 collection with ground-truth BDI-II responses and the baselines used for comparison.","marker":"Losada et al., 2020"},{"why":"Defines the ABIDE algorithm that determines the adaptive neighborhood size k* for each item-query.","marker":"Noia et al., 2024"},{"why":"Introduces the ABIDE-ZS adaptive zero-shot retrieval method that this paper's retrieval strategy builds on.","marker":"Ravenda et al., 2025"},{"why":"Supplies a retrieval-based BDI-II benchmark (Recall and Voting variants) that the paper must match or outperform on eRisk 2020.","marker":"Pérez et al., 2023"},{"why":"Supplies the UNSLC and UNSLE baselines for eRisk 2019 that the paper outperforms on the questionnaire-level metrics.","marker":"Burdisso et al., 2019"},{"why":"Supplies the iLab supervised BERT baseline used for the self-harm detection comparison.","marker":"Martínez-Castaño et al., 2020"},{"why":"Provides the chain-of-thought prompting strategy tested as one of the two prompt variants.","marker":"Wei et al., 2022"},{"why":"Establishes the earlier direction of linking social media post embeddings and question text embeddings to predict questionnaire responses, which this work extends.","marker":"Vu et al., 2020"}],"fun_headline_variants":["LLMs match trained screeners on depression using adaptive RAG","No training data needed: LLMs score BDI-II from Reddit posts","Questionnaire-guided LLMs outperform direct prompting for depression","Adaptive retrieval lets LLMs complete clinical questionnaires from posts","LLMs as psychological assessors: matching SOTA without training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the assumption that semantic similarity between a response choice's text and a user's post is reliable evidence of which symptom level the user would endorse, and that the eRisk self-reported BDI-II answers are trustworthy ground truth; if those two links break, the item scores have no basis.","fun_headline_variants_meta":{"raw":{"variants":["LLMs match trained screeners on depression using adaptive RAG","No training data needed: LLMs score BDI-II from Reddit posts","Questionnaire-guided LLMs outperform direct prompting for depression","Adaptive retrieval lets LLMs complete clinical questionnaires from posts","LLMs as psychological assessors: matching SOTA without training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1446,"prompt_tokens":984,"completion_tokens":462,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":600,"tokens_out":462,"duration_ms":4527,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:38:03.177268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a small evaluation set where clinicians label, for a sample of users and BDI-II items, which posts support which 0-3 score, and measure whether the posts retrieved by embedding similarity to each choice match those labels substantially better than chance. Alternatively, ablate the retrieval: feed each LLM the same number of randomly selected posts instead of the adaptively retrieved ones; if the questionnaire scores do not drop, the retrieval stage is not what carries the result.","supporting_citations":[],"review_version":1}