Pith. sign in

REVIEW 3 major objections 4 minor 21 references

INESC-ID @ eRisk 2025: Exploring Fine-Tuned, Similarity-Based, and Prompt-Based Approaches to Depression Symptom Identification

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper reports that five runs submitted to eRisk 2025 Task 1 achieved the highest official Information Retrieval evaluation scores, with a precision-focused ensemble ('unanimity') best on AP, R-PREC, and P@10, and a maximum-score…

desk verdict Solid shared-task system description with a clear internal comparison, but the 'beat 16 teams' claim is unverifiable from the paper alone and the F1-based development rationale doesn't match the official IR outcomes. read the letter →

arxiv 2506.02924 v1 pith:Q5NOW7DU submitted 2025-06-03 cs.CL cs.IRcs.LG

Diogo A.P. Nunes , Eugénio Ribeiro This is my paper
classification cs.CLcs.IRcs.LG
keywords eRiskdepressionsymptomsfine-tuningsentencesimilaritylargelanguagemodelspromptinginformationretrievalensembles
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that depression symptom identification in social media text can be effectively attacked as a binary relevance classification problem per Beck's Depression Inventory (BDI) symptom, with classification confidence later converted into ranked retrieval. The authors built five ranking runs from fine-tuned foundation models, cosine-similarity baselines, and LLM prompting, and they report that their submissions took the highest official Information Retrieval scores in eRisk 2025 Task 1, beating 16 other teams. The reason to care is practical: a system that finds and ranks sentences expressing specific depressive symptoms could support early detection and monitoring of depression from online writing at scale. The paper also shows that no single method is best for every symptom, which motivates their ensemble strategy.

What carries the argument

The load-bearing machinery is a three-valued regression mapping from the two available binary annotations to a continuous relevance score, which lets one fine-tuned DeBERTa-v3-large model per BDI symptom exploit both majority and unanimity labels. Around that core, the approach uses maximum cosine similarity to symptom-option embeddings (the 'maxcos' filter, also used to shrink the 17.5-million-sentence test set), 5-shot LLM prompting with semantically similar exemplars, and two ensemble rules: taking the maximum score across methods and taking the unanimous intersection of positive predictions ranked by minimum score.

What would settle it

Recompute AP, R-PREC, P@10, and NDCG@1000 for the five runs using the released relevance judgments and check whether the unanimity run is actually top on AP, R-PREC, and P@10 and the max run top on NDCG@1000; a different ordering, or a simple per-symptom regression-score ranking beating both ensembles, would contradict the central claim.

Watch

Extended reading notes

Core claim

The central discovery, on the authors' terms, is that fine-tuning a per-symptom regression model on the mapping majority-label-to-2/3, unanimity-label-to-1, negative-to-0, with up-sampled synthetic sentences, gives the best local F1, but the highest official IR scores come from ensembling complementary methods. The 'max' run ranks each sentence by the maximum score across fine-tuned and similarity runs, while the 'unanimity' run keeps only sentences positively predicted by all base methods plus a 5-shot LLM prompt and ranks them by the minimum score. The paper argues that this precision-conservative ensemble is what wins AP, R-PREC, and P@10, while the 'max' ensemble wins NDCG@1000.

Load-bearing premise

The design assumes that a method's F1 score on a random 20% validation split of the 26,290 labeled training sentences predicts how well that method will rank up to 1,000 relevant sentences per symptom on the 17.5-million-sentence test set under official IR metrics, because the official metrics could not be computed locally.

Editorial extensions

If this is right

  • If the reported official results hold, a precision-first ensemble of fine-tuned, similarity, and LLM prompt signals is a viable recipe for sentence-level symptom retrieval at the scale of millions of sentences.
  • Per-symptom fine-tuned models beat unsupervised similarity and zero-shot prompting on validation F1, so future participants can expect fine-tuning to give the strongest base rankers.
  • Synthetic positive sentences generated by several LLMs improve robustness to label noise and class imbalance, especially in the unanimity annotation setting.
  • Symptom difficulty varies widely, with Guilty Feelings consistently easy and Past Failure, Indecisiveness, and Loss of Interest in Sex consistently hard, so method selection should be symptom-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that the same ensemble recipe may transfer to other fine-grained mental-health retrievals, such as anxiety or PTSD symptom inventories, but the per-symptom fine-tuning cost and the need for labeled training sentences would have to be re-created for each new inventory.
  • The paper does not test whether the maxcos prefilter, which shrinks the test set before ranking, changes which sentences survive; a randomized candidate-sampling ablation could separate prefilter effects from ensemble effects.
  • Because official IR metrics could not be locally implemented, the validation-F1-to-AP proxy is unverified; comparing validation F1 with official AP symptom by symptom would test that proxy without any new data collection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes INESC-ID's submission to eRisk 2025 Task 1, which asks participants to rank up to 1,000 sentences per BDI-II symptom. The authors frame the task as binary classification/regression per symptom, fine-tune DeBERTa-v3-large with the label mapping of Eq. 1, augment training data with synthetic sentences from GPT-4o, Claude Sonnet 3.7, and Qwen2.5-32B, and compare with an unsupervised similarity baseline (maxcos) and k-shot GPT-4o-Mini prompting. They submit five runs, two of which are ensembles, and report in Table 6 that 'unanimity' is best on AP, R-PREC, and P@10 while 'max' is best on NDCG@1000. The paper claims these runs achieved the highest scores in the official evaluation, outperforming submissions from 16 other teams.

Significance. If the comparative claim is confirmed, this is a useful system-description paper for the eRisk shared task: it provides a per-symptom comparison of fine-tuned models, similarity methods, prompting, and ensembles, with a helpful analysis of majority vs. unanimity labels. The per-symptom best-method table (Table 5) and the discussion of synthetic-data up-sampling are valuable. The work is not circular: the main results come from supervised training and an external official test set. However, the manuscript's evidence for its headline claim is incomplete because Table 6 lists only the authors' five runs and cites an unpublished overview, and because the local development metric (validation F1) is not aligned with the official IR metrics. No code is provided, which limits reproducibility. These issues prevent the comparative claim from being independently verified from the paper itself.

major comments (3)
  1. [4.2 / Table 6] The central claim that the submitted runs 'outperformed submissions from 16 other teams' cannot be checked from this manuscript. Table 6 lists only the five INESC-ID runs and refers to the official overview [15] as 'To be published'; no other team's scores are shown, and no leaderboard or comparative table is included. Since the whole evaluation section consists of the authors' own rows, the comparative part of the claim is unsupported by the paper as submitted. Please include the official leaderboard or a table with the other teams' scores for the same metrics, or clearly state that the comparison is based on the organizers' published results with a precise citation.
  2. [3.5 / 4.1 / Tables 4 and 6] The development stage selects methods on validation F1, but the official evaluation uses IR metrics (AP, R-PREC, P@10, NDCG@1000). The reported results show that the F1-based ranking does not explain the official outcome: maxcos has the lowest development F1 (0.775 majority) yet achieves the highest P@10 among non-ensemble runs in both annotation settings (0.757 vs. 0.643 for mix23 in majority; 0.429 vs. 0.371 in unanimity), and aug-best has better development unanimity F1 than mix23 (0.804/0.805 vs. 0.791) but is worse on AP, R-PREC, and NDCG in both settings. The statement in Section 3.5 that the official IR metrics 'cannot be locally implemented due to under-specification' is not substantiated; with binary relevance labels, these metrics are standardly computable on the validation split. The paper should either compute the official metrics on the validation split, or explicitly analyze why the local proxy diverges from the official metrics. As written, the F1-based development narrative does not support the official outcome.
  3. [4.2 (run descriptions)] The description of the 'aug-best' run is ambiguous. The text says it selects the approach that 'performed best in the development stage per symptom (see Tab. 5)', but Table 5 has two columns ('Best in majority' and 'Best in unanimity') and the paper does not state which of the two settings was used to choose the per-symptom model for the official submission. Since the two columns often name different approaches (e.g., Loss of Pleasure, Changes in Sleeping Pattern, Indecisiveness), the selection rule must be specified for the run to be reproducible.
minor comments (4)
  1. [Declaration on Generative AI] The declaration states that the authors 'have not employed any Generative AI tools', but Sections 3.2 and 3.4 describe prompting GPT-4o, Claude Sonnet 3.7, Qwen2.5-32B, and GPT-4o-Mini to generate synthetic training data and to classify sentences. This is an internal contradiction. If the declaration is meant to refer only to the writing of the paper, that should be stated explicitly; otherwise the declaration must be corrected.
  2. [3.2] The synthetic-data generation procedure is under-specified: for each symptom, 100 sentences are requested from each of three LLMs, but the prompt template, the filtering/quality-control steps, and the handling of duplicates or ungrammatical outputs are not described. Since the authors later attribute part of aug-best's behavior to the 'obvious' nature of the synthetic sentences, a more precise description would strengthen reproducibility.
  3. [General] No code or configuration files are provided. For a paper whose contribution is primarily an empirical system comparison, releasing the fine-tuning scripts, the exact thresholds, and the prompt templates would substantially improve reproducibility. This is not a correctness issue but is relevant for a system-description paper.
  4. [Eq. (1)] The mapping in Eq. (1) is an ad-hoc choice (0, 2/3, 1), and no sensitivity analysis is reported for the specific values. This is not a fatal flaw, but a brief robustness check or a sentence noting that the validation results were stable under reasonable variations would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: development selects on local F1 while official IR scores come from an external organizer evaluation; the top-team comparison is unverifiable from this paper but not circular.

full rationale

The paper's derivation chain is a standard supervised pipeline: split 26,290 annotated sentences into stratified train/val sets, fine-tune deberta-v3-large per symptom, optionally augment with synthetic examples from three LLMs, run a cosine-similarity baseline (maxcos), and use GPT-4o-Mini prompting, finally submitting ranked test-set outputs for official IR evaluation. No equation defines an output in terms of the target metric. Section 3.5 explicitly states that official IR metrics 'cannot be locally implemented' and that the authors 'evaluated our approaches under classical classification metrics, namely F1', so the local F1 choice is an acknowledged proxy, not a hidden definition of the official result. The official scores in Table 6 are attributed to the organizers' separate overview [15], which is marked 'To be published'; this makes the comparative claim 'We were the best performing team for all evaluation metrics' externally unverifiable from the paper, but it is empirical verification, not circular reasoning. Similarly, the contradictory 'Declaration on Generative AI' stating 'The author(s) have not employed any Generative AI tools' while Sections 3.2 and 3.4 describe using GPT-4o, Claude Sonnet 3.7, Qwen2.5-32B, and GPT-4o-Mini is an integrity/inconsistency issue, not a circularity. No self-citation is load-bearing, and no fitted parameter is renamed as a prediction. The paper is self-contained in its methodology and its central claim depends on an external benchmark rather than on its own assumptions.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical claim is supported by supervised training and an external official evaluation, but several modeling choices are made by hand and the local validation metric differs from the official metric.

free parameters (3)
  • unanimity-weight mapping value = 2/3
    Eq. 1 assigns 2/3 to majority-positive, unanimity-negative sentences; the value is chosen by hand to encode label strength, with no sensitivity analysis.
  • maxcos threshold multiplier = 2 standard deviations above the mean
    Section 3.3; similarity scores are binarized using mean plus two standard deviations, an arbitrary heuristic not tuned or justified.
  • synthetic examples per LLM per symptom = 100
    Section 3.2; 100 positive sentences generated by each of GPT-4o, Claude Sonnet 3.7, and Qwen2.5-32B per symptom; the volume is chosen without ablation, and the authors note the augmentation effect was small.
assumptions (3)
  • domain assumption The unanimous annotation is a stronger relevance signal than majority-only annotation, and both are accurate enough to train a regression target.
    Eq. 1 maps majority-only positives to 2/3 and unanimous positives to 1; if this ordering is wrong, the training signal is distorted.
  • domain assumption Validation F1 on a random split of the official labeled training data is a reliable proxy for official IR metrics on the test set.
    Section 3.5 and Section 4.1; the authors could not implement official IR metrics locally and selected models by F1, assuming it correlates with AP, R-PREC, and NDCG.
  • domain assumption LLM-generated synthetic sentences are representative of real relevant social media sentences for each BDI symptom.
    Section 3.2; synthetic positives are added to mitigate class imbalance, and the authors note they were 'fairly obvious', which may bias training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of INESC-ID @ eRisk 2025: Exploring Fine-Tuned, Similarity-Based, and Prompt-Based Approaches to Depression Symptom Identification." pith.science (2026). https://pith.science/paper/Q5NOW7DU

@misc{pith2026250602924,
  author       = {Pith},
  title        = {Pith review of: INESC-ID @ eRisk 2025: Exploring Fine-Tuned, Similarity-Based, and Prompt-Based Approaches to Depression Symptom Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5NOW7DU}},
  note         = {Machine review of arXiv:2506.02924}
}
read the original abstract

In this work, we describe our team's approach to eRisk's 2025 Task 1: Search for Symptoms of Depression. Given a set of sentences and the Beck's Depression Inventory - II (BDI) questionnaire, participants were tasked with submitting up to 1,000 sentences per depression symptom in the BDI, sorted by relevance. Participant submissions were evaluated according to standard Information Retrieval (IR) metrics, including Average Precision (AP) and R-Precision (R-PREC). The provided training data, however, consisted of sentences labeled as to whether a given sentence was relevant or not w.r.t. one of BDI's symptoms. Due to this labeling limitation, we framed our development as a binary classification task for each BDI symptom, and evaluated accordingly. To that end, we split the available labeled data into training and validation sets, and explored foundation model fine-tuning, sentence similarity, Large Language Model (LLM) prompting, and ensemble techniques. The validation results revealed that fine-tuning foundation models yielded the best performance, particularly when enhanced with synthetic data to mitigate class imbalance. We also observed that the optimal approach varied by symptom. Based on these insights, we devised five independent test runs, two of which used ensemble methods. These runs achieved the highest scores in the official IR evaluation, outperforming submissions from 16 other teams.

Figures

Figures reproduced from arXiv: 2506.02924 by the authors.

Figure 1
Figure 1. Distribution of BDI symptom 𝐹1 performance across the multiple approaches, for both majority and unanimity labeling settings. Sorted by descending median value in the majority setting. unanimity evaluation settings. This shows that there was not a single best methodological approach for the detection of all BDI symptoms. However, as already suggested in Tab. 4, the foundation model fine-tuning approaches were by far… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 13 canonical work pages

  1. [15]

    Parapar, A

    J. Parapar, A. Perez, X. Wang, F. Crestani, Overview of eRisk 2025: Early Risk Prediction on the Internet (Extended Overview), in: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF), 2025 (To be published)

    work page 2025

  2. [1]

    Prince, V

    M. Prince, V. Patel, S. Saxena, M. Maj, J. Maselko, M. R. Phillips, A. Rahman, No Health Without Mental Health, The Lancet 370 (2007) 859–877. doi:10.1016/s0140-6736(07)61238-0

    work page doi:10.1016/s0140-6736(07)61238-0 2007

  3. [2]

    E. S. Paykel, Basic Concepts of Depression, Dialogues in Clinical Neuroscience 10 (2008) 279–289. doi:10.31887/dcns.2008.10.3/espaykel

    work page doi:10.31887/dcns.2008.10.3/espaykel 2008

  4. [3]

    S. M. Gold, O. Köhler-Forsberg, R. Moss-Morris, A. Mehnert, J. J. Miranda, M. Bullinger, A. Steptoe, M. A. Whooley, C. Otte, Comorbid Depression in Medical Diseases, Nature Reviews Disease Primers 6 (2020) 69. doi:10.1038/s41572-020-0211-z

    work page doi:10.1038/s41572-020-0211-z 2020

  5. [4]

    Yates, A

    A. Yates, A. Cohan, N. Goharian, Depression and Self-Harm Risk Assessment in Online Forums, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), ACL, 2017, pp. 2968–2978. doi:10.18653/v1/D17-1322

    work page doi:10.18653/v1/d17-1322 2017

  6. [5]

    Cohan, B

    A. Cohan, B. Desmet, A. Yates, L. Soldaini, S. Macavaney, N. Goharian, SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions, in: Proceedings of the International Conference on Computational Linguistics (COLING), 2018, pp. 1485–1497. URL: https://aclanthology.org/C18-1126/

    work page 2018

  7. [6]

    O’Dea, T

    B. O’Dea, T. W. Boonstra, M. E. Larsen, T. Nguyen, S. Venkatesh, H. Christensen, The Relationship Between Linguistic Expression in Blog Content and Symptoms of Depression, Anxiety, and Suicidal Thoughts: A Longitudinal Study, Plos One 16 (2021) e0251787. doi: 10.1371/journal.pone. 0251787

    doi:10.1371/journal.pone 2021

  8. [7]

    N. H. Yahya, H. Abdul Rahim, Linguistic Markers of Depression: Insights from English-Language Tweets Before and During the COVID-19 Pandemic, Language and Health 1 (2023) 36–50. doi: 10. 1016/j.laheal.2023.10.001

    work page 2023

Show all 21 references
  1. [8]

    R. N. Trifu, B. Nemes, , D. C. Herta, C. Bodea-Hategan, D. A. Talas, , H. Coman, Linguistic Markers for Major Depressive Disorder: A Cross-Sectional Study using an Automated Procedure, Frontiers in Psychology 15 (2024) 1355734. doi:10.3389/fpsyg.2024.1355734

    2024

  2. [9]

    Parapar, Javier and Martín-Rodilla, Patricia and Losada, David E and Crestani, Fabio, Overview of eRisk 2023: Early Risk Prediction on the Internet, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: International Conference of the CLEF Association, Spr...

    2023 doi

  3. [10]

    D. E. Losada, F. Crestani, J. Parapar, eRISK 2017: CLEF Lab on Early Risk Prediction on the Internet: Experimental Foundations, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: International Conference of the CLEF Association, 2017, pp. 346–360. doi: ...

    2017

  4. [11]

    Recharla, P

    N. Recharla, P. Bolimera, Y. Gupta, A. K. Madasamy, Exploring Depression Symptoms through Similarity Methods in Social Media Posts, in: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF), 2023, pp. 763–772. URL: https://ceur-ws.org/Vol-3497/paper-065.pdf

    2023

  5. [12]

    B. H. Ang, S. D. Gollapalli, S.-K. Ng, NUS-IDS@eRisk2024: Ranking Sentences for Depression Symptoms Using Early Maladaptive Schemas and Ensembles, in: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF), 2024, pp. 9–12. URL: https://ceur-ws.org/Vol-3740/ pa...

    2024

  6. [13]

    Tausczik, J

    Y. Tausczik, J. Pennebaker, The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods, Journal of Language and Social Psychology 29 (2009) 24–54. doi: 10.1177/ 0261927X09351676

    2009

  7. [14]

    Parapar, P

    J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, Overview of eRisk 2024: Early Risk Prediction on the Internet, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: International Conference of the CLEF Association, Part II, 2024, pp. 73–92. doi:1...

    2024 doi

  8. [16]

    Mikolov, K

    T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient Estimation of Word Representations in Vector Space, Computing Research Repository arXiv:1301.3781 (2013). doi: 10.48550/arXiv.1301. 3781

    2013 doi arXiv

  9. [17]

    Reimers, I

    N. Reimers, I. Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for ...

    2019 doi

  10. [18]

    doi:10.48550/arXiv.2303.08774

    OpenAI, GPT-4 Technical Report, Computing Research Repository arXiv:2303.08774 (2024). doi:10.48550/arXiv.2303.08774

    2024 doi arXiv

  11. [19]

    P. He, J. Gao, W. Chen, DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing, Computing Research Repository arXiv:2111.09543 (2021). doi:10.48550/arXiv.2111.09543

    2021 doi arXiv

  12. [20]

    doi:10.48550/arXiv.2412.15115

    Qwen Team, Qwen2.5 Technical Report, Computing Research Repository arXiv:2412.15115 (2024). doi:10.48550/arXiv.2412.15115

    2024 doi arXiv

  13. [21]

    doi: 10.48550/arXiv.2501

    DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, Computing Research Repository arXiv:2501.12948 (2025). doi: 10.48550/arXiv.2501. 12948

    2025 doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.