pith. sign in

arxiv: 2605.23052 · v1 · pith:L5FYAWY6new · submitted 2026-05-21 · 💻 cs.CL · cs.AI

DreamerNLplus: Interpretable Modeling of Mental Health Dynamics from Social Media Timelines using Hybrid Rule-Based and RAG Methods

Maryia Zhyrko , Daisy Monika Lal , Erik van Mulligen , Lifeng Han This is my paper

Pith reviewed 2026-05-25 05:26 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords mental health dynamicssocial media timelinesretrieval-augmented generationtemporal change detectionhybrid modelingpsychological state predictionsequence summarization
0
0 comments X

The pith

RAG-based hybrid modeling ranks first for detecting mental health improvement in timelines.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents DreamerNLplus, a hybrid framework that blends rule-based logic with retrieval-augmented generation to analyze mental health dynamics from social media timelines. The system tackles state prediction, temporal change detection, and sequence-level summarization. Its RAG component for summarization ranks first in identifying patterns of improvement and third in deterioration, indicating it can track recurring psychological shifts. The authors also report mismatches between classification and regression results plus difficulties in modeling transitions and aligning different evaluation metrics.

Core claim

The RAG-based method captures recurrent psychological change patterns across timelines, achieving first place for improvement detection and third place for deterioration detection in the sequence-level task.

What carries the argument

The retrieval-augmented generation pipeline that retrieves timeline segments to produce sequence-level summaries of mental health changes.

Load-bearing premise

Social media timelines contain sufficient and reliable signals of psychological state changes that can be captured by the described hybrid pipeline without substantial domain-specific validation or clinical ground truth.

What would settle it

A direct comparison of the system's predicted changes against independent clinical assessments of the same individuals' mental health over matching time periods.

Figures

Figures reproduced from arXiv: 2605.23052 by Daisy Monika Lal, Erik van Mulligen, Lifeng Han, Maryia Zhyrko.

Figure 1
Figure 1. Figure 1: Overview of the DreamerNLplus system across the CLPsych 2026 shared tasks. Task 1 models psycho [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Prompt2Predict-DeBERTa pipeline including [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Task 1 rule-based pattern matching approach using n-gram collocations to classify post sentences into [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Task 2 few-shot prompting pipeline (Identify [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task 3.1 summary generation methods from DreamerNLplus – rule-based (left) vs OS-LLMs (right). [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overview of the RAG-LLM Signature Mining [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Task 1.1 vs 1.2 Relation. Each point = one [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Targeted data augmentation strategy using [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Data Augmentation Examples (paraphrased to preserve privacy in accordance with shared task guidelines). [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Task 2 Ranking [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Task 3.1 eval on test set - all teams (our filtering: by using the best averaged rank submission from other [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
read the original abstract

We present DreamerNLplus, a hybrid framework for modeling mental health dynamics from social media timelines in the CLPsych 2026 shared task. Our system addresses three tasks: psychological state modeling, temporal change detection, and sequence-level summarization. For Task 1, we combine LLM-based data augmentation, DeBERTa classification, and Random Forest regression for structured state prediction. For Task 2, we use few-shot prompting with a locally deployed Llama 3.1 model to detect Switch and Escalation events using short-term temporal context. For Task 3.1, we explore both a deterministic rule-based summarization pipeline and a few-shot LLM-based approach, ranking \textbf{2nd} officially. Our RAG-based method achieves strong performance in Task 3.2, ranking \textbf{1st} for Improvement and \textbf{3rd} for Deterioration, demonstrating its ability to capture recurrent psychological change patterns across timelines. Our analysis reveals key challenges, including the mismatch between classification and regression performance, the difficulty of modeling temporal transitions, and the disagreement between semantic and similarity-based evaluation metrics. These findings highlight the complexity of modeling mental health dynamics and motivate future work on unified evaluation frameworks. We share our code and prompts at https://github.com/4dpicture/CLPsych2026

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper presents DreamerNLplus, a hybrid framework combining LLM-based data augmentation, DeBERTa classification, Random Forest regression, few-shot prompting with Llama 3.1, deterministic rule-based summarization, and RAG methods for three tasks in the CLPsych 2026 shared task on modeling mental health dynamics from social media timelines. It reports official rankings of 2nd on Task 3.1 and 1st for Improvement / 3rd for Deterioration on Task 3.2, analyzes challenges such as classification-regression mismatch and temporal transition modeling, and releases code and prompts.

Significance. If the reported shared-task rankings hold, the work demonstrates the viability of hybrid rule-based and RAG pipelines for capturing recurrent psychological change patterns in social media data. The explicit release of code and prompts is a clear strength that supports reproducibility and extension by other researchers.

major comments (1)
  1. [Abstract] Abstract: The central performance claims rest on shared-task rankings (1st/3rd on Task 3.2) but the manuscript provides no error bars, baseline comparisons, or detailed ablation results for the RAG component versus the rule-based pipeline; this makes it difficult to isolate the contribution of the hybrid design to the reported rankings.
minor comments (1)
  1. The analysis section notes disagreement between semantic and similarity-based metrics and the difficulty of modeling temporal transitions, but does not provide concrete examples or quantitative breakdowns from the submitted runs to illustrate these challenges.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive assessment of our work and the constructive feedback on the abstract. We address the single major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central performance claims rest on shared-task rankings (1st/3rd on Task 3.2) but the manuscript provides no error bars, baseline comparisons, or detailed ablation results for the RAG component versus the rule-based pipeline; this makes it difficult to isolate the contribution of the hybrid design to the reported rankings.

    Authors: We acknowledge that the abstract, constrained by length, emphasizes the official shared-task rankings without error bars or component-wise ablations. The full manuscript describes the hybrid pipeline and notes challenges such as classification-regression mismatch, but does not include the requested detailed comparisons. In the revised version we will add (i) baseline comparisons against the individual rule-based and RAG pipelines on Task 3.2 and (ii) ablation results that isolate the contribution of each component. For error bars, the shared-task protocol uses a single fixed test set; we will therefore report standard deviation across our internal development-set cross-validation runs to quantify stability. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical shared-task results externally anchored

full rationale

The paper reports system performance on CLPsych 2026 shared tasks using hybrid LLM/RAG/rule-based pipelines. All central claims (rankings of 1st/3rd on Task 3.2, 2nd on Task 3.1) are defined and scored by external task organizers rather than by any internal derivation, fitted parameter, or self-citation chain. No equations, predictions derived from fitted inputs, or load-bearing self-citations appear. The evaluation is externally falsifiable via the shared-task leaderboard and released code/prompts.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is a system description for a shared task and introduces no mathematical derivations, free parameters, axioms, or new postulated entities.

pith-pipeline@v0.9.0 · 5788 in / 1174 out tokens · 17849 ms · 2026-05-25T05:26:15.094366+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages

  1. [1]

    CL4Health WS at LREC2026, Palma, Spain , year=

    Dutch Metaphor Extraction from Cancer Patients' Interviews and Forum Data using LLMs and Human in the Loop , author=. CL4Health WS at LREC2026, Palma, Spain , year=

    work page

  2. [2]

    Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology , month=

    Overview of the CLPsych 2026 Shared Task: Capturing and Characterizing Mental Health Changes through Social Media Timeline Dynamics , author=. Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology , month=

    work page 2026

  3. [3]

    Overview of the CLP sych 2025 Shared Task: Capturing Mental Health Dynamics from Social Media Timelines

    Tseriotou, Talia and Chim, Jenny and Klein, Ayal and Shamir, Aya and Dvir, Guy and Ali, Iqra and Kennedy, Cian and Singh Kohli, Guneet and Hills, Anthony and Zirikly, Ayah and Atzil-Slonim, Dana and Liakata, Maria. Overview of the CLP sych 2025 Shared Task: Capturing Mental Health Dynamics from Social Media Timelines. Proceedings of the 10th Workshop on C...

    work page doi:10.18653/v1/2025.clpsych-1.16 2025

  4. [4]

    2025 , month = oct, publisher =

    Atzil-Slonim, Dana , title =. 2025 , month = oct, publisher =. doi:10.17605/OSF.IO/SJE8C , url =

    work page doi:10.17605/osf.io/sje8c 2025

  5. [5]

    Overview of the CLP sych 2022 Shared Task: Capturing Moments of Change in Longitudinal User Posts

    Tsakalidis, Adam and Chim, Jenny and Bilal, Iman Munire and Zirikly, Ayah and Atzil-Slonim, Dana and Nanni, Federico and Resnik, Philip and Gaur, Manas and Roy, Kaushik and Inkster, Becky and Leintz, Jeff and Liakata, Maria. Overview of the CLP sych 2022 Shared Task: Capturing Moments of Change in Longitudinal User Posts. Proceedings of the Eighth Worksho...

    work page doi:10.18653/v1/2022.clpsych-1.16 2022

  6. [6]

    Practice-Based Evidence in the Psychological Therapies: Toward Policy Implications for Research, Training, and Clinical Guidelines , editor =

    Atzil-Slonim, Dana , title =. Practice-Based Evidence in the Psychological Therapies: Toward Policy Implications for Research, Training, and Clinical Guidelines , editor =. 2026 , url =

    work page 2026

  7. [7]

    Prompt Engineering for Capturing Dynamic Mental Health Self States from Social Media Posts

    Chan, Callum and Khunkhun, Sunveer and Inkpen, Diana and Lossio-Ventura, Juan Antonio. Prompt Engineering for Capturing Dynamic Mental Health Self States from Social Media Posts. Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025). 2025. doi:10.18653/v1/2025.clpsych-1.22

    work page doi:10.18653/v1/2025.clpsych-1.22 2025

  8. [8]

    A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs

    Kermani, Arshia and Perez-Rosas, Veronica and Metsis, Vangelis. A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs. Prompt Engineering vs. RAG. Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025). 2025. doi:10.18653/v1/2025.clpsych-1.14

    work page doi:10.18653/v1/2025.clpsych-1.14 2025

  9. [9]

    Translational Psychiatry , volume=

    Natural language processing for mental health interventions: a systematic review and research framework , author=. Translational Psychiatry , volume=. 2023 , publisher=

    work page 2023

  10. [10]

    Journal of medical Internet research , volume=

    Machine learning and natural language processing in mental health: systematic review , author=. Journal of medical Internet research , volume=. 2021 , publisher=

    work page 2021

  11. [11]

    Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health) , pages=

    LT3: Generating medication prescriptions with conditional transformer , author=. Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health) , pages=

    work page

  12. [12]

    MaLei at MultiClinSUM: Summarisation of Clinical Documents using Perspective-Aware Iterative Self-Prompting with LLMs , author=

    work page

  13. [13]

    Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health) , pages=

    The manchester bees at peranssumm 2025: Iterative self-prompting with claude and o1 for perspective-aware healthcare answer summarisation , author=. Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health) , pages=

    work page 2025

  14. [14]

    arXiv preprint arXiv:2603.01910 , year=

    FLANS at SemEval-2026 Task 7: RAG with Open-Sourced Smaller LLMs for Everyday Knowledge Across Diverse Languages and Cultures , author=. arXiv preprint arXiv:2603.01910 , year=

    work page arXiv 2026

  15. [15]

    The Correlation Between Emotion in Text and Speech Segments is Limited: A Cross-Modal Study

    Lindevelt, David and Verberne, Suzan and Broekens, Joost. The Correlation Between Emotion in Text and Speech Segments is Limited: A Cross-Modal Study. Findings of the A ssociation for C omputational L inguistics: EACL 2026. 2026. doi:10.18653/v1/2026.findings-eacl.136

    work page doi:10.18653/v1/2026.findings-eacl.136 2026

  16. [16]

    Informatica , volume=

    An enhanced aspect-based sentiment analysis model based on roberta for text sentiment analysis , author=. Informatica , volume=

    work page

  17. [17]

    Ieee Access , volume=

    Comparative analysis of deep natural networks and large language models for aspect-based sentiment analysis , author=. Ieee Access , volume=. 2024 , publisher=

    work page 2024

  18. [18]

    Applied Intelligence , volume=

    An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa , author=. Applied Intelligence , volume=. 2021 , publisher=

    work page 2021

  19. [19]

    Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 , pages=

    Aspect-based sentiment analysis of scientific reviews , author=. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 , pages=

    work page 2020

  20. [20]

    Knowledge-based systems , volume=

    Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks , author=. Knowledge-based systems , volume=. 2022 , publisher=

    work page 2022

  21. [21]

    IEEE Transactions on Knowledge and Data Engineering , volume=

    A survey on aspect-based sentiment analysis: Tasks, methods, and challenges , author=. IEEE Transactions on Knowledge and Data Engineering , volume=. 2022 , publisher=

    work page 2022

  22. [22]

    Scientific Reports , volume=

    Triple dimensional psychology knowledge encouraging graph attention networks to exploit aspect-based sentiment analysis , author=. Scientific Reports , volume=. 2025 , publisher=

    work page 2025

  23. [23]

    Overview of the SIGHAN 2024 shared task for C hinese dimensional aspect-based sentiment analysis

    Lee, Lung-Hao and Yu, Liang-Chih and Wang, Suge and Liao, Jian. Overview of the SIGHAN 2024 shared task for C hinese dimensional aspect-based sentiment analysis. Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10). 2024

    work page 2024

  24. [24]

    2026 , eprint=

    DimStance: Multilingual Datasets for Dimensional Stance Analysis , author=. 2026 , eprint=

    work page 2026

  25. [25]

    2026 , eprint=

    DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis , author=. 2026 , eprint=

    work page 2026

  26. [26]

    Electronics , volume=

    A hybrid approach to dimensional aspect-based sentiment analysis using Bert and large language models , author=. Electronics , volume=. 2024 , publisher=

    work page 2024

  27. [27]

    2019 , eprint=

    Optuna: A Next-generation Hyperparameter Optimization Framework , author=. 2019 , eprint=

    work page 2019

  28. [28]

    Proceedings of the 8th International Conference on Weblogs and Social Media (ICWSM) , year =

    VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text , author =. Proceedings of the 8th International Conference on Weblogs and Social Media (ICWSM) , year =

    work page

  29. [29]

    Medication Extraction and Entity Linking using Stacked and Voted Ensembles on LLM s

    Romero, Pablo and Han, Lifeng and Nenadic, Goran. Medication Extraction and Entity Linking using Stacked and Voted Ensembles on LLM s. Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health). 2025. doi:10.18653/v1/2025.cl4health-1.26

    work page doi:10.18653/v1/2025.cl4health-1.26 2025

  30. [30]

    Aspect -- Sentiment Quad Prediction with Distilled Large Language Models

    Ventirozos, Filippos Karolos and Appleby, Peter and Shardlow, Matthew. Aspect -- Sentiment Quad Prediction with Distilled Large Language Models. Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era. 2025

    work page 2025

  31. [31]

    and Shardlow, Matthew

    Ventirozos, Filippos and Appleby, Peter A. and Shardlow, Matthew. Are You Sure You ' re Positive? Consolidating Chain-of-Thought Agents with Uncertainty Quantification for Aspect-Category Sentiment Analysis. Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025). 2025. doi:10.18653/v1/2025.realm-1.22

    work page doi:10.18653/v1/2025.realm-1.22 2025

  32. [32]

    arXiv preprint arXiv:2508.13953 , year=

    ReviewGraph: A Knowledge Graph Embedding Based Framework for Review Rating Prediction with Sentiment Features , author=. arXiv preprint arXiv:2508.13953 , year=

    work page arXiv

  33. [33]

    IEEE Transactions on Affective Computing , volume=

    Issues and challenges of aspect-based sentiment analysis: A comprehensive survey , author=. IEEE Transactions on Affective Computing , volume=. 2020 , publisher=

    work page 2020

  34. [34]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    work page 1972

  35. [35]

    Publications Manual , year = "1983", publisher =

    work page 1983

  36. [36]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    work page doi:10.1145/322234.322243 1981

  37. [37]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

    work page

  38. [38]

    Dan Gusfield , title =. 1997

    work page 1997

  39. [39]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    work page 2015

  40. [40]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

    work page