Pith. sign in

REVIEW 3 major objections 6 minor 80 references

Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ChatGPT-generated therapy dialogues, expert-checked, match real sessions in quality.

desk verdict A genuinely useful dataset release and annotation scheme, but the paper's central claim that improved classification performance verifies synthetic dialogue quality is not supported by the evidence as presented. read the letter →

arxiv 2412.12981 v1 pith:VB5PAMDJ submitted 2024-12-17 cs.CL

Vivek Kumar , Eirini Ntoutsi , Pushpraj Singh Rajawat , Giacomo Medda , Diego Reforgiato Recupero This is my paper · ORCID
classification cs.CL
keywords motivationalinterviewingLLMdataaugmentationmentalhealthNLPexpertannotationMISCin-contextlearningtextclassificationbiasmitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that synthetic motivational interviewing (MI) dialogues generated by ChatGPT and then checked by a domain expert are good enough to stand in for real therapy transcripts when training machine-learning classifiers. The authors introduce IC-AnnoMI, a dataset of 97 such dialogues built from an existing expert-annotated counselling corpus, with labels for high- and low-quality therapy. They argue that augmenting the original training data with these generated dialogues improves utterance-level classification of therapy quality and reduces the bias that comes from scarce, imbalanced data. If true, this would give the mental-health NLP community a practical route to more training data in a sensitive, low-resource domain without relying on unsupervised LLM output.

What carries the argument

The engine of the method is progressive in-context prompting: the authors prompt ChatGPT variants to rewrite whole MI sessions at once, then manually inspect the output and refine the prompt in a feedback loop until hallucinations, parroting, and off-topic text are eliminated. Each generated dialogue is then annotated under a two-stage scheme derived from the Motivational Interviewing Skills Code: a psychological stage scoring empathy, non-judgmental attitude, competence, ethical conduct, and reflectiveness on a five-point scale, and a linguistic stage checking context, text enrichment, MI enhancement, and language preservation as binary flags. The resulting per-dialogue labels feed an utterance-level binary classification task that separates high- from low-quality therapy, which is the experimental setup used to evaluate both the dataset and the model's understanding of the domain.

What would settle it

Re-annotate the 97 generated dialogues with multiple MISC-trained experts and check inter-rater agreement, or train on the synthetic dialogues alone and evaluate on held-out real sessions: if expert labels diverge sharply or synthetic-only training performs much worse than training on real dialogues, the claim that synthetic quality matches the original fails.

Watch

Extended reading notes

Core claim

The central claim is that IC-AnnoMI's 97 expert-annotated, ChatGPT-generated MI dialogues are comparable in therapeutic quality to the original AnnoMI sessions, so that adding them to the training set improves classification and mitigates inherent biases. The paper supports this with two kinds of evidence: transformer-based classifiers show higher balanced accuracy on the augmented dataset (best 0.59, up from 0.55), and expert MISC-based annotation gives the generated dialogues a mean psychological score of 3.31 out of 4, with context preserved in 95.88% of them. From this the authors conclude that the quality of the synthetic dialogues is 'in line with' the original ones and that LLM-generated, expert-supervised data can be used responsibly in supervised mental-health applications.

Load-bearing premise

The generated dialogues are assumed to have kept the therapeutic quality of the real sessions they were rewritten from, based on a single expert's annotations; if the rewriting quietly changed the therapy quality, the training labels are unreliable and the reported gains would not prove quality parity.

Editorial extensions

If this is right

  • The IC-AnnoMI dataset can serve as a ready-made training resource for models that classify therapy dialogue quality, lowering the entry barrier for mental-health NLP research.
  • LLM-generated, expert-annotated dialogues can increase the sample size of a counselling corpus without requiring new recordings of real therapy sessions, which are expensive and privacy-sensitive to obtain.
  • The balanced-accuracy gains reported for transformer models indicate that augmentation can mitigate the class-imbalance and sampling biases that plague scarce clinical datasets.
  • The paper's prompting pipeline shows a human-in-the-loop recipe for generating domain-plausible synthetic text: generate, inspect for hallucinations and parroting, refine the prompt, then have an expert annotate the final output.
  • The established baselines give future work a reference point for comparing other LLMs, such as LLaMA or Mistral, on the same therapy-quality classification task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the quality-parity conclusion rests on labels that are not independently verified by the reported annotation; a multi-annotator inter-rater reliability study on the 97 generated dialogues would settle whether therapeutic quality was truly preserved.
  • A testable extension suggested by the method: apply the same progressive-prompting plus expert-annotation pipeline to other structured clinical dialogues with established coding manuals, such as cognitive-behavioral therapy, to see whether the quality-parity result transfers beyond MI.
  • The paper leaves implicit that the observed bias mitigation is measured only through balanced accuracy; direct bias metrics, such as performance differences across client demographics or topics, would make the bias claim testable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces IC-AnnoMI, a dataset of 97 expert-annotated ChatGPT-generated motivational interviewing (MI) dialogues built on the AnnoMI corpus, with annotations covering psychological (MIsyn psych) and linguistic (MIsyn linguist) dimensions following the Motivational Interviewing Skills Code (MISC). The authors design a binary high/low quality classification task at the utterance level, evaluate classical machine learning and transformer models with and without augmentation, and report small balanced accuracy improvements when training on the augmented data. They interpret these improvements as verifying that the quality of the synthetic dialogues (MIsyn) is in line with the original dialogues (MIorg), and conclude that LLM-based augmentation with expert oversight can mitigate data scarcity and bias in mental health NLP.

Significance. If the quality-equivalence claim is established, the dataset and augmentation procedure would be a practical contribution to a low-resource domain, offering a template for expert-annotated LLM-generated clinical dialogue data. The paper ships a public dataset and code, uses a human-in-the-loop progressive prompting strategy, and attempts a MISC-grounded annotation scheme spanning both psychological and linguistic aspects. These are concrete strengths that support reproducibility. However, the central verification claim currently rests on an unverified label-inheritance assumption and on classification gains that lack statistical support, so the resource's value as a reliable training benchmark is not yet demonstrated.

major comments (3)
  1. [Section 5, Table 2] The binary high/low quality labels used for training on MIsyn are not explicitly derived from the MISC annotation. The text states that the 97 MIorg dialogues 'undergo an augmentation process followed by expert annotation,' but it never specifies whether the MIsyn binary labels were re-derived from the MIsyn psych and MIsyn linguist scores or simply inherited from the source MIorg dialogues. This is load-bearing: if ChatGPT's rewriting systematically improves low-quality source dialogues into fluent text, then low-quality MIsyn dialogues would carry incorrect labels, introducing noise that undermines both the augmentation results and the claim that MIsyn quality matches MIorg. The paper must clarify the label assignment procedure and provide evidence (e.g., a comparison of source and synthetic MISC scores, or a re-labeling protocol) that the quality class is preserved. The reported average MIsyn psych score of 3.31 on a 0–4 scale, which indicates generally high quality, makes the inheritance assumption particularly questionable given that the training set is intended to contain a mix of high- and low-quality dialogues.
  2. [Section 5, Table 2] The statement that 'the observed improved performance in employed LMs verifies that the quality of MIsyn is in line with MIorg' is stronger than the evidence supports. The balanced accuracy gains for transformer models are modest—for example, BERT changes from 0.54 to 0.56, and DistilBERT from 0.55 to 0.59—and no confidence intervals, significance tests, or repeated-run variance are reported. Given that even the best balanced accuracy (0.59) is close to chance, the observed differences could plausibly arise from random variation. The authors should report statistical significance tests (e.g., McNemar's test or bootstrap confidence intervals) and ideally multiple random seeds, and they should temper the verification claim accordingly. A finding of 'no statistically significant degradation' would be more defensible than 'quality verified.'
  3. [Section 3.2 and Ethics statement] The psychological annotation is described as performed by a single expert from a 'gold-standard institute,' but the paper provides no inter-annotator agreement statistics, no number of annotators, and no description of how ambiguous cases were resolved. Since MISC-based judgments of empathy, reflectiveness, and competence are subjective, single-expert annotation is a reliability concern that directly affects the credibility of the MIsyn psych and MIsyn linguist scores used to support the quality claim. The authors should report annotator count, expertise, and agreement measures (e.g., Cohen's kappa) if multiple annotators were involved, or explicitly discuss the limitation. In addition, the evaluation is partially self-referential: the same development process selected the test set from AnnoMI, iteratively refined prompts until outputs matched MIorg, and used expert judgment to certify those outputs. While the test set is held out, the prompt refinement loop could encode knowledge of the source distribution, and the paper should discuss this risk and what safeguards were taken.
minor comments (6)
  1. [Table 2 caption] The abbreviations 'N-Aug' and 'Aug' are used in the table but are not defined in the caption; define them as 'non-augmented' and 'augmented.'
  2. [Section 5 bullet] The sentence 'M Ipsych has received an average score of 3.31 for the 97 M Isyn.' should specify that the scale is 0–4 and that the score is averaged over the five psychological attributes and then over the dialogues; it would also help to state explicitly that higher scores mean better quality.
  3. [Section 5 bullet] The percentages '95.88%', '83.51%', '88,66%' and '88,66%' use a comma as the decimal separator in the latter two cases; use a period consistently (88.66%).
  4. [Appendix B, Eq. (2)] The formula for balanced accuracy appears non-standard; the usual definition is (TPR + TNR)/2. Please verify that Eq. (2) is correct and not a misformatted expression.
  5. [Throughout] Several LaTeX artifacts produce odd spacing in terms such as 'T herapistutt.', 'Client utt.', and 'M Ipsych'; please ensure the final PDF renders these as 'Therapist utt.', 'Client utt.', and 'MI psych' without extraneous spaces.
  6. [Abstract] The abstract ends with 'IC-AnnoM' instead of 'IC-AnnoMI'; correct the truncation.

Circularity Check

2 steps flagged · score 6.0 of 10

MIsyn quality labels are inherited from AnnoMI source dialogues, so the classification gains used to 'verify' quality equivalence reduce partly to construction.

  1. self definitional [Section 3.3 (Dataset creation) and Section 4.1 (Problem statement)]
    "To create IC-AnnoMI dataset, the 97 M Iorg. of the training set undergo an augmentation process followed by expert annotation using our developed MISC coding scheme. The annotation process overall results in 97 expert-annotated augmented MI dialogues (M Isyn.), containing 4,856 T herapistutt. and 4,792 Client utt. having a mix of high and low-quality MI dialogues."

    The binary MIquality label used as the classification target for the generated MIsyn dialogues is never re-derived from the reported MISC annotation. The described annotation scheme produces a 0-4 MIsyn psych score and binary linguistic components, not an explicit high/low binary quality label. Since MIsyn is generated from MIorg and the dataset is said to contain a mix of high and low quality, the binary labels are, by construction, the AnnoMI labels of the source dialogues. The classification task in Section 4.1 then treats those inherited labels as ground truth for MIsyn utterances, so the target variable for the synthetic data is the same variable used to define its quality.

  2. fitted input called prediction [Section 3.1 (Augmentation) and Section 5 (Result and discussion)]
    "This process continues until the prompt output quality is comparable with M Iorg.. ... The observed improved performance in employed LMs verifies that the quality of M Isyn. is in line with M Iorg.."

    The generation loop is explicitly tuned until MIsyn is 'comparable with MIorg', and the binary labels of MIsyn are the inherited MIorg labels. Adding MIsyn to training and measuring accuracy on a held-out MIorg test set therefore measures how well near-copies of MIorg with copied labels help classify MIorg. The 'quality in line' conclusion is presented as a verified prediction, but it is the expected consequence of the prompt-refinement loop plus label inheritance, not an independent confirmation of MIsyn quality. The reported expert MIsyn psych mean of 3.31/4 is not used to re-assign or check the binary labels, so the verification loop closes on the source labels rather than on an external quality measurement.

full rationale

The paper is a dataset-contribution work, and much of its pipeline is independently grounded: the base AnnoMI data is a public expert-annotated dataset, the test set is a held-out set of 36 MIorg dialogues, and the expert MISC scores provide some evidence about the generated dialogues. However, the central claim that 'the observed improved performance in employed LMs verifies that the quality of MIsyn is in line with MIorg' is partially circular. The binary high/low labels used as training targets for the 97 MIsyn dialogues are never explicitly re-established from the expert annotation; the only described source of such binary labels is AnnoMI, from which the synthetic dialogues were generated. Combined with the progressive prompting loop that continues 'until the prompt output quality is comparable with MIorg', the classification experiment largely measures how well MIorg-like paraphrases with MIorg labels help classify MIorg. This does not amount to an independent verification of MIsyn quality. The held-out test set and expert scores prevent the paper from being fully circular, but the load-bearing quality-equivalence conclusion reduces in part to construction, warranting a score of 6 rather than a higher one. The authors' self-citations to their own AnnoMI work are not the basis of this finding; AnnoMI is a publicly available dataset with external labels.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces a dataset and annotation scheme, not new physical or theoretical entities. Its central claims rest on the assumed validity of AnnoMI labels, MISC coding, and the fidelity of ChatGPT outputs.

assumptions (3)
  • domain assumption AnnoMI's expert-assigned high/low quality labels are correct and transfer to the generated dialogues that inherit them.
    Section 3.3 treats the 97 training dialogues as bases for augmentation and the generated dialogues inherit the quality mix without explicit label verification.
  • domain assumption The MISC 2.1 coding scheme, as applied by a single psychology expert, is a valid gold standard for assessing MI dialogue quality.
    Section 3.2 defines the annotation scheme but provides no inter-annotator agreement or validation against an external benchmark.
  • domain assumption ChatGPT-generated dialogues, after progressive prompt refinement and human filtering, preserve the therapeutic content and quality of the original AnnoMI sessions.
    Section 3.1 assumes that manual evaluation and prompt tuning yield outputs 'comparable with MIorg' before expert annotation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health." pith.science (2026). https://pith.science/paper/VB5PAMDJ

@misc{pith2026241212981,
  author       = {Pith},
  title        = {Pith review of: Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VB5PAMDJ}},
  note         = {Machine review of arXiv:2412.12981}
}
read the original abstract

Large language models (LLMs) have shown promising capabilities in healthcare analysis but face several challenges like hallucinations, parroting, and bias manifestation. These challenges are exacerbated in complex, sensitive, and low-resource domains. Therefore, in this work we introduce IC-AnnoMI, an expert-annotated motivational interviewing (MI) dataset built upon AnnoMI by generating in-context conversational dialogues leveraging LLMs, particularly ChatGPT. IC-AnnoMI employs targeted prompts accurately engineered through cues and tailored information, taking into account therapy style (empathy, reflection), contextual relevance, and false semantic change. Subsequently, the dialogues are annotated by experts, strictly adhering to the Motivational Interviewing Skills Code (MISC), focusing on both the psychological and linguistic dimensions of MI dialogues. We comprehensively evaluate the IC-AnnoMI dataset and ChatGPT's emotional reasoning ability and understanding of domain intricacies by modeling novel classification tasks employing several classical machine learning and current state-of-the-art transformer approaches. Finally, we discuss the effects of progressive prompting strategies and the impact of augmented data in mitigating the biases manifested in IC-AnnoM. Our contributions provide the MI community with not only a comprehensive dataset but also valuable insights for using LLMs in empathetic text generation for conversational therapy in supervised settings.

Figures

Figures reproduced from arXiv: 2412.12981 by the authors.

Figure 1
Figure 1. Development of the IC-AnnoMI dataset. quality and predefined output format are met. In the first step, a prompt template is developed based on MI dialogues’ context, plausibility, and quality for required outputs. Then, the generated output is manually evaluated for inconsistencies, and any deviation from the predefined output is used to tune the prompt further progressively. This pro￾cess continues until the prompt… view at source ↗
Figure 2
Figure 2. Progressive prompt refining. 3.2 MISC annotation The annotation scheme is developed and executed by an expert from gold-standard institute in psy￾chology by strictly adhering to the MISC 2.14 scheme. The developed annotation scheme is a combination of a two-stage annotation process. The first stage of annotation (MIpsych) covers the psychological dimension of the generated MI dia￾logues. The second stage (MIlinguist… view at source ↗
Figure 3
Figure 3. The distribution of client utterances in training [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The distribution of therapist utterances in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The confusion matrix of CML approaches for non-augmented and augmented experimental datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The confusion matrix of BERT model-based approaches for non-augmented and augmented experimental [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: The distribution of therapist utterances in [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 40 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Katherine Abramski, Salvatore Citraro, Luigi Lombardi, Giulio Rossetti, and Massimo Stella. 2023. https://doi.org/10.3390/bdcc7030124 Cognitive network science reveals bias in gpt-3, gpt-3.5 turbo, and gpt-4 mirroring math anxiety in high-school students . Big Data and Cognitive Computing, 7(3)

    work page doi:10.3390/bdcc7030124 2023

  4. [4]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \'e rouane Debbah, \'E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867

    arXiv 2023

  5. [5]

    Dion Alperstein and Louise Sharpe. 2016. https://doi.org/https://doi.org/10.1016/j.jpain.2015.10.021 The efficacy of motivational interviewing in adults with chronic pain: A meta-analysis and systematic review . The Journal of Pain, 17(4):393--403

    work page doi:10.1016/j.jpain.2015.10.021 2016

  6. [6]

    Timothy R Apodaca, Brian Borsari, Kristina M Jackson, Molly Magill, Richard Longabaugh, Nadine R Mastroleo, and Nancy P Barnett. 2014. Sustain talk predicts poorer outcomes among mandated college student drinkers receiving a brief motivational intervention. Psychology of Addictive Behaviors, 28(3):631

    work page 2014

  7. [7]

    Nicklaus Badyal, Derek Jacoby, and Yvonne Coady. 2023. https://doi.org/10.1109/UEMCON59035.2023.10316060 Intentional biases in LLM responses . In 14th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference, UEMCON 2023, New York, NY, USA, October 12-14, 2023 , pages 502--506. IEEE

    arXiv 2023

  8. [8]

    Simone Balloccu, Ehud Reiter, Karen Jia-Hui Li, Rafael Sargsyan, Vivek Kumar, Diego Reforgiato, Daniele Riboni, and Ondrej Dusek. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.674 Ask the experts: sourcing a high-quality nutrition counseling dataset through human- AI collaboration . In Findings of the Association for Computational Linguistics: EMN...

    doi:10.18653/v1/2024.findings-emnlp.674 2024

Show all 80 references
  1. [9]

    Moyers, Steve Sussman, Caitlin Smith, Louise A

    Elizabeth Barnett, Theresa B. Moyers, Steve Sussman, Caitlin Smith, Louise A. Rohrbach, Ping Sun, and Donna Spruijt-Metz. 2014. https://doi.org/https://doi.org/10.1016/j.jsat.2013.11.004 From counselor skill to decreased marijuana use: Does change talk matter? Journal of Subst...

    2014 doi

  2. [10]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. https://doi.org/10.1145/3442188.3445922 On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Tra...

    2021

  3. [11]

    Laura Ana Maria Bostan, Evgeny Kim, and Roman Klinger. 2020. https://aclanthology.org/2020.lrec-1.194 G ood N ews E veryone: A corpus of news headlines annotated with emotions, semantic roles, and reader perception . In Proceedings of the Twelfth Language Resources and Evaluat...

    2020

  4. [12]

    Laura-Ana-Maria Bostan and Roman Klinger. 2018. https://aclanthology.org/C18-1179 An analysis of annotated corpora for emotion classification in text . In Proceedings of the 27th International Conference on Computational Linguistics, pages 2104--2119, Santa Fe, New Mexico, USA...

    2018

  5. [13]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...

    2020

  6. [14]

    Sven Buechel and Udo Hahn. 2017. https://doi.org/10.18653/v1/W17-0801 Readers vs. writers vs. texts: Coping with different perspectives of text understanding in emotion annotation . In Proceedings of the 11th Linguistic Annotation Workshop, pages 1--12, Valencia, Spain. Associ...

    2017 doi

  7. [15]

    Xunxin Cai, Meng Xiao, Zhiyuan Ning, and Yuanchun Zhou. 2023. https://doi.org/10.1109/ICDM58522.2023.00107 Resolving the imbalance issue in hierarchical disciplinary topic inference via llm-based data augmentation . In IEEE International Conference on Data Mining, ICDM 2023, S...

    2023

  8. [16]

    Delwyn Catley, Kari Jo Harris, Kathy Goggin, Kimber Richter, Karen Williams, Christi Patten, Ken Resnicow, Edward Ellerbeck, Andrea Bradley-Ewing, Domonique Malomo, et al. 2012. Motivational interviewing for encouraging quit attempts among unmotivated smokers: study protocol o...

    2012

  9. [17]

    Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zornitsa Kozareva. 2022. https://doi.org/10.18653/v1/2022.naacl-main.260 Improving in-context few-shot learning via self-supervised training . In Proceedings of the 2022 Conference o...

    2022 doi

  10. [18]

    Arijit Ghosh Chowdhury and Aman Chadha. 2024. https://aclanthology.org/2024.eacl-srw.20 Generative data augmentation using llms improves distributional robustness in question answering . In Proceedings of the 18th Conference of the European Chapter of the Association for Compu...

    2024

  11. [19]

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. https://doi.org/10.18653/v1/2020.acl-main.372 G o E motions: A dataset of fine-grained emotions . In Proceedings of the 58th Annual Meeting of the Association for Computati...

    2020 doi

  12. [20]

    Dorottya Demszky, Diyi Yang, David S Yeager, Christopher J Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, et al. 2023. Using large language models in psychology. Nature Reviews Psychology, 2(11):688--701

    2023

  13. [21]

    Danilo Dess \` , Rim Helaoui, Vivek Kumar, Diego Reforgiato Recupero, and Daniele Riboni. 2020. http://ceur-ws.org/Vol-2596/paper1.pdf TF-IDF vs word embeddings for morbidity identification in clinical notes: An initial study . In Proceedings of the First Workshop on Smart Per...

    2020

  14. [22]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  15. [23]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    2022 arXiv

  16. [24]

    Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/f26119b4ffe38c24d97e4c49d334b99e-Paper-Conference.pdf Flocks of stochastic parrots: Differentially private prompt learning for large language ...

    2023

  17. [25]

    Jacques Gaume, Gerhard Gmel, Mohamed Faouzi, and Jean-Bernard Daeppen. 2009. https://doi.org/https://doi.org/10.1016/j.jsat.2008.12.001 Counselor skill influences outcomes of brief motivational interventions . Journal of Substance Abuse Treatment, 37(2):151--159

    2009 doi

  18. [26]

    Benedetta Giovanola and Simona Tiribelli. 2023. https://doi.org/10.1007/S00146-022-01455-6 Beyond bias and discrimination: redefining the AI ethics principle of fairness in healthcare machine-learning algorithms . AI Soc. , 38(2):549--563

    2023 doi

  19. [27]

    Jennifer Hettema, Julie Steele, and William R Miller. 2005. Motivational interviewing. Annu. Rev. Clin. Psychol., 1:91--111

    2005

  20. [28]

    Van Hoang, Eoin Rogers, and Robert Ross. 2024. https://aclanthology.org/2024.clpsych-1.3 How can client motivational language inform psychotherapy agents? In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 23--40, St. ...

    2024

  21. [29]

    Siddharth Karamcheti, Laurel Orr, Jason Bolton, Tianyi Zhang, Karan Goel, Avanika Narayan, Rishi Bommasani, Deepak Narayanan, Tatsunori Hashimoto, Dan Jurafsky, et al. 2021. Mistral--a journey towards reproducible language model training

    2021

  22. [30]

    Vivek Kumar., Simone Balloccu., Zixiu Wu., Ehud Reiter., Rim Helaoui., Diego Recupero., and Daniele Riboni. 2023. https://doi.org/10.5220/0011531400003523 Data augmentation for reliability and fairness in counselling quality classification . In Proceedings of the 1st Workshop ...

    2023 doi

  23. [31]

    Vivek Kumar, Simone Balloccu, Zixiu Wu, Ehud Reiter, Rim Helaoui, Diego Reforgiato Recupero, and Daniele Riboni. 2023 a . Data augmentation for reliability and fairness in counselling quality classification

    2023

  24. [32]

    Vivek Kumar, Giacomo Medda, Diego Reforgiato Recupero, Daniele Riboni, Rim Helaoui, and Gianni Fenu. 2023 b . How do you feel? information retrieval in psychotherapy and fair ranking assessment. In Advances in Bias and Fairness in Information Retrieval, pages 119--133, Cham. S...

    2023

  25. [33]

    Vivek Kumar, Diego Reforgiato Recupero, Daniele Riboni, and Rim Helaoui. 2021. https://doi.org/10.1109/ACCESS.2020.3043221 Ensembling classical machine learning and deep learning approaches for morbidity identification from clinical notes . IEEE Access, 9:7107--7126

    2021

  26. [34]

    Vivek Kumar, Diego Reforgiato Recupero, Rim Helaoui, and Daniele Riboni. 2022. https://doi.org/10.1109/ACCESS.2022.3201542 K-lm: Knowledge augmenting in language models within the scholarly domain . IEEE Access, 10:91802--91815

    2022

  27. [35]

    Vivek Kumar, Prayag Tiwari, and Sushmita Singh. 2023 c . https://doi.org/10.18653/v1/2023.wassa-1.55 VISU at WASSA 2023 shared task: Detecting emotions in reaction to news stories using transformers and stacked embeddings . In Proceedings of the 13th Workshop on Computational ...

    2023 doi

  28. [36]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942

    2019 arXiv

  29. [37]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.acl-main.703 BART : Denoising sequence-to-sequence pre-training for natural language generation, translatio...

    2020 doi

  30. [38]

    Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.397 H alu E val: A large-scale hallucination evaluation benchmark for large language models . In Proceedings of the 2023 Conference on Empirical Methods in Na...

    2023 doi

  31. [39]

    Rumeng Li, Xun Wang, and Hong Yu. 2023 b . https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.474 Two directions for clinical data generation with large language models: Data-to-label and label-to-data . In Findings of the Association for Computational Linguistics: EMNLP 2023, Si...

    2023 doi

  32. [40]

    Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. 2023 c . https://doi.org/10.18653/V1/2023.EMNLP-MAIN.647 Synthetic data generation with large language models for text classification: Potential and limitations . In Proceedings of the 2023 Conference on Empirical Methods in ...

    2023 doi

  33. [41]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  34. [42]

    Burke, Robert Butters, Derrik Tollefson, Christopher Butler, and Stephen Rollnick

    Brad Lundahl, Teena Moleni, Brian L. Burke, Robert Butters, Derrik Tollefson, Christopher Butler, and Stephen Rollnick. 2013. https://doi.org/https://doi.org/10.1016/j.pec.2013.07.012 Motivational interviewing in medical care settings: A systematic review and meta-analysis of ...

    2013 doi

  35. [43]

    Martins and Daniel W

    Renata K. Martins and Daniel W. McNeil. 2009. https://doi.org/https://doi.org/10.1016/j.cpr.2009.02.001 Review of motivational interviewing in promoting health behaviors . Clinical Psychology Review, 29(4):283--293

    2009 doi

  36. [44]

    Manual for the motivational interviewing skill code (misc)

    William R Miller, Theresa B Moyers, Denise Ernst, and Paul Amrhein. Manual for the motivational interviewing skill code (misc)

  37. [45]

    William R Miller and Stephen Rollnick. 2012. Motivational interviewing: Helping people change. Guilford press

    2012

  38. [46]

    Sergio Morales, Robert Claris \' o , and Jordi Cabot. 2023. https://doi.org/10.1109/ASE56229.2023.00018 Automating bias testing of llms . In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023 , pages 1705--1707. IEEE

    2023

  39. [47]

    TB Moyers, JK Manuel, D Ernst, T Moyers, J Manuel, D Ernst, and C Fortini. 2014. Motivational interviewing treatment integrity coding manual 4.1 (miti 4.1). Unpublished manual

    2014

  40. [48]

    Theresa B Moyers, Tim Martin, Jon M Houck, Paulette J Christopher, and J Scott Tonigan. 2009. From in-session behaviors to drinking outcomes: a causal chain for motivational interviewing. Journal of consulting and clinical psychology, 77(6):1113

    2009

  41. [49]

    Theresa B Moyers and Stephen Rollnick. 2002. A motivational interviewing perspective on resistance in psychotherapy. Journal of clinical psychology, 58(2):185--193

    2002

  42. [50]

    Roberto Navigli, Simone Conia, and Bj\" o rn Ross. 2023. https://doi.org/10.1145/3597307 Biases in large language models: Origins, inventory, and discussion . J. Data and Information Quality, 15(2)

    2023 doi

  43. [51]

    World Health Organization. 2022. World mental health report: Transforming mental health for all

    2022

  44. [52]

    Ver \'o nica P \'e rez-Rosas, Rada Mihalcea, Kenneth Resnicow, Satinder Singh, and Lawrence An. 2016. https://doi.org/10.18653/v1/W16-0305 Building a motivational interviewing dataset . In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, ...

    2016 doi

  45. [53]

    Madeline Reinert, Danielle Fritze, and Theresa Nguyen. 2021. The state of mental health in america 2022

    2021

  46. [54]

    Laria Reynolds and Kyle McDonell. 2021. https://doi.org/10.1145/3411763.3451760 Prompt programming for large language models: Beyond the few-shot paradigm . In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA '21, New York, NY, USA. A...

    2021

  47. [55]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108

    2019 arXiv

  48. [56]

    Advait Sarkar. 2023. https://doi.org/10.1145/3596671.3597650 Exploring perspectives on the impact of artificial intelligence on the creativity of knowledge work: Beyond mechanised plagiarism and stochastic parrots . In Proceedings of the 2nd Annual Meeting of the Symposium on ...

    2023

  49. [57]

    Richard Shiffrin and Melanie Mitchell. 2023. https://doi.org/10.1073/pnas.2300963120 Probing the psychology of ai models . Proceedings of the National Academy of Sciences, 120(10):e2300963120

    2023 doi

  50. [58]

    Vasudevan

    Benjamin Smith, Anahita Khojandi, and Rama K. Vasudevan. 2024. https://doi.org/10.1145/3609502 Bias in reinforcement learning: A review in healthcare applications . ACM Comput. Surv. , 56(2):52:1--52:17

    2024 doi

  51. [59]

    Yu Song, Santiago Miret, Huan Zhang, and Bang Liu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.380 H oney B ee: Progressive instruction finetuning of large language models for materials science . In Findings of the Association for Computational Linguistics: EMNLP 202...

    2023 doi

  52. [60]

    Tiberiu Sosea and Cornelia Caragea. 2020. Canceremo: A dataset for fine-grained emotion detection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8892--8904

    2020

  53. [61]

    Hills, and Yoed N

    Massimo Stella, Thomas T. Hills, and Yoed N. Kenett. 2023. https://doi.org/10.1073/pnas.2312911120 Using cognitive psychology to understand gpt-like models needs to extend beyond human biases . Proceedings of the National Academy of Sciences, 120(43):e2312911120

    2023 doi

  54. [62]

    Smith, and Tao Yu

    Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. https://openreview.net/pdf?id=qY1hlv7gwg Selective annotation makes language models better few-shot learners . In The Eleve...

    2023

  55. [63]

    Nicholas L Syn, Andrea Li-Ann Wong, Soo-Chin Lee, Hock-Luen Teoh, James Wei Luen Yip, Raymond Cs Seet, Wee Tiong Yeo, William Kristanto, Ping-Chong Bee, L M Poon, Patrick Marban, Tuck Seng Wu, Michael D Winther, Liam R Brunham, Richie Soong, Bee-Choo Tai, and Boon-Cher Goh. 20...

    2018

  56. [64]

    Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. 2023. https://doi.org/10.48550/ARXIV.2303.04360 Does synthetic data generation of llms help clinical text mining? CoRR, abs/2303.04360

    2023 doi arXiv

  57. [65]

    Leili Tavabi, Trang Tran, Kalin Stefanov, Brian Borsari, Joshua Woolley, Stefan Scherer, and Mohammad Soleymani. 2021. https://doi.org/10.18653/v1/2021.clpsych-1.13 Analysis of behavior classification in motivational interviewing . In Proceedings of the Seventh Workshop on Com...

    2021 doi

  58. [66]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  59. [67]

    Satvik Tripathi, Rithvik Sukumaran, and Tessa S Cook. 2024. https://doi.org/10.1093/jamia/ocad258 Efficient healthcare with large language models: optimizing clinical workflow and enhancing patient care . Journal of the American Medical Informatics Association, page ocad258

    2024 doi

  60. [68]

    Alper Kursat Uysal and Serkan Gunal. 2014. The impact of preprocessing on text classification. Information Processing & Management, 50(1):104--112

    2014

  61. [69]

    kelly is a warm person, joseph is a role model

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai - Wei Chang, and Nanyun Peng. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.243 "kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters . In Findings of the Association for...

    2023 doi

  62. [70]

    Yuqing Wang, Yun Zhao, and Linda Petzold. 2023. https://proceedings.mlr.press/v219/wang23c.html Are large language models ready for healthcare? a comparative study on clinical language understanding . In Proceedings of the 8th Machine Learning for Healthcare Conference, volume...

    2023

  63. [71]

    Anuradha Welivita and Pearl Pu. 2022. https://aclanthology.org/2022.coling-1.293 Curating a large-scale motivational interviewing dataset using peer support forums . In Proceedings of the 29th International Conference on Computational Linguistics, pages 3315--3330, Gyeongju, R...

    2022

  64. [72]

    Stanislaw Wozniak and Jan Kocon. 2023. https://doi.org/10.1109/ICDMW60847.2023.00108 From big to small without losing it all: Text augmentation with chatgpt for efficient sentiment analysis . In IEEE International Conference on Data Mining, ICDM 2023 - Workshops, Shanghai, Chi...

    2023

  65. [73]

    Zixiu Wu, Simone Balloccu, Vivek Kumar, Rim Helaoui, Diego Reforgiato Recupero, and Daniele Riboni. 2023. https://doi.org/10.3390/fi15030110 Creation, analysis and evaluation of annomi, a dataset of expert-annotated counselling dialogues . Future Internet, 15(3)

    2023 doi

  66. [74]

    Zixiu Wu, Simone Balloccu, Vivek Kumar, Rim Helaoui, Ehud Reiter, Diego Reforgiato Recupero, and Daniele Riboni. 2022. https://doi.org/10.1109/ICASSP43922.2022.9746035 Anno-mi: A dataset of expert-annotated counselling dialogues . In ICASSP 2022 - 2022 IEEE International Confe...

    2022

  67. [75]

    Zixiu Wu, Rim Helaoui, Vivek Kumar, Diego Reforgiato Recupero, and Daniele Riboni. 2021. https://doi.org/10.1145/3395035.3425228 Towards detecting need for empathetic response in motivational interviewing . In Companion Publication of the 2020 International Conference on Multi...

    2021

  68. [76]

    Guojun Yan, Jiahuan Pei, Pengjie Ren, Zhaochun Ren, Xin Xin, Huasheng Liang, Maarten de Rijke, and Zhumin Chen. 2022. Remedi: Resources for multi-domain, multi-service, medical dialogues. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development...

    2022

  69. [77]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf Xlnet: Generalized autoregressive pretraining for language understanding . In...

    2019

  70. [78]

    Kai-Ching Yeh, Jou-An Chi, Da-Chen Lian, and Shu-Kai Hsieh. 2023. Evaluating interfaced llm bias. In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), pages 292--299

    2023

  71. [79]

    Ping Yu, Hua Xu, Xia Hu, and Chao Deng. 2023. Leveraging generative AI and large language models: A comprehensive roadmap for healthcare integration. Healthcare (Basel), 11(20):2776

    2023

  72. [80]

    Jiayi Yuan, Ruixiang Tang, Xiaoqian Jiang, and Xia Hu. 2023. Large language models for healthcare data augmentation: An example on patient-trial matching. AMIA Annu. Symp. Proc., 2023:1324--1333

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.