{"id":"48870ca5-f42a-406c-b7dd-2a3c7d87c0f1","arxiv_id":"2411.17672","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-generated synthetic synopses conditioned on target PHQ-8 scores, added to real DAIC-WOZ synopses, reduce PHQ-8 regression error (RMSE 4.64, MAE 3.66) in a single-run BERT evaluation.","lead":"The paper uses an open-source LLM to condense clinical interview transcripts into synopses, then generates synthetic synopses with deliberately different depression scores to enlarge a small, imbalanced dataset. A BERT regressor trained on the real plus synthetic synopses predicts PHQ-8 scores on DAIC-WOZ with lower error than models trained on real data alone.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic PHQ-8 labels are never validated; if Llama 3.2 writes stereotyped text that does not reflect the assigned severity, the reported test improvement is an artifact of LLM style rather than clinical signal.","rationale":"The reader's weakest_assumption is precisely the load-bearing condition: synthetic synopses must actually reflect the assigned PHQ-8 score. The paper reports no validation of this, and the entire utility evaluation rests on it. If Llama 3.2 produces text that merely matches a prompt-provided score in a stereotyped way, the BERT model can achieve low test RMSE on the test set (which is also Llama-generated) by learning these stylistic cues, without learning clinically generalizable depression signal. This would invalidate the central claim that synthetic data improves depression prediction. The proposed clinician-rating check is feasible and would directly settle the issue. I agree with the reader's assessment and thus recommend no change to the conditional verdict.","tokens_in":11640,"tokens_out":10993,"duration_ms":98425,"concrete_test":"Have two independent clinician raters (blinded to the assigned score) rate a stratified sample of 50 synthetic synopses on the PHQ-8 scale, and compute correlation or ICC with the assigned scores. If the correlation is below 0.5 or raters cannot distinguish low vs. high assigned scores, the synthetic labels are not valid and the training signal is suspect. Additionally, as a computational control, train BERT on the same synthetic texts but with randomly permuted labels; if test RMSE remains near the reported 4.64, the model is exploiting label-text artifacts rather than learned severity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that score-conditioned synthetic synopses improve PHQ-8 prediction. The pipeline relies on the assumption that a synopsis generated by Llama 3.2 when prompted with a target PHQ-8 score actually expresses that severity. This assumption is never tested. The synthetic synopsis is conditioned on a randomly sampled score; the LLM is told to 'match' that score, but no human or independent clinical rating checks whether the resulting text is consistent with the requested severity. As a result, the training labels may simply reflect the LLM's stereotyped depiction of depression (e.g., more negative emotion words, hopeless statements) rather than realistic clinical presentation. Since the test set is also composed of Llama 3.2-generated synopses (from real transcripts via step 1), the BERT model can learn to detect these LLM-specific stylistic markers, producing low RMSE on the test set without learning transferable clinical signal. This would not transfer to raw transcripts or to synopses generated by a different LLM, undermining the claimed 'improved depression prediction.' The paper's fidelity PCA only shows embedding-space overlap, and privacy metrics do not address label fidelity. Therefore the reported RMSE/MAE improvement is not evidence of clinically valid synthetic augmentation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-step chain-of-thought pipeline in which Llama 3.2 first converts DAIC-WOZ clinical interview transcripts into concise synopses and sentiment analyses, then generates new synthetic synopses/sentiments conditioned on a randomly sampled PHQ-8 score. A BERT regressor is trained on real synopses, synthetic synopses, or the combination, with the combined dataset reported to achieve the best test RMSE of 4.64 and MAE of 3.66, outperforming a zero-shot GPT-4o baseline, a Random Forest baseline, and a Dual Encoder baseline. The paper additionally reports embedding-distance-based privacy metrics and a PCA fidelity visualization. The central claim is that synthetic data augmentation improves depression severity prediction while preserving privacy.","tokens_in":11916,"tokens_out":4497,"duration_ms":41673,"significance":"If the results are validated, the pipeline would offer a practical, privacy-aware approach to augmenting small, imbalanced mental-health datasets with an open-source LLM. The paper makes its code available, uses an open-source model, and addresses a real clinical need. The conceptual idea of generating severity-conditioned summaries is appealing. However, the current experimental design does not yet establish that the reported improvements come from the synthetic generation quality rather than from dataset-size or label-distribution effects, nor that the synthetic labels carry clinically valid signal. The significance is therefore conditional on addressing these validity concerns.","major_comments":[{"comment":"The main comparison is confounded by dataset size and label rebalancing. The combined training set contains 416 samples (107 real + 309 synthetic) versus 107 real samples for the BERT 'Train' condition, and the synthetic data is explicitly designed to oversample high PHQ-8 scores. The reported improvement from RMSE 5.59 to 4.64 could be due simply to having more training data or to a more balanced label distribution, not to any property of the LLM-generated synopses. The authors should include an ablation that trains BERT on an equally sized real dataset (e.g., by oversampling real transcripts) and on a rebalanced-only real dataset, so that the effect of the synthetic generation itself can be isolated.","section":"Utility, Table 1"},{"comment":"The synthetic labels are produced by construction rather than measured from the text. The second-stage prompt explicitly instructs Llama 3.2 to 'create a new synopsis ... that matches a depression/PHQ8 score of {PHQ8 Score}', and that same sampled score is then used as the training label for the resulting text. There is no human or clinical rating to verify that the generated text actually reflects the assigned severity. If the LLM produces stereotyped depression language (e.g., more negative emotion words) that does not correspond to realistic clinical presentation, the BERT model can learn those stylistic markers. This concern is amplified because the test set is also composed of Llama 3.2-generated synopses from real transcripts, so the low test error may reflect learning the LLM's text-generation style rather than transferable clinical signal. The paper needs to validate synthetic label fidelity, for example by having clinicians rate a sample of synthetic synopses against their assigned PHQ-8 scores, or by evaluating on raw transcripts rather than LLM-generated summaries.","section":"Chain of Thought Prompting, Appendix prompt"},{"comment":"No uncertainty quantification is reported. Each entry in Table 1 is a single number for RMSE and MAE, with no standard deviation across random seeds, no confidence intervals, and no significance tests. Given the small training set and the stochasticity of BERT fine-tuning, a 0.03 RMSE difference (e.g., 4.67 for Dual Encoder versus 4.64 for the combined BERT) is likely within noise. The authors should run multiple seeds and report means with variances, and perform paired significance tests where appropriate, before claiming that the combined model outperforms the baselines.","section":"Table 1, Results and Discussions"},{"comment":"The privacy evaluation does not support the claim that the synthetic data protects participant privacy. The minimum-embedding-distance metric only shows that synthetic synopses are farther from real synopses than real synopses are from each other in BERT embedding space. This is expected, since the synthetic synopses are deliberately written as new stories, and it does not rule out membership inference, attribute inference, or re-identification through other means. The abstract and conclusion make broad claims about 'safeguarding participant privacy' that go beyond what this metric demonstrates. The authors should either temper these claims or evaluate privacy with a more meaningful threat model, such as a nearest-neighbor re-identification attack against the actual synthetic data and transcripts.","section":"Privacy, Table 2"}],"minor_comments":[{"comment":"The phrase 'despressive disorders' is a typo and should read 'depressive disorders.'","section":"Related Work"},{"comment":"The sentence ending 'by emphasizing style transfer through in learning' appears to be missing the word 'context'; it should read 'through in-context learning.'","section":"Chain of Thought Prompting"},{"comment":"The sentence 'we quantitatively evaluate the privacy of our synthetic data by analyzing we quantitatively evaluate' contains a duplicated clause and should be rewritten.","section":"Privacy"},{"comment":"The prompt template for the second stage is incomplete: the placeholder for the depression-level description ends with '\"One of \"' and the actual mapping from PHQ-8 score ranges to text descriptions is not provided.","section":"Appendix"},{"comment":"The paper does not specify how the test set is preprocessed: whether the test transcripts are also converted to synopses with the same Llama 3.2 model, and if so, this should be stated explicitly because it affects the interpretation of the test error.","section":"Experimental Setup"},{"comment":"Details on the sampling distribution for the random PHQ-8 scores are missing; the paper says 'randomly generated PHQ-8 score (ranging from 0 to 24)' but does not say whether the sampling is uniform or stratified, which is relevant given the goal of rebalancing.","section":"Model Details"},{"comment":"The GPT-4o baseline is a zero-shot model evaluated on synopses, not fine-tuned on the training set; this should be explicitly labeled as a zero-shot baseline in the table caption or text to avoid implying an equivalent comparison.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the open-source code is a strength. However, the two central claims — that synthetic augmentation improves performance and that it preserves privacy — are not yet established because of the confounded utility comparison, the unvalidated synthetic labels, and the weak privacy metric. The label-conditioning issue is particularly serious: because the test set is also generated by the same LLM, the reported improvement may be an artifact of style transfer rather than clinical signal. I would encourage the editor to send the manuscript back for a revision that includes controlled ablations, synthetic label validation, and significance testing. The paper is not beyond repair, but it is not ready for publication in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable paper to send out for review, but the central claim — that the LLM synthetic data pipeline improves depression prediction — is not actually supported by the reported experiments. The two-step pipeline (transcript to synopsis/sentiment, then score-conditioned regeneration) is a specific and reasonably motivated variation on existing synthetic-data work in mental health NLP. The writing is clear, the code is shared, and the privacy motivation is legitimate.\n\nThe main problem is the comparison. The combined training set is 416 samples versus 107 for the real-only BERT, and there is no control for dataset size or label rebalancing. Any method that adds more data, even duplicated real data, could produce the same RMSE drop. So the improvement cannot be attributed to the synthetic pipeline. The 0.03 RMSE advantage over the Dual Encoder baseline is small, and with no error bars or significance tests, it is not convincing.\n\nThere is also a validity concern that the stress-test note gets right. The target PHQ-8 score is injected into the prompt that generates each synthetic sample, so the label is baked into the text by construction. The authors never check whether a human or an independent clinical rating would assign the same severity to the generated synopsis. And because the test set also uses Llama-generated synopses (the model input is synopsis+sentiment for both training and test), the model could be learning the stylistic signature of Llama's 'depressed' writing rather than transferable clinical signal. To support the claim, the authors would need to show transfer to raw transcripts or to synopses from a different generator.\n\nThat said, the paper is not a write-off. The idea of distilling a transcript into a synopsis as a privacy-preserving intermediate is worth exploring, and the authors have made an effort to report fidelity and privacy metrics, even if those metrics are weak proxies. With a proper control (e.g., training on a similarly sized real dataset, or real data with simple oversampling), label validation, and error bars, this could be a useful contribution.\n\nFor peer review: yes, it deserves a serious referee. The flaws are fixable and the problem is important. But I would not cite it as evidence that LLM synthetic data helps depression prediction until the evaluation is redone.","headline":"The synthetic-data pipeline is plausible but the evaluation is confounded by dataset size and shared generator style; still worth a referee.","tokens_in":12434,"tokens_out":4326,"would_cite":false,"duration_ms":37146,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated interview synopses improve depression severity prediction, the paper claims, with a combined real-plus-synthetic training set producing the lowest error.","keywords":["synthetic data","large language models","depression prediction","PHQ-8","data augmentation","chain-of-thought prompting","BERT","clinical interviews"],"falsifier":"Ask clinicians, blind to the assigned score, to rate the depression severity of each of the 309 synthetic synopses; if their ratings do not track the PHQ-8 scores written into the prompts, then the synthetic labels are unreliable and the reported test gains could reflect the LLM's stereotypes rather than clinical signal. A second decisive check would be to train the same BERT model on synthetic data whose PHQ-8 labels have been randomly permuted; if performance stays the same, the model is not using the severity signal at all.","tokens_in":11424,"feed_emoji":"🧠","tokens_out":6102,"duration_ms":45926,"temperature":0.7,"pith_summary":"This paper proposes a two-step pipeline in which an open-source large language model, Llama 3.2, distills therapist-interview transcripts into short synopses with sentiment analyses and then rewrites those synopses at new, randomly sampled PHQ-8 depression severity scores. The claim is that adding these synthetic synopses to the real training data improves a BERT model's ability to predict depression severity, with the combined model reaching the lowest errors in the study (RMSE 4.64, MAE 3.66) and beating larger or purpose-built baselines trained only on real data. If true, the pipeline offers a way to enlarge and rebalance small, sensitive mental-health datasets without distributing raw patient transcripts. The authors also report that synthetic points sit farther from real transcripts in embedding space than real points sit from each other, which they read as evidence of privacy protection.","feed_headline":"LLM-made therapy synopses improve depression prediction","feed_subtitle":"Adding Llama-generated summaries to real transcripts cut prediction error below GPT-4o and Dual Encoder baselines.","key_machinery":"The load-bearing mechanism is a chain-of-thought prompting pipeline built on Llama 3.2. In the first link, the raw interview transcript is compressed into a synopsis and a sentiment analysis; in the second, the model writes a fresh storyline and sentiment that matches a randomly sampled PHQ-8 score (0 to 24), using the original summary only as a structural reference. This 'distill then rewrite' design simultaneously solves two problems: the short synopsis fits BERT's 512-token limit (average 184 words versus 1389 for raw transcripts), and the forced rewrite at a new severity level keeps the synthetic text from being a near-copy of any real transcript. The assigned PHQ-8 score is then used as the training label for that synthetic sample.","core_discovery":"The paper's central claim is that synthetic data produced by its chain-of-thought prompting pipeline is faithful enough to real clinical interviews to serve as effective training data, and distant enough to preserve privacy, so that augmenting the real DAIC-WOZ training set with 309 synthetic synopses yields better depression severity prediction than training on real data alone. In the authors' experiments, the BERT model trained on real and synthetic synopses together achieved RMSE 4.64 and MAE 3.66, outperforming a BERT trained on real synopses (5.59/4.71), a BERT trained on synthetic data alone (4.80/4.06), and the Dual Encoder and Random Forest baselines trained on real transcripts. The synthetic data also rebalanced the training set's severity distribution, which had only 46 of 189 patients above the PHQ-8 cutoff of 10.","pith_inferences":["The paper's privacy evidence rests on embedding distances, which measure dissimilarity, not re-identification risk; a stronger test would be a membership-inference attack, and a fair reading of this paper should treat the privacy guarantee as suggestive rather than proven.","Because the conditioning score is the only severity signal the LLM sees, the pipeline is only as clinically meaningful as Llama 3.2's understanding of PHQ-8 anchors; a direct test would compare human ratings of synthetic texts against the assigned score.","The same distill-and-rewrite pipeline could be applied to other self-report instruments (PHQ-9, GAD-7) or other sensitive clinical corpora; the paper does not test this, but nothing in its design is specific to depression.","The finding that synthetic-only training beats real-synopsis training hints that the real training set's imbalance was actively hurting the model; rebalancing by resampling real data rather than synthesizing new data would isolate how much of the gain is due to balance alone."],"forward_implications":["If the central claim is correct, synthetic synopses can substitute for raw transcripts when privacy constraints block access to real data; the model trained on synthetic data alone (RMSE 4.80) already outperformed the BERT trained on real synopses.","Combining synthetic and real data rebalances skewed severity distributions, which should reduce the risk of biased predictions for severely depressed patients underrepresented in real corpora.","The synopsis-plus-sentiment representation itself becomes a reusable, compact input format for downstream mental-health NLP tasks, since it preserves performance while staying within model token limits.","LLMs such as Llama 3.2 can act as scalable generators of labeled clinical narratives, lowering the cost of building larger depression-detection datasets.","The pipeline's privacy argument implies that versions of the real transcripts can be shared in summarized, rewritten form without the closest-neighbor leakage seen in the raw data."],"supporting_citations":[{"why":"Defines the PHQ-8 score, which is the depression severity label the pipeline conditions on and predicts.","marker":"(Kroenke et al. 2009)"},{"why":"Provides the DAIC-WOZ dataset of clinical interview transcripts, the real data source for generation and evaluation.","marker":"(Gong and Poellabauer 2017)"},{"why":"Supplies BERT, the model architecture fine-tuned for depression score regression in the experiments.","marker":"(Devlin 2018)"},{"why":"Dual Encoder is the strongest purpose-built baseline that the combined synthetic-plus-real model claims to outperform.","marker":"(Lau, Zhu, and Chan 2023)"},{"why":"Random Forest on selected text serves as a classical baseline for DAIC-WOZ depression assessment.","marker":"(Sun et al. 2017)"},{"why":"GPT-4o is used as a zero-shot baseline for depression score prediction from synopses.","marker":"(OpenAI 2024)"},{"why":"Supplies the style-transfer-by-prompting idea that motivates rewriting synopses at new severity levels without fine-tuning.","marker":"(Reif et al. 2021)"}],"fun_headline_variants":["LLM-generated therapy synopses sharpen depression prediction","AI-made summaries boost depression severity accuracy","Synthetic data from LLMs improves depression detection","Fake therapy notes cut depression prediction error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a synthetic synopsis generated from a prompt containing a target PHQ-8 score actually exhibits that level of depression severity; the score is used as the training label with no clinician check that the text matches it.","fun_headline_variants_meta":{"raw":{"variants":["LLM-generated therapy synopses sharpen depression prediction","AI-made summaries boost depression severity accuracy","Synthetic data from LLMs improves depression detection","Fake therapy notes cut depression prediction error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1264,"prompt_tokens":975,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":232}},"tokens_in":591,"tokens_out":289,"duration_ms":3416,"temperature":1.0,"reasoning_tokens":232,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:51:02.850932+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask clinicians, blind to the assigned score, to rate the depression severity of each of the 309 synthetic synopses; if their ratings do not track the PHQ-8 scores written into the prompts, then the synthetic labels are unreliable and the reported test gains could reflect the LLM's stereotypes rather than clinical signal. A second decisive check would be to train the same BERT model on synthetic data whose PHQ-8 labels have been randomly permuted; if performance stays the same, the model is not using the severity signal at all.","supporting_citations":[{"cited_title":"W.; Spitzer, R","cited_arxiv_id":null,"evidence_quote":"Defines the PHQ-8 score, which is the depression severity label the pipeline conditions on and predicts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DAIC-WOZ dataset of clinical interview transcripts, the real data source for generation and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Dual Encoder is the strongest purpose-built baseline that the combined synthetic-plus-real model claims to outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Random Forest on selected text serves as a classical baseline for DAIC-WOZ depression assessment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4o is used as a zero-shot baseline for depression score prediction from synopses."}],"review_version":1}