{"id":"9cca2379-6e51-4c9e-855b-dccec1c43594","arxiv_id":"2608.07316","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM persona questionnaire responses yield interpretable text features that transfer, with modest accuracy, to classifying depression in real human speech.","lead":"The authors had nine large language models act as personas and complete psychometric questionnaires with written explanations, then trained random forests to predict the questionnaire scores from the language features. The trained models separated real clinically depressed speakers from controls, suggesting that synthetic text can carry some transferable psychometric signal.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human-transfer AUCs may be verbosity artifacts: length-sensitive network features (N nodes, N edges, N components) are used without any length control in the Androids Corpus analysis.","rationale":"The central claim is the transfer from synthetic LLM explanations to real clinical speech. The in-sample R2, SHAP directions, and diary transfer are interesting but not sufficient: diaries are generated by the same LLM family with the same feature pipeline, so they share systematic artifacts. The human transfer is where the claim becomes falsifiable. The reader's weakest_assumption identifies the same length-proxy concern, and I agree. The paper is openly transparent about the length sensitivity of N nodes/N edges/N components (Section 4.1) and about the absence of human questionnaire scores in the corpus, but it does not apply any length control to the human analysis. This is not an internal inconsistency; it is an external confound that can fully explain the observed AUC if transcript lengths differ by diagnosis. It is the most load-bearing concern because it attacks the one result that makes the paper more than an LLM-auditing exercise. Other issues (model selection after observing Mistral Small's diary transfer, circular in-sample R2) would weaken confidence but would not, by themselves, overturn the central claim. The proposed length-control reanalysis can be run on the existing transcripts and would settle the question. Therefore the reader's CONDITIONAL verdict remains unchanged pending that test.","tokens_in":34704,"tokens_out":5510,"duration_ms":50759,"concrete_test":"Re-run the human transfer with transcript length controlled. (1) Compute total word count (and speaking time, if available) per participant and test whether it differs between depressed and control groups. (2) Refit the Network+Emotion RF on the LLM training data and then evaluate on human transcripts after (a) adding log(word count) as a covariate, (b) restricting to the length-invariant features (degree/valence assortativity, clustering, shortest path, diameter, all eight EmoAtlas z-scores), and (c) truncating/padding every transcript to the same word count before feature extraction. (3) Compare AUC and LOO-CV accuracy under these three controls with the unadjusted Tables 7 and 9.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The human-transfer step (Section 2.2.5, Tables 7 and 9) is the only evidence that the LLM-trained language-to-score mapping overlaps with genuine human depressive markers. The models used there are the Network+Emotion RFs, and three of their eight network features—N nodes, N edges, N components—are explicitly acknowledged in Section 4.1 to scale mechanically with text length. The Androids Corpus comparison involves 63 clinically depressed and 52 control transcripts, but no analysis controls for word count, speaking duration, or transcript length. If the depressed and control groups differ in how much they spoke, a model that has learned (from LLM questionnaire explanations) to associate higher scores with larger networks can separate the groups without encoding anything about depressive content. EmoAtlas emotion z-scores are length-normalized, but the RF can exploit the length-proxy features; the reported AUCs of .695 and .780 could therefore be verbosity artifacts. Because the paper presents the human transfer as the decisive test of the central claim, the missing length control is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NLP Psychometrics, a framework that trains random-forest regressors on LLM-generated questionnaire explanations (produced under controlled 'cognitive digital shadow' personas) to predict psychometric scores (SWLS, PHQ-9, DASS-21) from interpretable network and emotion features extracted from text. The authors report strong in-sample R² values, SHAP-based feature attributions, and two transfer tests: (i) to LLM-generated diary entries without retraining, and (ii) to real transcribed clinical speech from the Androids Corpus (63 depressed vs. 52 control participants). The central claim is that the language-to-score mapping learned from synthetic data overlaps with genuine markers of depressive language in humans, as evidenced by AUCs of 0.695 (PHQ-9) and 0.780 (DASS-21 Depression) and classification accuracies of 62% and 68% in the human transfer.","tokens_in":34901,"tokens_out":3795,"duration_ms":34734,"significance":"If the human-transfer result is correct, the paper provides a notable demonstration that synthetic LLM data can be used to learn interpretable, text-based markers of depression that generalize to authentic human speech. The framework is methodologically transparent: it uses established psychometric instruments, an ablation design across four feature families, SHAP for interpretability, and it openly reports model failures (e.g., GPT-OSS-Uncensored) and inconsistent transfer (Qwen-4B-Instruct in Appendix B). The paper's explicit discussion of limitations and its framing of the framework as an auditing tool rather than a validated clinical measure are strengths. However, the human-transfer result, which is load-bearing for the entire enterprise, is currently threatened by a plausible length confound, as the authors themselves acknowledge in Section 4.1. The central claim is therefore defensible but not yet established.","major_comments":[{"comment":"The human-transfer analysis uses only the Network+Emotion random forest, whose feature set includes N nodes, N edges, and N components—features the paper explicitly states in Section 4.1 'scale with raw text length rather than purely with discourse structure.' The Androids Corpus analysis does not control for word count, speaking duration, or transcript length, and the depressed (N=63) and control (N=52) groups are not reported as length-matched. If the two groups differ in verbosity, the AUCs of 0.695 and 0.780 could partly or entirely reflect a length proxy rather than depressive content. This is load-bearing because the human transfer is the only evidence that the synthetic mapping captures genuine human depressive markers. The authors' own Section 4.1 concludes that 'requires explicit verbosity controls before clinical interpretation,' but no such controls are applied to the human-transfer test. Please rerun the human-transfer analysis with length normalization (e.g., dividing N nodes and N edges by word count, or including transcript length as a covariate) and/or demonstrate that the groups do not differ in length and that results survive a length-matched subsample analysis.","section":null},{"comment":"The main-text transfer analyses use Mistral Small, selected after the authors observed that Qwen-4B-Instruct failed to transfer PHQ-9 to diaries under the reference condition (Appendix B). Because the model choice is informed by transfer performance, the reported effect sizes (e.g., r=0.907 for PHQ-9 diary transfer) are at risk of selection bias. The paper does report the Qwen results in an appendix, which is good, but it should additionally state clearly that the human-transfer results are only for the model that succeeded in the diary transfer, and discuss how this selection affects the generalizability of the human-transfer claim. Ideally, report human-transfer results for all LLMs for which data are available, or justify why the selected model is representative.","section":null},{"comment":"The in-sample R² values are computed on data where the persona prompt explicitly includes a mental-health level, and the LLM generates both the score and the explanation. High R² may therefore reflect the LLM's internal consistency between the assigned persona and the generated text—a form of circularity—rather than a general language-to-score law. The human transfer is the only test that breaks this circularity. This makes the length confound in the human transfer (first major comment) even more consequential: if the human result is confounded, the paper's central claim rests on the self-referential in-sample performance, which is not independently informative. The authors should either provide an additional out-of-sample test on human data with proper controls or temper the claim that the mapping 'transfers to humans' until such controls are provided.","section":null}],"minor_comments":[{"comment":"The symbol 'r' is used for the Spearman correlation in Tables 2-4 and for the Mann-Whitney effect size in Tables 5-9. The captions should explicitly distinguish these two statistics to avoid confusion.","section":null},{"comment":"The phrase 'r up to 0.91' refers to the rank-biserial effect size from the Mann-Whitney test, not a Pearson or Spearman correlation. The abstract should clarify this to avoid misinterpretation.","section":null},{"comment":"The AUC formula '1−U/(n_low·n_high)' is correct for the Mann-Whitney U statistic, but the direction convention (whether higher U corresponds to higher predicted scores in the depressed group) should be stated explicitly.","section":null},{"comment":"Several URLs in Table 1 are broken across lines and some are incomplete (e.g., the Mistral Small entry). These should be formatted as proper hyperlinks or footnotes.","section":null},{"comment":"The persona mental-health distribution is a simplified mutually exclusive approximation ('no symptoms' 50%, depression 20%, anxiety 20%, stress 10%). The paper notes this is 'indicative rather than epidemiologically exact,' but it would be helpful to state the sensitivity of the main RF results to these weights, since they directly shape the training labels.","section":null},{"comment":"The comparison between the PHQ-9 and DASS-21 Depression transfer effects uses median score differences on different scales (0-27 vs. 0-21). The AUC values are the more appropriate comparison; the text should avoid implying that the raw median differences are directly comparable.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of cs.CL and makes an interesting methodological contribution, but the central human-transfer claim is currently undercut by the absence of any length control in the Androids Corpus analysis. The authors are clearly aware of the issue (Section 4.1), which is reassuring, but they have not yet acted on it. The additional analysis requested (length-normalized features or length-matched subsamples) is feasible with the existing data and would resolve the main concern. I would also encourage the editor to ensure that the public reframing of the paper's clinical implications matches the current evidence. The paper's transparency—reporting the failed Qwen transfer and the model selection step—is a strength and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The framework is genuinely new: an interpretable pipeline from LLM persona-generated questionnaire explanations to network/emotion features, random forest prediction, SHAP attribution, and transfer first to diary text, then to real transcribed clinical speech. The transfer result—AUC .695 (PHQ-9) and .780 (DASS-21 depression) separating 63 clinically depressed speakers from 52 controls—is the most important claim, and it is the one I would worry about.\n\nThe worry is a length confound. Section 4.1 admits that three of the eight network features (N nodes, N edges, N components) scale mechanically with text length, and the human-data transfer uses exactly those features with no word-count or duration control. The depressed and control transcripts are not reported as length-matched. If the groups spoke different amounts, those features can separate them without encoding depressive content. The authors acknowledge the need for verbosity controls, but only in a limitations paragraph, not in the transfer analysis. For the decisive external validation, that omission is load-bearing.\n\nCredit where due: the ablation across nine feature sets and nine LLMs is thorough; the SHAP stories are coherent (income and affect for SWLS; neuroticism and low-assortativity network structure for depression); and the authors are unusually transparent—they call the framework an auditing tool, disclose that Qwen-4B-Instruct failed diary transfer and they switched to Mistral Small, and flag the circularity of in-sample R2.\n\nSofter issue: model selection. The human-transfer model was chosen after seeing which LLM worked best in-domain and on diaries. Disclosed, but the external result is the best case, not a random draw. The in-sample R2 is circular (same LLM generates scores and text), but they don't overclaim it.\n\nThis paper is for NLP-for-mental-health and computational psychometrics researchers. It deserves serious peer review, but with major revision. Ask for word-count covariates, length-normalized network features, and the Androids length distributions. Code and data release would help. If the transfer survives length control, it is a solid contribution; if not, the framework still works as an LLM-auditing tool, but the clinical-transfer story goes away.","headline":"Novel framework, but the headline human-transfer result is undercut by a length confound the authors themselves identify.","tokens_in":35404,"tokens_out":4275,"would_cite":true,"duration_ms":38283,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language-to-score models trained purely on AI-generated questionnaire answers separate clinically depressed speakers from controls in real transcribed speech, with AUC up to 0.780.","keywords":["NLP psychometrics","cognitive digital shadows","textual forma mentis networks","depression detection","large language models","random forests","SHAP","transfer learning"],"falsifier":"Re-run the human-transfer analysis with N nodes, N edges, and N components either removed or divided by word count, or with transcripts truncated to a common length; if the DASS-21 depression AUC of 0.780 collapses toward chance, the claimed human transfer is largely a verbosity artifact.","tokens_in":34494,"feed_emoji":"🧠","tokens_out":9437,"duration_ms":71435,"temperature":0.7,"pith_summary":"This paper tries to establish that psychometric scores — life satisfaction, depression, anxiety, and stress — can be read off the structure and emotional content of free text, and that the mapping can be learned from synthetic data and then applied to real human language. Nine LLMs, prompted to impersonate personas that vary in sociodemographics, personality, and mental-health profile, answered standard questionnaires (SWLS, PHQ-9, DASS-21) and wrote a short explanation for each item. Random-forest regressors trained on the emotion and network features of those explanations separate clinically depressed speakers from controls in the Androids Corpus transcripts without retraining, with an AUC of 0.780 for DASS-21 depression and 0.695 for PHQ-9. The framework's point is that psychometric prediction from language can be made interpretable — via syntactic network topology and emotional profiles, not opaque embeddings — and that synthetic personas can expose what LLMs encode, provided the mapping is validated on human data.","feed_headline":"Depression detection trained on AI text works on real speech","feed_subtitle":"Models trained on LLM questionnaire answers separate 63 depressed from 52 control speakers, AUC 0.78.","key_machinery":"The carrying machinery is the textual forma mentis network (TFMN), a syntactic-semantic network built from each respondent's item explanations: content words are nodes, edges encode syntactic dependencies and synonymy relations, and the network yields eight structural features (nodes, edges, components, clustering, average shortest path, diameter, degree assortativity, valence assortativity) plus eight emotion z-scores from EmoAtlas. These features, combined with Big Five persona traits and sociodemographics, are fed into ablated random-forest regressors; SHAP attributes each prediction to specific features. The training signal comes from cognitive digital shadows: nine LLMs prompted to impersonate randomly generated personas and complete standardized questionnaires, so each text carries a known psychometric score. Transfer from synthetic to human text is the decisive test: the same random forests, untouched, are applied to diary entries and to transcribed Androids Corpus interviews.","core_discovery":"On its own terms, the paper's central discovery is that a text-to-score mapping trained exclusively on LLM-generated questionnaire explanations carries over to authentic human depressive language. Using only network and emotion features — textual forma mentis network descriptors (nodes, edges, components, clustering, path length, diameter, degree and valence assortativity) and eight Plutchik emotion z-scores — a random forest trained on Mistral Small's DASS-21 depression explanations separates 63 clinically depressed speakers from 52 controls with an AUC of 0.780 and 68% classification accuracy, while the analogous PHQ-9 model reaches 0.695 and 62%. The same models transfer to LLM-generated diaries with rank correlations up to 0.9. The paper reads the topological signature — low degree assortativity, hub-centred star-like networks with many syntactic edges — as a linguistic trace of rumination, and notes that the direction reverses for anxiety. It also reports that sociodemographics alone explain essentially no variance in depression, anxiety, or stress, while emotion features and income drive life-satisfaction predictions.","pith_inferences":["A sharper test of the human transfer would drop or length-normalize the three size-sensitive network features (N nodes, N edges, N components); because the clinical and control interviews were not matched for duration or word count, part of the reported AUC could be a verbosity effect rather than a psycholinguistic one, a point the paper itself flags in its limitations.","The reversed assortativity signal between depression and anxiety, if it survives verbosity controls, gives a concrete topological hypothesis that could be tested directly on human corpora with matched-length transcripts.","The synthetic-to-human pipeline suggests a low-cost way to generate training data for other constructs, such as burnout or PTSD, where no paired text-score corpus yet exists — provided a length-control protocol is built in from the start."],"forward_implications":["If the transfer result is right, psychometric scoring of unseen text no longer requires a matched corpus of human questionnaire-plus-explanation responses; synthetic persona data can bootstrap models for depression-related constructs.","The framework gives an interpretable audit channel for LLMs: SHAP attributions show whether a model's psychometric answers are driven by persona metadata (income, neuroticism), emotional content, or discourse topology, exposing model-specific biases such as the abliterated GPT-OSS variant carrying almost no psychometric signal.","Depression-specific topology (hub-centred, low-assortativity discourse) and anxiety-specific topology (integrated, distributed discourse) are claimed as distinct signatures; if confirmed, the direction of degree assortativity becomes a candidate marker separating rumination from worry.","Cross-domain transfer is uneven — anxiety transfers poorly to diaries — so any application of a text-to-score mapping must be validated register by register and construct by construct."],"supporting_citations":[{"why":"Supplies the Androids Corpus of transcribed clinical and control speech used for the human-transfer test.","marker":"[75]"},{"why":"Provides the Whisper-based Italian transcription of the corpus that the transfer models actually score.","marker":"[77]"},{"why":"Defines EmoAtlas and the textual forma mentis networks and emotion z-scores that generate all network and emotion features.","marker":"[10]"},{"why":"Defines the DASS-21 subscales, including the depression subscale that achieves the strongest human-transfer AUC.","marker":"[21]"},{"why":"Defines the PHQ-9 depression scale, the second instrument transferred to human speech.","marker":"[22]"},{"why":"Establishes the cognitive digital shadows framework used to generate the synthetic questionnaire explanations.","marker":"[18]"},{"why":"Supplies the persona-prompting design that varies sociodemographics and personality in the digital shadows.","marker":"[23]"},{"why":"Defines random forests, the regressor whose trained mapping is transferred without retraining.","marker":"[70]"}],"fun_headline_variants":["AI text-trained model reads real depression speech, AUC 0.78","LLM-taught depression detector hits real speech, AUC 0.78","From AI answers to real patients: text model separates depression","Depression model: trained on AI, works on real speech, AUC 0.78"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the network features separating depressed from control speakers reflect linguistic content rather than the amount of speech, because three of the eight network features grow mechanically with text length and the clinical and control interviews were analyzed without length matching.","fun_headline_variants_meta":{"raw":{"variants":["AI text-trained model reads real depression speech, AUC 0.78","LLM-taught depression detector hits real speech, AUC 0.78","From AI answers to real patients: text model separates depression","Depression model: trained on AI, works on real speech, AUC 0.78"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":3066,"prompt_tokens":1088,"completion_tokens":1978,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":1897}},"tokens_in":704,"tokens_out":1978,"duration_ms":13426,"temperature":1.0,"reasoning_tokens":1897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:24:26.612309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the human-transfer analysis with N nodes, N edges, and N components either removed or divided by word count, or with transcripts truncated to a common length; if the DASS-21 depression AUC of 0.780 collapses toward chance, the claimed human transfer is largely a verbosity artifact.","supporting_citations":[{"cited_title":"The androids corpus: A new publicly available benchmark for speech based depression detection.Depression, 47:11–9, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Androids Corpus of transcribed clinical and control speech used for the human-transfer test."},{"cited_title":"Modeling depressive patterns in italian discourse: Insights from natural language processing, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the Whisper-based Italian transcription of the corpus that the transfer models actually score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DASS-21 subscales, including the depression subscale that achieves the strongest human-transfer AUC."},{"cited_title":"The phq-9: validity of a brief depression severity measure.Journal of general internal medicine, 16(9):606–613, 2001","cited_arxiv_id":null,"evidence_quote":"Defines the PHQ-9 depression scale, the second instrument transferred to human speech."},{"cited_title":"Digital shadows in mental health map how llms simulate depression, anxiety, and stress through language and psychometrics.PsyArXiv,","cited_arxiv_id":null,"evidence_quote":"Establishes the cognitive digital shadows framework used to generate the synthetic questionnaire explanations."}],"review_version":1}