{"id":"756037e6-2de0-40c7-9541-70d80151d5b7","arxiv_id":"2505.06387","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Language network features from emotion-inducing interviews correlate modestly with adolescent psychopathology scores in the training set, but the analysis does not demonstrate out-of-sample prediction.","lead":"This paper tests whether the structure of words and emotions in teenagers' interviews can reveal their mental health scores. It finds modest correlations between language network features and three clinical measures, but the models were not tested on unseen data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The predictive claim in the Conclusions is unsupported because the reported r values come from 4-fold CV on the same 232 interviews, with feature selection and hyperparameter tuning performed in a way that can inflate cross-validated estimates; an honest out-of-sample test is missing.","rationale":"The reader's strongest claim and weakest assumption point directly to the same load-bearing issue: the paper frames the central contribution as prediction but the analysis is explicitly oriented toward in-sample pattern discovery. The methods use 4-fold CV on 232 participants, with feature selection and model selection operating on the same data; the authors even acknowledge that they were not interested in predictability of unseen data. A 4-fold CV can be a legitimate estimate of out-of-sample performance only if every data-dependent choice—feature selection, hyperparameter tuning, model selection—is nested inside the folds and the reported p-values reflect the entire selection process. Here the text is ambiguous about nesting, and the decision to prioritize GBM because it scored higher than RFR in the same tables is itself a form of model selection on the test folds. The permutation test does not fix this because it only permutes labels and does not simulate the full feature-selection/model-selection procedure. I am not claiming the authors behaved dishonestly; the limitation is stated openly in the paper. But the conclusion 'successfully predict' overstates what the evidence can support. I would therefore maintain the reader's REJECT verdict, while noting that the exploratory SHAP findings—such as modularity and core-periphery structure relating to social maladjustment—could be reframed as testable hypotheses for future work with proper held-out validation.","tokens_in":20025,"tokens_out":3673,"duration_ms":40319,"concrete_test":"Set aside 20% of the 232 participants (about 47) before any modeling. On the remaining 185, perform feature selection and GBM/RFR hyperparameter tuning only inside training folds of an inner CV, then train the final model on all 185 and evaluate once on the untouched held-out set; repeat over 10 different random 80/20 splits and record the distribution of out-of-sample Pearson r for Social Maladjustment, Specific Internalising, and Neurodevelopmental Risk. If the mean held-out r is not clearly above zero and its 95% bootstrap CI includes values near zero, the Conclusions' predictive claim should be withdrawn or explicitly reframed as an exploratory, in-sample association finding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Conclusions state that TFMNs 'can successfully predict psychometric levels of psychopathology in adolescents.' For that claim to hold, the reported correlations must reflect prediction on data not used to select features, tune hyperparameters, or choose the model family. The paper's own Limitations section says there was 'a general lack of interest in predictability of unseen data' and that cross-validation was used 'on the whole dataset' to maximize training. That is an explicit admission that predictive generalization was not the goal. Concretely, the feature-selection procedure in Methods ('Machine Learning Models') ranks features by SHAP values and discards features if reshuffling does not deteriorate performance 'during training/validation.' If this selection is applied before or across the 4 CV folds, or if the validation folds contribute to SHAP-based feature rankings, the cross-validated r in Tables 2-5 is optimistically biased. Even under a correctly nested implementation, the p-values reported alongside r are not valid evidence of generalizable prediction: the same 232 participants contributed to choosing k=4, to feature selection, to grid-search hyperparameters, and to the decision to report GBM because it outperformed RFR in Table 2. The permutation test in Table 5 addresses only random labels, not selection-induced optimism, and appears to report a single shuffled run rather than a null distribution over the full pipeline. Therefore, the central predictive claim is not supported by the analysis; what remains is an exploratory association claim, which the SHAP patterns may support as hypotheses but not as validated predictions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a framework based on textual forma mentis networks (TFMNs) to predict three psychopathology factor scores—social maladjustment, specific internalising behaviors, and neurodevelopmental risk—in 232 adolescents from the Healthy Brain Network. Network features and emotion z-scores are extracted from transcripts of emotionally charged interviews and fed into Random Forest and Gradient Boosting regressions, evaluated with four-fold cross-validation. The authors report significant correlations (r = 0.37, 0.33, 0.34) and use SHAP values to interpret feature contributions. The central conclusion is that TFMNs 'can successfully predict psychometric levels of psychopathology in adolescents based on transcripts of emotionally charged interviews.'","tokens_in":20282,"tokens_out":4971,"duration_ms":46862,"significance":"If the predictive claim were supported, the paper would offer a novel, interpretable language-based marker for adolescent mental health, with concrete network features (modularity, core-periphery structure, centrality) and emotion scores linked to distinct psychopathology dimensions. The use of independent parent-report questionnaires as targets is a clear strength, and the SHAP analysis provides falsifiable, feature-level associations. However, the reported evidence does not establish out-of-sample prediction, and the authors explicitly disclaim interest in predicting unseen data. As currently presented, the contribution is best characterized as an exploratory association study, which is substantially weaker than the stated predictive claim.","major_comments":[{"comment":"The Conclusions state that 'TFMNs can successfully predict psychometric levels of psychopathology in adolescents,' but the Limitations section states that cross-validation was used on the whole dataset and that the authors had 'a general lack of interest in predictability of unseen data.' These statements are directly contradictory. Four-fold cross-validation on the same 232 transcripts, with no held-out sample or nested procedure, does not support a claim of successful prediction. This is the central claim of the paper, and it is unsupported by the reported analysis.","section":"Conclusions; Discussion (Limitations)"},{"comment":"Feature selection and hyperparameter tuning are not nested within the cross-validation procedure. The Methods describe features ranked by SHAP and discarded if reshuffling does not deteriorate performance 'during training/validation,' and grid search is used for hyperparameters, but no nested loop or independent validation set is described. If feature selection or hyperparameter selection uses any of the data that also contribute to the cross-validated r values in Table 2, those correlations are optimistically biased. The authors must clarify the exact procedure and, if necessary, re-run the analysis with fully nested cross-validation or an external hold-out set.","section":"Methods - Machine Learning Models; Methods - Explainable AI and feature contribution analysis"},{"comment":"The 'permutation test' reports a single random shuffle of the target variable, not a null distribution over many shuffles. The p-values in Table 5 (e.g., p = 0.25, p = 0.10) are not permutation-based p-values; they appear to be ordinary significance tests applied to one shuffled dataset. A valid permutation test requires repeating the shuffle many times and comparing the observed performance to the resulting null distribution. As reported, this analysis does not support the claim that the observed results are 'unlikely due to chance.'","section":"Results - Table 5"},{"comment":"The syntactic distance parameter k=4 is selected from the cumulative distribution function of syntactic distances in the same interview transcripts. This data-dependent choice, combined with feature selection and hyperparameter tuning on the same corpus, means that the p-values reported alongside the cross-validated correlations are not valid frequentist inferences for generalizable prediction. The authors should either provide an honest out-of-sample evaluation or explicitly reframe the claims as descriptive associations and remove 'predict' language from the title, abstract, and conclusions.","section":"Methods - Transforming texts in textual forma mentis networks"}],"minor_comments":[{"comment":"The abstract and the Results use 'XGBoost' while the Methods consistently refer to a Gradient Boosting Machine (GBM); please standardize the terminology throughout.","section":"Abstract; Results - Machine learning models"},{"comment":"The text states that Ze > 1.96 indicates both over- and under-represented emotions; the under-represented case should be Ze < -1.96.","section":"Methods - Emotional Profiling and Emotion Quantification"},{"comment":"The text contains the typo 'Modalurity' where 'Modularity' is intended.","section":"Results - Explainable AI for Social Maladjustment"},{"comment":"Table 1 mentions 'Reciprocity (RC)' as an excluded feature, but Reciprocity is not defined in the Methods feature list; please either define it or remove the reference.","section":"Results - Network features"}],"recommendation":"reject","confidential_remarks":"The central predictive claim is contradicted by the paper's own limitations section, and the validation strategy cannot support the stated conclusion. This is not a matter of a few missing details; the predictive framing would need to be dropped or the analysis redone with proper nested or held-out evaluation. I recommend rejection. The paper is currently an arXiv preprint under review, and the methodological issues are central rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is worth reading for the SHAP patterns but not for the predictive claim. The abstract and conclusions say TFMNs 'can successfully predict psychometric levels of psychopathology,' but the Methods and Discussion make clear that generalization to unseen data was never attempted. The Limitations section says cross-validation was used on the whole dataset and that there was 'a general lack of interest in predictability of unseen data.' That is an explicit admission that the reported r values (0.37, 0.33, 0.34) are in-sample associations, not predictive performance.\n\nWhat's new: applying the TFMN framework to adolescent clinical interviews from HBN and linking structural features (modularity, core, betweenness, local efficiency) and emotions to p-factor dimensions. The specific SHAP findings—modularity and core-periphery structure associated with social maladjustment, disgust and betweenness with internalizing, lowered local efficiency with neurodevelopmental risk—are not in the prior literature and are plausible, theory-driven hypotheses. The paper is also honest about its limits: it acknowledges the sample size, the lack of nested CV, and the co-referencing issue. That honesty earns credit.\n\nSoft spots, in order. First, the feature selection procedure is described as reshuffling features and discarding those that don't hurt performance 'during training/validation.' If that selection is done on the same folds used to compute the reported correlations, the r values are optimistically biased. The paper does not describe a nested or fully wrapped pipeline. Second, k=4 for syntactic distance is chosen from the same corpus's distance distribution, so it is a data-dependent free parameter, not an independent choice. Third, the permutation test in Table 5 looks like a single shuffled run per model, not a null distribution over the whole pipeline; that makes it weak evidence against selection-induced optimism. Fourth, there is no correction for the many correlations and models tested; the p-values next to r should be read as exploratory.\n\nNone of this kills the descriptive value. If the paper were reframed as 'we identify candidate language-network markers associated with adolescent psychopathology,' the analysis would be a reasonable exploratory study. As published, the predictive framing overreaches.\n\nWho's it for: researchers working on language-based markers in computational psychiatry, especially those using cognitive network features. They should treat the SHAP patterns as hypotheses to test on held-out data.\n\nIt deserves a serious referee—the question is important and the dataset is public and appropriate—but the manuscript needs major revision or a re-scoped claim before it could be accepted.","headline":"Interesting exploratory associations between language-network features and adolescent psychopathology, but the 'prediction' claim is unsupported by an analysis that explicitly avoids out-of-sample testing.","tokens_in":20872,"tokens_out":2097,"would_cite":false,"duration_ms":21388,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Textual forma mentis networks predict adolescent psychopathology levels from emotionally charged interview transcripts.","keywords":["textual forma mentis networks","cognitive network science","psychopathology prediction","adolescent mental health","explainable AI","SHAP","natural language processing","emotion profiling"],"falsifier":"Retrain the same regression pipeline on the 232 interviews and evaluate it on a held-out sample of new adolescent interviews with the same protocol, or on participants excluded from the original selection: if the correlation between predicted and observed factor scores drops to zero or becomes negative, the central claim that TFMN features predict psychopathology is unsupported. A cheaper check is nested cross-validation with feature selection applied only inside each training fold; the r ≈ 0.33–0.37 values would need to survive that re-evaluation to support the predictive reading.","tokens_in":19823,"feed_emoji":"🧠","tokens_out":5932,"duration_ms":56480,"temperature":0.7,"pith_summary":"This paper tries to establish that the structure and emotional tone of an adolescent's spoken language carry measurable information about their psychopathology levels. Using transcripts of 232 short emotion-eliciting interviews, the authors construct textual forma mentis networks (TFMNs), networks in which words are connected when a parser finds syntactic dependencies, synonyms, or emotional associations between them. Network features plus emotion z-scores feed Random Forest and Gradient Boosting regression models, which reach significant Pearson correlations of r = 0.37 for social maladjustment, r = 0.33 for specific internalising behaviours, and r = 0.34 for neurodevelopmental risk. The authors frame the result as evidence about patterns in this dataset rather than as a validated clinical predictor. If the claim holds, language-derived network structure becomes an interpretable window into adolescent mental health.","feed_headline":"Word networks in teen interviews predict distress scores","feed_subtitle":"Structure and emotion in 232 transcripts correlate with three clinical scales at r ≈ 0.33–0.37","key_machinery":"The central object is the textual forma mentis network (TFMN), a cognitive network in which nodes are words or concepts from a transcript and edges connect words that are syntactically dependent within a sentence, extended by synonym links and annotated with emotion scores. Edges are drawn between non-stopwords whose syntactic distance in the dependency parse is at most k = 4, which captures about 94% of short-distance syntactic relations in these transcripts. Network measures such as degree, core structure, clustering, path lengths, centrality, modularity, global and local efficiency, together with emotion z-scores, become the feature set; Random Forest and gradient-boosting regressions model three latent psychopathology factors, and SHAP values, a game-theoretic explanation method, attribute each prediction to individual features.","core_discovery":"Textual forma mentis networks can successfully predict psychometric levels of psychopathology in adolescents based on transcripts of emotionally charged interviews. The central finding is that combinations of semantic/syntactic network features and emotional profiles outperform either alone, and that the model's decisions are interpretable: higher modularity and a pronounced core–periphery structure predict higher social maladjustment; higher betweenness centrality with stronger disgust predicts higher internalising scores; lower local efficiency predicts higher neurodevelopmental risk. A permutation test with shuffled target variables removes the correlations, supporting the claim that real structure, not chance, drives the associations. Because the evaluation uses four-fold cross-validation on the same 232-participant dataset, the paper presents the result as evidence about structure in these data rather than as a validated predictor for new individuals.","pith_inferences":["A direct test would be whether the same k = 4 TFMN features trained on one interview protocol transfer to a different protocol or age band; the paper's own reading suggests this is unknown.","Because the authors state they had no interest in predicting unseen data, the reported r values are likely upper bounds; with feature selection inside the loop, more conservative estimates would probably be lower.","If disgust-driven betweenness is truly a rumination signature, then interventions that shift conversational focus should lower both disgust expression and the betweenness bottleneck in follow-up transcripts.","Since modularity reflects topic separation, the social-maladjustment result could be checked against human topic annotations, linking network modularity to independently rated conversational coherence."],"forward_implications":["The reproducibility of TFMN features makes language a candidate transdiagnostic marker for adolescent mental-health screening.","The SHAP patterns generate concrete, testable hypotheses: tightly clustered conversation topics go with social maladjustment, repeated bridge concepts and disgust go with internalising tendencies, and weakly integrated concepts go with neurodevelopmental risk.","Combined structural and emotional features should be preferred over word-count or emotion-only baselines in future language-based mental-health studies.","The same pipeline can be reapplied to larger interview datasets, where nested cross-validation would test true out-of-sample prediction."],"supporting_citations":[{"why":"Supplies the 232 adolescent interview transcripts and psychometric scores used as data.","marker":"[5]"},{"why":"Provides the p-factor hierarchical model from which the three target factor scores are derived.","marker":"[7]"},{"why":"Defines the transdiagnostic dimensions social maladjustment, internalising, and neurodevelopmental risk used as targets.","marker":"[21]"},{"why":"Provides the EmoAtlas tool that builds TFMNs and computes emotion z-scores from transcripts.","marker":"[39]"},{"why":"Introduces textual forma mentis networks as text-mining reconstructions of associative knowledge.","marker":"[44]"},{"why":"Frames TFMNs within cognitive network science and justifies network features as mental-lexicon proxies.","marker":"[45]"},{"why":"Supplies the word-emotion lexicon underlying the emotional profiling z-scores.","marker":"[28]"},{"why":"Shows that speech-graph fragmentation marks thought disorder in psychosis, the network-marker precedent this study extends.","marker":"[31]"},{"why":"Demonstrates that graph connectivity and NLP markers distinguish psychosis groups, motivating combined structural features.","marker":"[29]"}],"fun_headline_variants":["Teen language networks reveal mental health signs","Word network patterns forecast teen psychopathology","Interview speech maps predict teen distress","AI reads teen talk for distress markers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that correlations found by fitting models to the same 232 interviews, with choices like the syntactic-distance cutoff made from that same data, reflect a real link between language and psychopathology rather than patterns specific to this one group, an assumption the authors explicitly did not test since they say they had no interest in predicting unseen data.","fun_headline_variants_meta":{"raw":{"variants":["Teen language networks reveal mental health signs","Word network patterns forecast teen psychopathology","Interview speech maps predict teen distress","AI reads teen talk for distress markers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1377,"prompt_tokens":997,"completion_tokens":380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":329}},"tokens_in":613,"tokens_out":380,"duration_ms":4062,"temperature":1.0,"reasoning_tokens":329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:44:18.639703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same regression pipeline on the 232 interviews and evaluate it on a held-out sample of new adolescent interviews with the same protocol, or on participants excluded from the original selection: if the correlation between predicted and observed factor scores drops to zero or becomes negative, the central claim that TFMN features predict psychopathology is unsupported. A cheaper check is nested cross-validation with feature selection applied only inside each training fold; the r ≈ 0.33–0.37 values would need to survive that re-evaluation to support the predictive reading.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 232 adolescent interview transcripts and psychometric scores used as data."},{"cited_title":"Caspi, R","cited_arxiv_id":null,"evidence_quote":"Provides the p-factor hierarchical model from which the three target factor scores are derived."},{"cited_title":"Holmes, S","cited_arxiv_id":null,"evidence_quote":"Defines the transdiagnostic dimensions social maladjustment, internalising, and neurodevelopmental risk used as targets."},{"cited_title":"Semeraro, S","cited_arxiv_id":null,"evidence_quote":"Provides the EmoAtlas tool that builds TFMNs and computes emotion z-scores from transcripts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces textual forma mentis networks as text-mining reconstructions of associative knowledge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames TFMNs within cognitive network science and justifies network features as mental-lexicon proxies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the word-emotion lexicon underlying the emotional profiling z-scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that speech-graph fragmentation marks thought disorder in psychosis, the network-marker precedent this study extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates that graph connectivity and NLP markers distinguish psychosis groups, motivating combined structural features."}],"review_version":1}