{"id":"b3629e3e-f159-4a08-9075-aa450edd51f8","arxiv_id":"2504.16271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A first exploratory NLP study of attachment classification from psychotherapy transcripts reports below-baseline accuracy for single models and above-baseline accuracy only for a five-model majority vote.","lead":"This paper tests whether language models can guess a psychotherapy patient's attachment style from transcripts, using labels from the Patient Attachment Coding System. The best single models average about 60 percent accuracy, below the 62 percent baseline of predicting the most common class, so clinical utility is not yet demonstrated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 'almost 60%' accuracy is below the 61.80% majority baseline in the 150-word test set; the only above-baseline result is an untested five-model majority vote on 12 sessions, so the feasibility claim is not established.","rationale":"The reader's CONDITIONAL verdict already rests on the same weakness: the best single model is below the majority baseline and the only above-baseline result is an untested ensemble on 12 test sessions. My stress-test makes this concrete: with an effective sample of 12 sessions, the 5.6-point advantage of the majority vote is within sampling noise, and the paper provides no significance test. This is a correctness risk, not an internal inconsistency, because the authors acknowledge the baseline in Section 6.2 and frame the study as preliminary. I therefore do not move the verdict; the paper remains a useful task definition and failure analysis. If the permutation test fails, the abstract's claim of 'almost 60%' should be downgraded to 'below majority baseline' or the paper should be reframed as a negative result. A secondary, separate concern is the length/verbosity confound suggested by Figure 1, but it is secondary to the baseline problem.","tokens_in":14244,"tokens_out":9043,"duration_ms":97655,"concrete_test":"Run a session-level permutation test for the 150-word condition: fix the five trained models and the turn-level test instances, but randomly permute the attachment labels among the 12 test sessions, moving each session's turns together. Recompute the majority-vote accuracy for 10,000 permutations to obtain the null distribution; the one-sided p-value is the fraction of permutations with accuracy at or above 67.42%. If p > 0.05, the above-baseline result is not significant and the feasibility claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that attachment assessment is feasible at 'almost 60%' accuracy collapses against the appropriate baseline. In the 150-word test setting (Section 5.2, Table 2), the majority class 'preoccupied' constitutes 61.80% of test instances. The five RoBERTa-large models average 59.55% accuracy, below that trivial baseline. The only above-baseline figure is the 67.42% majority-vote accuracy, which is an ensemble of the same five models evaluated on 12 held-out sessions with no significance test. Because all turns from a session share the same attachment label, the effective sample size is 12 sessions, not the number of turn-level test instances; a 5.6-point improvement over the majority baseline on 12 exchangeable units is well within sampling noise. The abstract's 'almost 60%' therefore misstates the evidence: the individual models are at or below chance relative to the test prior, and the ensemble result is not shown to be reliable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an exploratory study on automatically classifying patient attachment style (avoidant, secure, preoccupied) from psychotherapy transcripts. The authors extract patient speech turns from 78 therapy sessions annotated with the Patient Attachment Coding System (PACS), label each turn with the source document's attachment label, and train several RoBERTa-based classifiers. They report a mean test accuracy of 59.55% for their best setup (RoBERTa-large, minimum input length 150 words), with a majority vote across five models reaching 67.42%. They also analyze the effect of input length, domain-adaptive pre-training, and discuss clinical and ethical implications.","tokens_in":14433,"tokens_out":6085,"duration_ms":53463,"significance":"If the reported classification accuracy were reliable, the paper would represent a valuable first step toward automating attachment assessment, with potential implications for personalized psychotherapy and research. However, the central evidence is not statistically established: the individual models do not exceed the majority-class baseline, and the only above-baseline result is an ensemble with no significance testing and an effective sample size of 12 sessions. The paper does provide a useful exploratory analysis and a thoughtful discussion of error patterns and clinical considerations, and it honestly reports that domain adaptation did not help.","major_comments":[{"comment":"The mean test accuracy of the five RoBERTa-large models is 59.55%, which is below the 61.80% majority-class baseline that the authors themselves report in Section 6.2; individual models therefore do not outperform a trivial majority classifier. The only above-baseline result is the majority vote (67.42%), but this is a post-hoc ensemble with no confidence interval or significance test, and with only 12 test sessions the difference is not shown to be reliable. Please provide a session-level significance test (e.g., permutation or bootstrap) and report confidence intervals.","section":"Section 5.2, Table 2"},{"comment":"All 7,255 speech turns are labeled with the document-level PACS label, and the evaluation treats each turn as an independent instance. Because all turns from a session share the same label and are likely correlated, the effective sample size is 12 test sessions, not the number of turns, which inflates the apparent reliability of the accuracy estimates. Please report a session-level evaluation (e.g., majority vote within each session or a mixed-effects model) and discuss the implications for the conclusions.","section":"Section 3.1"},{"comment":"The conclusion that increasing input length improves performance is based on cross-validation accuracy across different thresholds, but no statistical significance is reported, and the class distribution changes with concatenation (at minimum length 150, the test set majority becomes 61.80%). The improvement may be driven by a more skewed class prior rather than better classification. Please compare each length condition against the corresponding majority baseline and test the trend with appropriate error bars.","section":"Section 4.3 and Table 1"},{"comment":"The statement 'Our best results achieve an average accuracy of almost 60%' is presented without the crucial context that this accuracy is below the majority-class baseline of the test set. This framing overstates the evidence and should be corrected by reporting the baseline comparison and the ensemble result with appropriate caveats.","section":"Abstract and Introduction"}],"minor_comments":[{"comment":"The description of the split says 'stratification according to class balance,' but the test set appears highly skewed toward preoccupied (61.80%); please clarify the stratification procedure and report the per-class session counts in train and test.","section":"Section 3.1"},{"comment":"The table header says 'Std dev' but the text refers to 'relative standard deviation'; please clarify which quantity is reported.","section":"Section 5.1"},{"comment":"There is a typo in the sentence 'most of the models do not rely only on the majority label that they ignore the avoidant and secure labels' ('that' should likely be 'and').","section":"Section 6.2"},{"comment":"The error bars represent standard deviation, but confidence intervals would be more informative given the small test set.","section":"Figure 3"},{"comment":"The limitation section does not address the statistical non-independence of turns or the fact that individual models do not beat the majority baseline; please add a discussion of these points.","section":"Limitations"},{"comment":"The paper does not report the number of test instances for each minimum length condition; please add this to Table 1 or the text.","section":"Table 1 / Section 5"}],"recommendation":"reject","confidential_remarks":"The paper's abstract and introduction overstate the evidence, and the central feasibility claim is not supported by the current evaluation. The statistical issues (pseudo-replication, lack of significance testing, small effective sample) are fundamental and require a substantially different analysis to be convincing. The work might be better positioned as a negative result or a dataset/benchmark contribution if the data were made available and the analysis were redone at the session level."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThis is a genuinely exploratory paper with a useful new task definition and an honest failure analysis. The authors are the first, to my knowledge, to try automatic PACS attachment classification from therapy transcripts, and they do it on the only PACS-labeled corpus available. The clinical motivation is well argued, and the discussion of which confusions matter most for treatment is thoughtful and specific. The writing is clear, and they even report a failed generative-model experiment without hiding it.\n\nThe problem is the central quantitative claim. In the 150-word test condition, the majority class 'preoccupied' covers 61.80% of instances, and the five RoBERTa-large models average 59.55% accuracy — below that trivial baseline. The only above-baseline number, 67.42%, comes from a majority vote of those same five models, evaluated on 12 held-out sessions with no significance test. Given the per-split accuracies range from 51.69% to 68.54%, that ensemble result is likely to be noise. The abstract's 'almost 60%' is technically true as an average but misstates the evidence: individual models are at or below chance relative to the test prior.\n\nThere's also a deeper evaluation issue. Every turn is labeled with its source session's attachment class, so the model may be latching onto stylometric or topic signals rather than attachment-relevant discourse. The authors note that secure patients produce shorter turns; their concatenation procedure then changes the class distribution, so the reported length effect is confounded. The document-level split prevents direct leakage, but with only 12 test sessions the effective sample size is far smaller than the turn count suggests.\n\nThese are the usual soft spots for a small exploratory study, and the authors acknowledge many of them. Still, the paper should be read as a task definition and failure analysis, not as evidence that automatic attachment assessment works.\n\nWho is this for? Clinical NLP and psychotherapy researchers who want a baseline for a new task and a sensible discussion of mislabeling costs. It deserves peer review — the task is real, the reporting is mostly transparent, and the below-baseline result is worth putting on the record. A serious referee should ask for clustered evaluation, significance testing against the majority baseline, and a less confident abstract.","headline":"A well-motivated first attempt at PACS attachment classification, but the headline accuracy is below the majority baseline and the only above-baseline result is an untested ensemble on 12 sessions, so the feasibility claim is not established.","tokens_in":14945,"tokens_out":3504,"would_cite":true,"duration_ms":33489,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper demonstrates that patient attachment style — the relational pattern at the center of PACS-based psychotherapy assessment — can be recovered from therapy transcripts by NLP classifiers, with the best configuration reaching a mean…","keywords":["attachment style detection","psychotherapy transcripts","Patient Attachment Coding System","natural language processing","RoBERTa","speech turn classification","domain-adaptive pre-training"],"falsifier":"A decisive check: train the same RoBERTa-large 150-word setup on the same turns but with session-level attachment labels randomly permuted across training documents; if test accuracy stays clearly above chance, the reported signal is a confound, not attachment. The paper's own wide split-to-split spread (51.69% to 68.54%) already warns that the result is sensitive to which patients land in the test fold.","tokens_in":14046,"feed_emoji":"💬","tokens_out":8493,"duration_ms":70846,"temperature":0.7,"pith_summary":"The paper tries to establish that attachment style — a clinically important patient characteristic normally rated from full sessions by trained coders — is detectable in the words patients speak in psychotherapy, using standard language models. It treats each patient speech turn (or a concatenation of consecutive turns up to a minimum length) as an independent instance labeled with the PACS attachment classification of its source session, and trains RoBERTa-based classifiers on the only available PACS-annotated corpus. The best setup, RoBERTa-large with a 150-word minimum input length, reaches a mean test accuracy of 59.55%, and a majority vote over five such models reaches 67.42%; longer inputs generally improve accuracy. The authors present this as a first exploratory step, with the immediate consequence that larger annotated corpora and longer-context models are the natural next investment.","feed_headline":"Language models sort therapy talk by attachment style at ~60 percent","feed_subtitle":"A first test shows attachment styles are partly readable in patient speech; longer turns improve accuracy.","key_machinery":"The mechanism that carries the argument is the minimum-input-length ablation: consecutive patient turns from the same session are concatenated until they reach a threshold — 0, 50, 100, 150, or 250 words — and each resulting window is labeled with the PACS attachment style of its source document. The classifiers are RoBERTa-base, RoBERTa-large, and MentalRoBERTa, optionally further pre-trained on unlabeled counseling transcripts; the best-performing configuration uses RoBERTa-large at the 150-word threshold. By comparing accuracy across thresholds, the paper tests whether attachment is expressed in discourse-level patterns rather than in isolated short utterances; the upward trend in accuracy with longer input is the load-bearing evidence for that claim.","core_discovery":"On the paper's own terms, the discovery is that automatic turn-level classification can distinguish the three PACS attachment categories — avoidant, secure, and preoccupied — from patient speech alone, and does so better when short turns are merged into longer passages. In the test condition, each instance is a patient speech window of at least 150 words labeled with the attachment style of the whole session; the five fine-tuned RoBERTa-large models average 59.55% accuracy, the majority vote reaches 67.42%, and the confusion matrices show a persistent overprediction of 'preoccupied.' The paper interprets these results as evidence that attachment-relevant signal is present in patient language, while acknowledging that the class skew, the small corpus, and the variance across splits keep the finding from being deployment-ready.","pith_inferences":["A model that sees only patient turns cannot exploit turn-taking itself; extending the same ablation to therapist-patient exchanges might sharpen attachment detection, since PACS codes how patients manage emotional proximity with the therapist.","The document-level labeling assumption implies that turn accuracy may overstate the model's grasp of attachment: an analysis of which turns the model gets right, or an attention-based feature study, could reveal whether it keys on content topics rather than relational language.","The majority-vote gain the paper reports suggests the five training splits learn different, partly complementary signals; ensembling across more splits or across model architectures is a cheap extension that could be tested immediately.","A direct next experiment is to measure accuracy per therapist and per therapy modality (Brief Relational Therapy vs. CBT), since the corpus mixes both and the paper's document-level split may not balance them."],"forward_implications":["If the central claim holds, automatic attachment screening is feasible enough to justify building larger PACS-annotated corpora, which the paper identifies as the main bottleneck.","Longer input windows — up to entire session transcripts, and possibly including therapist speech — become the obvious next step, since accuracy rose as the minimum input length grew.","Repeated automated measurement of attachment throughout treatment becomes practical, whereas manual PACS annotation takes about 90 minutes per session and needs roughly 30 hours of coder training.","Because the models overpredict 'preoccupied' and miss most 'avoidant' patients, any applied use would require calibration against the clinical cost of each confusion — especially the preoccupied/avoidant swap, which the paper argues is the most damaging.","The one generative attempt (Llama 3.1 70B at a 150-word minimum) scored only 29% accuracy, suggesting that fine-tuned discriminative encoders, not out-of-the-box generative models, are the near-term route."],"supporting_citations":[{"why":"Supplies the PACS coding system and the English validation transcripts that form the labeled task corpus.","marker":"Talia et al., 2017"},{"why":"Supplies the RoBERTa-base and RoBERTa-large encoders used for classification.","marker":"Liu et al., 2019"},{"why":"Supplies MentalRoBERTa, the mental-health-adapted encoder compared in the experiments.","marker":"Ji et al., 2022"},{"why":"Supplies the domain-adaptive pre-training method tested on top of the encoders.","marker":"Gururangan et al., 2020"},{"why":"Supplies the MaChAmp training framework used to run the classification models.","marker":"van der Goot et al., 2021"}],"fun_headline_variants":["AI reads attachment style in therapy transcripts at ~60%","First test: AI sorts therapy talk by attachment style","NLP model spots attachment style in patient speech alone","Attachment style leaks from talk: AI detects it at ~60%","Ensemble voting lifts attachment-style accuracy to 67%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that every patient speech turn carries the attachment style of the whole session, so each turn can be judged as an independent instance; if attachment is expressed only across longer discourse patterns, or if the model latches onto speaker identity, topic, or session content rather than attachment-relevant language, the reported accuracy does not measure what it claims.","fun_headline_variants_meta":{"raw":{"variants":["AI reads attachment style in therapy transcripts at ~60%","First test: AI sorts therapy talk by attachment style","NLP model spots attachment style in patient speech alone","Attachment style leaks from talk: AI detects it at ~60%","Ensemble voting lifts attachment-style accuracy to 67%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000775,"raw_usage":{"total_tokens":3382,"prompt_tokens":850,"completion_tokens":2532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2451}},"tokens_in":466,"tokens_out":2532,"duration_ms":18699,"temperature":1.0,"reasoning_tokens":2451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:07:10.463173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check: train the same RoBERTa-large 150-word setup on the same turns but with session-level attachment labels randomly permuted across training documents; if test accuracy stays clearly above chance, the reported signal is a confound, not attachment. The paper's own wide split-to-split spread (51.69% to 68.54%) already warns that the result is sensitive to which patients land in the test fold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PACS coding system and the English validation transcripts that form the labeled task corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the domain-adaptive pre-training method tested on top of the encoders."}],"review_version":1}