{"id":"5efc92bb-a03b-4180-b701-2c46f6ce4b13","arxiv_id":"2609.01832","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Depression symptom categories are separable in Gemma-3-27B-PT layer-21 residual stream, and semantic projection onto those axes preserves clinician-annotated rank ordering on held-out text.","lead":"The paper extracts directions inside a large language model's internal activations that correspond to depression symptom groups, and shows these directions can rank held-out depressive text by clinician-annotated symptom category. It also finds a single direction that separates depressive narratives from happy-moment texts about 79% of the time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Held-out symptom ordering depends on labels generated by the same two psychiatrists and DSM-5 groupings used to build the Symptom Vectors; the mood axis has no externally labeled held-out samples.","rationale":"The paper is a good-faith effort with real internal-consistency evidence: the PERMDISP gate, agreement of three raw distance metrics at layer 21, in-sample rank ordering, and the authors' explicit limitation statements all support a plausible mechanism. I considered whether the more serious weakness is the layer-selection loop or the HappyDB valence-versus-genre confound; both are real, but the label non-independence is the single most load-bearing issue because §3.2 is the paper's core external-validation result, and the mood axis has no externally labeled held-out data at all. The Naturalistic corpus is held out as text instances, but the labels that define the 'correct' rank order come from the same two psychiatrists applying the same DSM-5 groupings used to construct the Symptom Vectors. Re-running the analysis on externally labeled ReDSM5 passages and on a fresh independent annotation of the book excerpts would settle whether the shared-annotation channel is doing the work. The Depression Vector AUC of 0.789 is also affected by genre and text-length differences, but it is framed as a gate rather than the main symptom-vector claim. I therefore keep the reader's CONDITIONAL verdict: these are addressable validation gaps, not demonstrated internal inconsistencies.","tokens_in":19099,"tokens_out":9978,"duration_ms":93505,"concrete_test":"Re-analyze the §3.2 rank-ordering under label sources that are independent of the axis-construction annotations. Concretely: (1) recompute the rank ordering and Dunn comparisons on the ReDSM5 subset only, using the original licensed-psychologist nine-symptom labels mapped to the three groups, and report the somatic and suicidality axes separately by source; (2) have a different clinician team, blind to the study's three-group collapse, annotate a random sample of at least 50 Darkness Visible and Handbook excerpts at the nine-item level, map those labels to the three groups, and recompute the mood-axis ordering. If the matching group is no longer first on any axis, especially mood, the shared-annotation channel is implicated; if the ordering survives, the held-out generalization claim is materially strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central generalization evidence in §3.2 is a rank-ordering check: the Naturalistic group carrying a given annotation scores highest on the corresponding Symptom Vector. The labels and the vectors are not independent. In Methods §2.1.1, the same two psychiatrist authors assigned both the Core Clinical excerpts, from which the Symptom Vectors are built via Eq. 6, and the Darkness Visible and Handbook of Depression excerpts to the same three symptom groups using the same DSM-5 A1–A9 item groupings. ReDSM5 is the only subset with externally produced labels, but those nine-symptom labels are programmatically remapped under the same item assignment, and Table 1 shows the ReDSM5 subset contains no mood passages. Consequently, the mood-axis rank ordering, the first axis claimed in the abstract, rests exclusively on book excerpts annotated by the same two psychiatrists who defined the axes. Unless the annotators' within-DSM grouping choices are shown to match an independent clinician consensus, the held-out result cannot distinguish \"residual stream encodes clinician consensus\" from \"residual stream encodes this pair of annotators' grouping.\" This is not a contradiction of the claim, but it is the load-bearing gap between the current evidence and the claim of clinician-aligned generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper asks whether the internal activations of an unmodified pretrained LLM contain clinician-aligned, symptom-level depression signals. Using residual stream activations of Gemma-3-27B-PT on 51 symptom-description excerpts drawn from six validated clinical instruments, the authors select layer 21 as the operating layer by maximizing PERMANOVA pseudo-F subject to a PERMDISP eligibility gate across eight distance-metric by normalization combinations. They construct three Symptom Vectors as group centroids at that layer, project held-out naturalistic text (Darkness Visible, Handbook of Depression, and ReDSM5) onto them with a Gram-pseudoinverse decorrelation, and report that the clinician-annotated group for each axis attains the highest median coefficient on its matching axis (Table 4). They then define a Depression Vector as the difference between the mean of the three Symptom Vectors and a positive-affect centroid and show that it separates 423 held-out Naturalistic excerpts from 423 HappyDB entries with AUC = 0.789 (§3.3). The central claims are that symptom categories are most separable at layer 21, that per-symptom projection coefficients preserve clinician-annotated rank ordering on text not used in vector construction, and that a single depression direction can serve as a valence gate.","tokens_in":19453,"tokens_out":18267,"duration_ms":153953,"significance":"If the held-out effects survive the additional controls requested below, this would be a genuinely useful result for clinical NLP and mechanistic interpretability: a no-fine-tuning readout of a clinically motivated symptom taxonomy directly from residual stream activations, with an explicit distinction between in-sample consistency and held-out generalization. The methodological strengths are real: the PERMDISP eligibility gate prevents misreading dispersion differences as centroid shifts; the convergence of three raw distance metrics on layer 21 is a nontrivial robustness observation; the Core Clinical results are honestly labeled as in-sample checks; the Gram-pseudoinverse projection and the two-stage valence-gate design are clearly specified; code and part of the corpora are released; and the Discussion explicitly refuses to equate annotation agreement with clinical validity. The load-bearing weaknesses are the label independence of the held-out mood axis, the absence of matched neutral and negative controls for the Depression Vector, the in-sample selection of the operating layer, and source-level clustering in the statistical tests.","major_comments":[{"comment":"The held-out mood-axis result is not supported by externally labeled data. Table 1 shows that ReDSM5 contributes no mood passages, so all 141 held-out mood examples come from Darkness Visible and the Handbook of Depression, which were annotated by the same two psychiatrist authors using the same DSM-5 A1–A9 item groupings that define the Symptom Vector axes (§2.1.1). The mood axis is therefore the first axis claimed in the abstract and the one whose held-out rank ordering cannot distinguish “residual stream encodes clinician consensus” from “residual stream encodes this pair of annotators’ grouping choices.” The within-axis median ordering in Table 4 does hold as reported, but the mood result is also the weakest in effect size: the held-out mood-group median on the mood axis is negative (−0.08, versus +0.77 in-sample), and on the suicidality axis the non-matching mood and somatic groups score almost as high (0.85 and 0.91) as the matching group (1.40). Please report the rank ordering and separation statistics separately for passages with externally sourced labels (the ReDSM5 somatic and suicidality subsets) versus author-annotated book excerpts, and obtain either independent third-party annotations of the book excerpts or an externally labeled corpus with mood items; without this, the clinician-aligned generalization claim for the mood axis is not established.","section":"§2.1.1, Table 4"},{"comment":"The Depression Vector test compares two corpora that differ in emotional valence but also in genre, source, and register: Naturalistic is book-length clinical and memoir prose plus Reddit posts, while HappyDB is short crowdsourced happy-moment sentences, and the absolute cosine differences are tiny (median 0.229 versus 0.218 on an overall 0.18–0.25 band). The authors themselves note that the two HappyDB outliers with the highest Depression Scores are neutral texts (“Their sore throat cleared up”; “They went shopping”), which shows the score fires on non-depressive content and strengthens the need for a matched control. Please add a matched neutral corpus and a non-depressive negative-valenced control (for example, anxiety-themed text) of comparable length and register to both Naturalistic and HappyDB, and re-report the AUC with those controls; as it stands, the “emotional valence gate” interpretation of AUC = 0.789 is confounded with genre and text-length differences.","section":"§2.4, §3.3"},{"comment":"The operating layer ℓ* = 21 is selected by maximizing PERMANOVA pseudo-F on the same 51 Core Clinical excerpts from which the Symptom Vectors are then built, and no independent portion of the data is used to confirm the layer choice. The agreement of three raw distance metrics at layer 21 is suggestive, but these metrics are computed on the same passages and are not independent tests. Please provide a stability analysis, for example by selecting the layer on random subsets of the Core Clinical corpus, or by reporting the held-out rank ordering and the Naturalistic-versus-HappyDB AUC across a range of layers around 21; this would show whether “layer 21” is a reproducible property of the model or an artifact of in-sample selection.","section":"§2.2.3"},{"comment":"The inferential statistics treat the 423 held-out passages as independent samples, but the passages are heavily clustered: 172 excerpts come from a single handbook and 57 from a single memoir, and the three annotation groups have different source compositions (mood is 100% book-sourced, suicidality is about 75% ReDSM5, somatic is about 62% ReDSM5). The Dunn post-hoc and Mann–Whitney tests therefore overstate significance, and the between-group median differences in Table 4 could partly reflect source identity rather than symptom content. Please report source-level medians and cluster-robust or source-aggregated tests, and re-check the Table 4 rank ordering under permutations that respect the source structure.","section":"§3.2, Fig. 3"}],"minor_comments":[{"comment":"The paper motivates the Moore–Penrose pseudoinverse by the near-singularity of the Gram matrix (condition number 14,611.6), but for a full-rank 3×3 matrix G+ equals G−1 unless a truncation tolerance is applied; please report the tolerance (for example, the rcond parameter) actually used, and if no truncation was applied, state explicitly that the projection uses the ordinary inverse and justify the “regularization” language.","section":"§2.3, Eq. (6)"},{"comment":"The data availability statement says the rewritten Core Clinical and Positive Affect corpora are released but does not state whether the rewritten Naturalistic corpus (the held-out evaluation set) is available; please state its availability status or the copyright basis for withholding it, since this corpus carries the main generalization claims.","section":"Data availability"},{"comment":"The balancing procedure is described only in general terms, and a reader cannot tell from Table 1 why ReDSM5 contributes zero mood passages; because the 33 Darkness Visible and 108 Handbook mood excerpts already reach the 141 target, no ReDSM5 mood posts were needed, and the paper should say so explicitly to avoid the appearance of a data gap.","section":"§2.1.1, Table 1"},{"comment":"The Depression centroid in Eq. (7) is the unweighted mean of the three Symptom Vectors, while the Positive Affect centroid in Eq. (8) and the scored excerpt centroid in Eq. (10) are token-weighted means; since the Depression Vector and the resulting scores inherit this asymmetry, the choice should be stated explicitly and its effect on the reported cosine values discussed.","section":"§2.4, Eqs. (7)–(10)"},{"comment":"The construction of the Positive Affect corpus is described only by genre examples and a final count of nine excerpts; since this corpus defines the positive endpoint of the Depression Vector (Eq. (8)), the selection procedure (how many candidate texts were considered, who chose them, and why nine) should be specified.","section":"§2.1.1"},{"comment":"Because the same third-person rewrite prompt was applied to the corpora that define the Symptom Vectors and the Depression Vector and to the held-out Naturalistic corpus, the evaluation establishes generalization only for the rewritten third-person register; the Discussion lists this as limitation three, but the abstract and introduction should make explicit that the validated input channel requires the same standardization step.","section":"§2.1.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper fits the journal's scope well, and the authors are unusually candid about limitations. The main risk is not that the results are fabricated but that the headline generalization (“clinician-aligned symptom signal on held-out text”) is stronger than the label independence supports: the mood axis in the held-out set is labeled by the same two authors who defined the axes, and the Depression Vector test lacks matched neutral and negative controls. I am requesting the additional analyses in the major comments rather than rejecting because the somatic and suicidality axes do have externally sourced ReDSM5 labels, the held-out text was genuinely excluded from axis construction, the AUC = 0.789 is real, and the code appears to be released for verification. I would also ask the authors, during revision, to double-check the licensing statements for the withheld corpora and to clarify exactly which rewritten corpora are downloadable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper adapts semantic projection to LLM residual streams and shows that layer-21 activations of Gemma-3-27B-PT encode symptom categories (mood, somatic, suicidality) in a way that preserves clinician-annotated rank order on held-out text. That is a solid subfield contribution for clinical NLP and mechanistic interpretability, and it is executed carefully: the Gram-pseudoinverse decorrelation handles collinearity well, the pipeline is clearly described, and the limitations section is unusually candid.\n\nWhat is actually new is mapping a multi-symptom clinical taxonomy onto decorrelated per-symptom axes in an LLM's internal activations without fine-tuning. The held-out rank-ordering in Table 4 is real evidence, and the AUC=0.789 on depression vs. happy text is a useful sanity check.\n\nThe stress-test concern lands: the held-out Naturalistic labels for Darkness Visible and Handbook of Depression were assigned by the same two psychiatrists using the same DSM-5 groupings that define the Symptom Vectors. The mood axis has no externally labeled held-out samples—ReDSM5 supplies only somatic and suicidality. This means the generalization claim is weaker than advertised; the result is better described as showing internal consistency plus a partially independent held-out check, not fully independent validation. The operating layer was also chosen by in-sample maximization of PERMANOVA pseudo-F on the same corpus used to build the vectors, which introduces some risk of overfitting to that layer. And the Depression Vector contrast confounds valence with genre and text length. These are specific, addressable weaknesses rather than fatal flaws—the authors acknowledge most of them explicitly.\n\nWho this is for: researchers working on clinical NLP, mechanistic interpretability of sentiment/emotion in LLMs, or applications of semantic projection. A reader in those areas will find the methodology adaptable and the limitations instructive. It deserves a serious referee, and with better characterization of the label-dependence issue (e.g., leave-one-annotator-out or ReDSM5-only analysis) it would be a clean paper.\n\nI would bring it to reading group and cite it if I worked in this area.","headline":"Plausible extension of semantic projection to LLM residual streams for depression symptoms, with clear limitations on label independence that the paper itself acknowledges.","tokens_in":19847,"tokens_out":1285,"would_cite":true,"duration_ms":13517,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A pre-trained large language model's internal activations encode clinician-aligned depression symptoms, recoverable without fine-tuning.","keywords":["depression","large language models","mechanistic interpretability","symptom vectors","semantic projection","residual stream","clinical natural language processing","mental health"],"falsifier":"Applying the same Symptom Vector construction and projection procedure to a different LLM family (e.g., LLaMA-3-70B) and a new held-out corpus with independent clinician annotations (not involving the same annotators) should reproduce the rank ordering (each group highest on its own axis); failure to do so would indicate the result is model-specific or annotation-confounded.","tokens_in":20451,"feed_emoji":"🧠","tokens_out":2665,"duration_ms":25052,"temperature":0.7,"pith_summary":"This paper claims that the internal residual stream of an unmodified large language model (Gemma-3-27B-PT) contains distinct, decorrelated signals for three groups of depression symptoms (mood, somatic, suicidality) that align with clinician judgments. The authors construct Symptom Vectors from clinical instrument descriptions at layer 21, where the three categories are maximally separable. When held-out naturalistic narratives are projected onto these vectors, the resulting coefficients preserve the clinician-annotated rank ordering of symptom categories. Additionally, a single Depression Vector derived from the same layer distinguishes depressive from non-depressive text with an AUC of 0.789. If correct, this provides a mechanistic, interpretable method for extracting symptom-level information from patient speech without requiring model fine-tuning or generated text.","feed_headline":"Depression symptoms read directly from an LLM's internal activations","feed_subtitle":"Without fine-tuning, layer-21 projections of Gemma-3-27B-PT preserve clinician-annotated symptom rankings and separate depressive from…","key_machinery":"The central object is the set of three Symptom Vectors (mood, somatic, suicidality) formed from class centroids of residual stream activations at layer 21, along with a Depression Vector defined as their centroid minus a positive affect centroid. The argument is carried by the Gram-pseudoinverse decorrelation procedure, which resolves the near-collinearity (condition number ~14,600) of the clinical category centroids into stable, interpretable projection coefficients that isolate each symptom's unique contribution.","core_discovery":"The residual stream activations of Gemma-3-27B-PT at transformer layer 21 contain linear directions (Symptom Vectors) corresponding to mood, somatic, and suicidality symptom groups, constructed from six validated clinical instruments. These vectors, decorrelated via a Gram-pseudoinverse correction, yield projection coefficients on held-out naturalistic text that preserve the same rank ordering as independent clinician annotation: each symptom group attains the highest median coefficient on its own axis. Moreover, a Depression Vector defined as the difference between the depression centroid and a positive affect centroid separates held-out depressive narratives from happy-moment descriptions (AUC=0.789), acting as a valence gate for subsequent symptom projection.","pith_inferences":["The approach likely extends to other mental health conditions (e.g., anxiety, PTSD) if similar clinically validated symptom descriptions are available to construct basis vectors.","The strong collinearity of symptom centroids (condition number 14,600) may indicate a fundamental representational overlap in the model's internal space, reflecting the real-world comorbidity of depression symptoms; this is handled by the pseudoinverse but limits resolution.","The single-model, single-layer finding should be tested across architectures (LLaMA, GPT) and scales; failure to replicate would suggest the effect is model-specific rather than general to transformer LMs."],"forward_implications":["No fine-tuning or generated text is needed: symptom-level signals are directly readable from a pre-trained LLM's internal activations.","The two-stage pipeline (valence gate via Depression Vector, then symptom projection) could be applied to patient speech for continuous, per-symptom monitoring in research settings.","Clinician-aligned symptom coefficients provide a more granular alternative to traditional severity sum scores, preserving dimensional profiles.","The methodology offers a template for extracting other clinical constructs from LLM internal representations using validated instrument descriptions."],"supporting_citations":[{"why":"Semantic projection framework extended here from static word embeddings to transformer residual stream activations.","marker":"[32]"},{"why":"Prior evidence that sentiment is linearly represented in LLM residual streams, motivating the search for symptom directions.","marker":"[7]"},{"why":"Demonstrates emotion-specific representations in LLaMA models, supporting the plausibility of symptom-level encoding.","marker":"[8]"},{"why":"PERMANOVA method used for quantifying separability of symptom groups in the residual stream.","marker":"[29]"},{"why":"PERMDISP test used as eligibility gate to ensure pseudo-F reflects centroid displacement rather than dispersion differences.","marker":"[30]"}],"fun_headline_variants":["LLM's layer 21 holds clinician-aligned depression symptom vectors","Depression symptom vectors emerge in Gemma layer 21, no fine-tuning","Internal LLM activations rank depression symptoms like clinicians do","Layer-21 vectors separate depressive speech with AUC 0.789","Mechanistic map: Gemma encodes mood, somatic, suicidality axes"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The held-out Naturalistic corpus was annotated by the same two psychiatrist authors using the same DSM-5 item groupings that define the Symptom Vector axes, and every excerpt was rewritten into third-person prose by another large language model; if the annotation definitions or the rewrite bias both axes and held-out labels in the same direction, the observed rank ordering overstates true generalization.","fun_headline_variants_meta":{"raw":{"variants":["LLM's layer 21 holds clinician-aligned depression symptom vectors","Depression symptom vectors emerge in Gemma layer 21, no fine-tuning","Internal LLM activations rank depression symptoms like clinicians do","Layer-21 vectors separate depressive speech with AUC 0.789","Mechanistic map: Gemma encodes mood, somatic, suicidality axes"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"cost":0.00022582,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00022582,"upstream_inference_prompt_cost":0.0001519,"upstream_inference_completions_cost":7.392e-05},"total_tokens":1349,"prompt_tokens":1085,"completion_tokens":264,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":184}},"tokens_in":1085,"tokens_out":264,"cost_source":"openrouter_usage","duration_ms":3737,"temperature":1.0,"reasoning_tokens":184,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-08T22:43:38.713824+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"falsifier":"Applying the same Symptom Vector construction and projection procedure to a different LLM family (e.g., LLaMA-3-70B) and a new held-out corpus with independent clinician annotations (not involving the same annotators) should reproduce the rank ordering (each group highest on its own axis); failure to do so would indicate the result is model-specific or annotation-confounded.","supporting_citations":[{"cited_title":"Semantic projection recovers rich human knowledge of multiple object features from word embeddings , volume =","cited_arxiv_id":null,"evidence_quote":"Semantic projection framework extended here from static word embeddings to transformer residual stream activations."},{"cited_title":"and Geiger, Atticus and Nanda, Neel","cited_arxiv_id":null,"evidence_quote":"Prior evidence that sentiment is linearly represented in LLM residual streams, motivating the search for symptom directions."},{"cited_title":"LL a MA s Have Feelings Too: Unveiling Sentiment and Emotion Representations in LL a MA Models Through Probing","cited_arxiv_id":null,"evidence_quote":"Demonstrates emotion-specific representations in LLaMA models, supporting the plausibility of symptom-level encoding."},{"cited_title":", title =","cited_arxiv_id":null,"evidence_quote":"PERMANOVA method used for quantifying separability of symptom groups in the residual stream."},{"cited_title":"Generalized Hierarchical Multivariate CAR Models for Areal Data","cited_arxiv_id":null,"evidence_quote":"PERMDISP test used as eligibility gate to ensure pseudo-F reflects centroid displacement rather than dispersion differences."}],"review_version":1}