Measuring Psychological States Through Semantic Projection: A Theory-Driven Approach to Language-Based Assessment

Davide Marocco; Maria Luongo; Nicola Milano

REVIEW 2 major objections 1 minor 2 references

Measuring Psychological States Through Semantic Projection: A Theory-Driven Approach to Language-Based Assessment

T0 review · 2 major / 1 minor · reviewed 2026-05-08 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Semantic projection onto axes from clinical scale items produces psychological scores from text that align with standard clinical measures.

desk verdict Semantic projection onto clinical-scale axes gives an unsupervised scoring method but the validation looks partly circular and under-specified. read the letter →

arxiv 2605.04873 v1 submitted 2026-05-06 cs.CL

Maria Luongo , Davide Marocco , Nicola Milano This is my paper · ORCID

classification cs.CL

keywords semanticprojectionlanguage-basedassessmentpsychologicalstatesdepressionanxietyworryunsupervisedmeasurementclinicalscales

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper develops an unsupervised method that turns natural language responses into scores for depression, anxiety, and worry by projecting sentence embeddings onto semantic axes. Each axis is built directly from lexical anchors and items drawn from existing validated questionnaires, so the resulting scores stay tied to established clinical definitions without any task-specific training data. Strong correlations emerge between these projection scores and participants' questionnaire results, especially when responses come in structured formats such as word selections or short phrases. The approach therefore offers a way to assess psychological states that remains interpretable and does not require retraining models for every new population or context.

What carries the argument

semantic projection of Sentence-BERT embeddings onto axes defined by lexical anchors and items from clinical scales

What would settle it

A replication study in which projection scores show no reliable correlation with participants' scores on the same clinical scales used to construct the axes, or fail split-half reliability checks.

Watch

Extended reading notes

Core claim

Psychological constructs are operationalized as interpretable semantic axes derived from lexical anchors and items taken from validated clinical scales for depression, anxiety, and worry. Participant responses in several formats are embedded with Sentence-BERT and projected onto these axes to yield continuous scores. These scores exhibit strong associations with the original clinical measures, particularly for structured formats, while free-text responses improve when processed at the sentence level rather than as whole documents. The results position semantic projection as a theory-driven, fully unsupervised alternative to supervised language models for psychological assessment.

Load-bearing premise

Lexical anchors and items from validated clinical scales, when placed in Sentence-BERT space, accurately and fully represent the target constructs of depression, anxiety, and worry.

Editorial extensions

If this is right

Structured response formats such as selected words, written words, and phrases produce stronger alignment with clinical measures than whole free-text responses.
Sentence-level aggregation markedly improves results on free-text input compared with document-level projection.
The method supplies continuous, interpretable scores without supervised training on labeled questionnaire data.
Direct comparisons to lexicon-based sentiment tools and distributional checks support its reliability as an assessment technique.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same axis-construction process could be applied to additional psychological constructs simply by substituting the relevant scale items.
Because the axes are built from public clinical scales, the approach might support measurement in settings where administering full questionnaires is impractical.
Multilingual embeddings could allow the same clinical-scale anchors to generate comparable scores across languages without new validation data.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main takeaway is that this paper builds semantic axes directly from items on standard depression, anxiety, and worry scales, embeds text with Sentence-BERT, and projects responses onto those axes to produce scores. It tests the approach on several response formats and reports correlations with the original clinical measures plus split-half reliability and a VADER comparison. That combination of theory-driven axes and projection across formats is the actual novelty; most prior work trains supervised models on questionnaire targets instead. The results look stronger for constrained formats like selected words or phrases than for raw free text, and sentence-level aggregation helps the latter. Those patterns are worth noting for anyone thinking about response design in language-based assessment. The approach is at least trying to stay interpretable and avoid fitting to specific training distributions. The soft spots are more substantial. Because the axes are defined from the same scale items later used for criterion validation, the reported associations could largely reflect surface lexical overlap rather than independent measurement of the constructs. The fact that performance drops in free text is consistent with that possibility. The abstract supplies no effect sizes, no sample details, no exact projection formula, and no controls such as orthogonal axes or external construct-validity checks. Without those pieces it is hard to judge how much new signal is actually being captured. This is the sort of paper that might interest computational psychologists or NLP groups working on mental-health measurement who want to move beyond black-box prediction. A reader could extract the basic projection idea and try it on their own data, but the current evidence is too preliminary to treat as a settled method. I would send it to peer review because the core framing is coherent and the authors are engaging with real limitations of supervised alternatives, even if the execution needs tighter validation and more transparent reporting.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces a theory-driven unsupervised framework for assessing psychological states (depression, anxiety, worry) from text. It defines semantic axes in Sentence-BERT space using lexical anchors and items drawn from validated clinical scales, projects participant responses (across selected words, generated words, phrases, and free text) onto these axes to produce continuous scores, and evaluates them via correlations with the same clinical measures, split-half reliability, attenuation corrections, Wasserstein distributional comparisons, and benchmarks against VADER sentiment analysis. Stronger associations are reported for structured response formats, with improvements from sentence-level aggregation in free text; the work positions semantic projection as an interpretable, scalable alternative to supervised language models.

Significance. If the validity issues can be addressed, the approach would represent a meaningful contribution by offering a fully unsupervised, theory-grounded method that avoids the need for large labeled training sets and provides direct interpretability through explicit semantic axes. The attention to response format and aggregation strategies highlights practical considerations for language-based assessment that are often overlooked.

major comments (2)

[Abstract and Methods] Abstract and implied Methods (axis construction): The semantic axes are defined using items from the same validated clinical scales later employed for criterion validation. This creates a partial circularity risk, as reported correlations may largely reflect lexical/semantic overlap between the input text and scale language rather than independent capture of the underlying constructs. The pattern of stronger results for constrained formats (selected/written words, phrases) versus free text is consistent with surface-level similarity rather than deep construct measurement. No controls (orthogonal axes, independent expert construct-validity ratings, or ablation of anchor-specific terms) are described.
[Abstract and Results] Abstract and Results: No sample size, demographic details, exact mathematical definition of the projection operation (e.g., how anchors are combined into an axis vector), effect sizes with confidence intervals, or statistical controls for confounds (text length, lexical diversity) are provided. These omissions make it impossible to assess the magnitude, robustness, or generalizability of the claimed associations and reliability coefficients.

minor comments (1)

[Abstract] The abstract references attenuation corrections and Wasserstein distance but does not report the numerical outcomes or implementation details for these analyses.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful and constructive review. We address each major comment point by point below, providing the strongest honest defense of the manuscript while indicating where revisions will be incorporated.

read point-by-point responses

Referee: [Abstract and Methods] Abstract and implied Methods (axis construction): The semantic axes are defined using items from the same validated clinical scales later employed for criterion validation. This creates a partial circularity risk, as reported correlations may largely reflect lexical/semantic overlap between the input text and scale language rather than independent capture of the underlying constructs. The pattern of stronger results for constrained formats (selected/written words, phrases) versus free text is consistent with surface-level similarity rather than deep construct measurement. No controls (orthogonal axes, independent expert construct-validity ratings, or ablation of anchor-specific terms) are described.

Authors: We appreciate the referee's identification of this methodological consideration. The axes are constructed from validated scale items precisely to ground the projections in established psychological theory, with participant responses generated independently and embedded via Sentence-BERT to capture semantic rather than purely lexical similarity. The observed pattern across response formats is consistent with our interpretation that more structured prompts better elicit construct-relevant language. However, we acknowledge that lexical overlap remains a plausible alternative explanation. In the revision we will add ablation analyses that remove scale-specific anchor terms, introduce comparisons to orthogonal axes derived from unrelated constructs, and expand the discussion of construct validity versus surface similarity. Expert ratings would require new data collection and are noted as a direction for future work. revision: partial
Referee: [Abstract and Results] Abstract and Results: No sample size, demographic details, exact mathematical definition of the projection operation (e.g., how anchors are combined into an axis vector), effect sizes with confidence intervals, or statistical controls for confounds (text length, lexical diversity) are provided. These omissions make it impossible to assess the magnitude, robustness, or generalizability of the claimed associations and reliability coefficients.

Authors: We regret that these elements were not foregrounded in the abstract. The full Methods section defines the projection as the cosine similarity between a response embedding and the axis vector (the mean Sentence-BERT embedding of the lexical anchors and scale items). Sample size and demographics appear in the Participants subsection. In the revised manuscript we will update the abstract to report sample size, key demographics, and a concise statement of the projection operation. We will also add effect sizes with 95% confidence intervals throughout the Results and include regression models that control for text length and lexical diversity (type-token ratio) as potential confounds. These changes will directly address the concerns about transparency and robustness. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity detected in the derivation chain

full rationale

The paper operationalizes psychological constructs as semantic axes derived from lexical anchors and items drawn from validated clinical scales, embeds participant text via Sentence-BERT, performs geometric projection to obtain scores, and then reports empirical correlations of those scores against the same clinical measures. This sequence contains no self-definitional loop, no fitted parameter renamed as a prediction, and no load-bearing self-citation or uniqueness theorem imported from prior author work. The projection step is a fixed, unsupervised cosine-style operation in embedding space whose output is not mathematically forced to match the validation criteria; any observed association is an empirical result that could in principle be low or null. The method therefore remains self-contained against external benchmarks (the clinical questionnaires) with independent geometric content, satisfying the default expectation of no circularity.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central claim rests on the assumption that Sentence-BERT embeddings and lexical anchors from clinical scales can faithfully represent psychological constructs in a continuous semantic space without additional fitting.

assumptions (2)

domain assumption Sentence-BERT embeddings preserve semantic relationships relevant to psychological constructs from clinical scales
Core to embedding responses and performing projection; invoked throughout the method description.
domain assumption Lexical anchors and scale items define valid, interpretable semantic axes for depression, anxiety, and worry
Directly used to operationalize constructs; stated in the framework introduction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Measuring Psychological States Through Semantic Projection: A Theory-Driven Approach to Language-Based Assessment." pith.science (2026). https://pith.science/paper/2605.04873

@misc{pith2026260504873,
  author       = {Pith},
  title        = {Pith review of: Measuring Psychological States Through Semantic Projection: A Theory-Driven Approach to Language-Based Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2605.04873}},
  note         = {Machine review of arXiv:2605.04873}
}

read the original abstract

Recent advances in natural language processing have enabled increasingly accurate estimation of psychological traits from language. However, most existing approaches rely on supervised models trained to predict questionnaire scores, limiting interpretability and generalizability across contexts. The present study introduces a theory-driven and fully unsupervised framework for measuring psychological states directly from natural language using semantic projection. Psychological constructs were operationalized as interpretable semantic axes derived from lexical anchors and items from validated clinical scales assessing depression, anxiety, and worry. Participants textual responses were embedded using Sentence-BERT and projected onto these axes to generate continuous psychological scores across multiple response formats, including selected words, generated words, phrases, and free-text responses. Projection scores were evaluated through correlations with standardized clinical measures , split-half reliability analyses, attenuation corrections, distributional similarity using Wasserstein distance, and comparisons with lexicon-based sentiment analysis (VADER). Results showed strong associations between projection scores and clinical measures, particularly for structured formats such as selected words, written words, and phrases. Free-text responses produced weaker results when analyzed as whole texts, but performance improved substantially when sentence-level aggregation strategies were applied. These findings support semantic projection as an interpretable and scalable alternative to supervised language models for psychological assessment and highlight the importance of response format and text-processing strategies in language-based mental health measurement.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[1]

Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, 214–223. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora,S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A.,Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon,R., Chatt...

work page doi:10.1037/h0040957 2017
[2]

Kolouri, S., Zou, Y ., & Rohde, G. K. (2016). Sliced Wasserstein kernels for probability distributions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5258–5267. Liu, Y ., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V . (2019). RoBERTa: A robustly optimized BERT pretrai...

work page doi:10.1016/0005-7967(90)90135-6 2016

Pith tools

Reviewed May 8, 2026 · model on record in the stance chip above.

[1] [1]

Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, 214–223. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora,S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A.,Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon,R., Chatt...

work page doi:10.1037/h0040957 2017

[2] [2]

Kolouri, S., Zou, Y ., & Rohde, G. K. (2016). Sliced Wasserstein kernels for probability distributions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5258–5267. Liu, Y ., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V . (2019). RoBERTa: A robustly optimized BERT pretrai...

work page doi:10.1016/0005-7967(90)90135-6 2016