ALIGNS: Unlocking nomological networks in psychological measurement through a large language model

Referee: [Abstract] Abstract and Evaluation sections: The three reported evaluations (PROMIS convergence, temperament re-dimensioning, and expert ratings) are presented at a high level without quantitative accuracy metrics, error analysis, or details on training data exclusion rules. This leaves the central claim that ALIGNS produces valid nomological networks resting on unshown evidence, as no hold-out behavioral datasets or direct comparisons to published correlation matrices from independent samples are described.

Authors: We agree that additional quantitative detail would strengthen the presentation. The manuscript already reports classification accuracy tests for model development, and we will expand the Evaluation sections in revision to include specific accuracy metrics, error analysis for the classification task, and explicit rules for training data exclusion to avoid leakage. For the PROMIS evaluation, we will add a direct comparison to published correlation matrices from independent samples to support the reported convergence. The temperament and expert evaluations are more qualitative in focus, but we will include quantitative elements such as inter-rater agreement statistics from the expert assessment. These changes will make the supporting evidence more explicit without altering the core claims. revision: yes

Referee: [Evaluation sections] System description and Evaluation sections: The nomological links and new indicators are generated via LLM semantic inference from item text. Without explicit incorporation of observed covariances or external empirical criteria, it is unclear how the networks avoid reproducing latent semantic associations in the model rather than discovering or validating empirical relations; the internal nature of all three evaluations does not address this.

Authors: The referee correctly identifies that generation relies on semantic inference from item text after training on validated measures. The training corpus consists of established questionnaires whose items and structures reflect empirical covariances documented in the psychometric literature; the model therefore generalizes from those embedded relations rather than operating on raw semantics alone. We acknowledge this remains an indirect grounding and does not incorporate new observed covariances from fresh samples. In revision we will expand the System description to explicitly discuss this distinction, the reliance on training data, and the resulting limitations. We will also clarify that the three evaluations serve as illustrative applications rather than direct empirical tests, and we will add a dedicated limitations subsection on semantic versus empirical alignment. revision: partial

Referee: [Abstract] Abstract: The assumption that training on validated questionnaire measures suffices to produce accurate, unbiased networks is load-bearing for the validity claim, yet no tests against independent empirical benchmarks (e.g., meta-analytic correlation matrices) are reported to rule out model-specific artifacts or semantic drift.

Authors: We recognize that this assumption is central and that explicit benchmarking against independent data would further support it. The reported classification accuracy tests provide an internal check on the model's ability to recover known structures from the training distribution. To address potential artifacts or drift, we will add in revision direct comparisons of generated networks for anxiety/depression and child temperament against published meta-analytic correlation matrices. These additions will help demonstrate alignment with established empirical patterns and will be accompanied by discussion of possible model-specific biases. revision: yes