pith. sign in

arxiv: 2605.15734 · v1 · pith:BMJESM6Cnew · submitted 2026-05-15 · 💻 cs.AI

Can We Trust AI-Inferred User States. A Psychometric Framework for Validating the Reliability of Users States Classification by LLMs in Operational Environments

Izabella Krzeminska , Michal Butkiewicz , Ewa Komkowska This is my paper

Pith reviewed 2026-05-20 19:09 UTC · model grok-4.3

classification 💻 cs.AI
keywords user state classificationLLM reliabilitypsychometric validationadaptive systemsmetric stabilityreplication evaluationAI inference
0
0 comments X

The pith

Metrics from LLMs for inferring user states lack stability at the individual level, preventing their use as real-time indicators in adaptive systems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests the assumption that metrics produced by large language models to classify user states remain stable and interpretable when examined one score at a time. It applies replication procedures to 213 metrics generated by three different LLMs and finds that the great majority fail to produce consistent results on repeated trials at the single-instance level. This instability rules out direct use of those scores to drive immediate adaptations in live systems, even though the same metrics often become stable once averaged. The authors also present a replicable evaluation framework that future developers can apply to check whether any given metric meets the necessary threshold before deployment.

Core claim

The study shows that only 31 of 213 metrics met reliability criteria at the level of individual scores across the three LLMs. The lack of stability at this level means such scores cannot be treated as trustworthy indicators of user state for real-time adaptive systems, although the metrics retain value for post-hoc analyses that relate interaction patterns to outcomes such as satisfaction, trust, and engagement. The main contribution is the proposal of a replicable evaluation framework that makes measurable checks of metric applicability possible.

What carries the argument

Replication evaluation procedures that measure repeatability of each metric at both the individual-score level and after aggregation, applied across 213 metrics and three bimodal LLMs.

If this is right

  • Only metrics that pass individual-level reliability tests should be used to drive real-time adaptations.
  • Unstable individual metrics can still support post-hoc identification of rules linking interactions to user experience parameters.
  • Designers of adaptive systems must perform explicit reliability validation rather than assume stability.
  • Ongoing monitoring is required to detect any later violations of reliability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same framework could be applied to test reliability of AI inferences in domains other than user-state classification.
  • Teams might need to redesign prompt strategies or combine multiple models to reach acceptable individual stability.
  • Operational deployments may require periodic re-validation instead of a single upfront check.

Load-bearing premise

That the replication procedures applied to the chosen 213 metrics and three LLMs produce evidence generalizable to reliability questions in typical operational environments.

What would settle it

A replication study on a comparable set of metrics and LLMs that finds the majority of metrics producing consistent individual scores across repeated inferences would falsify the central claim.

Figures

Figures reproduced from arXiv: 2605.15734 by Ewa Komkowska, Izabella Krzeminska, Michal Butkiewicz.

Figure 1
Figure 1. Figure 1: Scheme of Research Procedure 4.1 The Experimental Stage Dataset Composition The stability test was conducted by running ten pipelines with com￾mands to measure respective set of metrics through an identical dataset four times. For the immediate utility of the metric tests, the material used for the tests was the target material, i.e., telephone calls from a contact centre. The test material included audio … view at source ↗
Figure 2
Figure 2. Figure 2: Matrix of metrics - ICC(3,1) single comparison between GPT-4o audio and Gemini 2.0 [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Matrix of metrics - ICC(3,1) single comparison (between GPT-4o audio and Gemini 2.5 [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Matrix of metrics - ICC(3,1) single comparison between Gemini 2.0 Flash and Gemini [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
read the original abstract

The use of large language models to assess user states in conversational and adaptive systems is based on the assumption that the metrics used for such assessment are stable and interpretable at the level of individual scores. This paper empirically tests this assumption, focusing on the psychometric reliability of artificial intelligence (AI) measures of user states. This study employed replication evaluation procedures to assess the repeatability of a broad set of metrics across three different bimodal large language models (GPT-4o audio, Gemini 2.0 Flash, Gemini 2.5 Flash). Analyses include both individual score reliability and aggregated reliability, allowing us to distinguish metrics potentially useful for real-time adaptation from those that retain their value only in aggregated analyses. The results demonstrate that metric reliability cannot be considered a default property in interpretive domains. The lack of stability at the level of individual scores precludes the interpretation of such scores as indicators of user state in real-time adaptive systems, even if these metrics demonstrate stability after aggregation. At the same time, the study indicates that individually unstable metrics can retain analytical utility in post-hoc studies, identifying rules governing interactions and their relationships with user experience parameters such as satisfaction, trust, and engagement. The main contribution of this work, besides quantifying the severity of the problem (only 31 of 213 metrics met the criteria), is the proposal of a replicable evaluation framework, enabling measurable evaluations of metric applicability. This approach supports more responsible AI design of adaptive systems, in which the interpretation of results requires explicit validation of reliability and monitoring for violations over time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript reports an empirical replication study assessing the psychometric reliability of 213 user-state metrics inferred by three bimodal LLMs (GPT-4o audio, Gemini 2.0 Flash, Gemini 2.5 Flash). It distinguishes individual-score stability from aggregated stability, concludes that only 31 metrics meet criteria for real-time adaptive use, and proposes a replicable evaluation framework to validate metric applicability in operational environments.

Significance. If the central findings hold, the work supplies concrete evidence that stability cannot be assumed for LLM-inferred user states and supplies a practical distinction between metrics usable in real-time systems versus those limited to post-hoc analysis. The explicit quantification (31/213) and the proposed validation framework constitute measurable contributions that can guide responsible design of adaptive conversational systems.

major comments (2)
  1. [Evaluation Procedures] Evaluation Procedures section: the replication tests rely on fixed conversation logs and standardized prompts. To support the claim that individual-score instability precludes real-time interpretation, the regime must reproduce sources of fluctuation present in live systems (changing history, user-specific phrasing, multi-turn drift, prompt sensitivity). Without systematic variation in context length or user population, the observed instability may be an artifact of the static test design rather than a general property of LLM-inferred states.
  2. [Results] Results section, paragraph reporting the 31/213 count: the manuscript states that only 31 metrics met the individual-stability criteria, yet provides no explicit reliability thresholds, ICC values, or statistical tests used to classify stability. Without these details it is difficult to judge whether the threshold is conservative enough to justify the strong claim that such scores cannot be used in real-time adaptive systems.
minor comments (2)
  1. [Abstract] Abstract: the phrase 'replication evaluation procedures' is used without a forward reference to the precise protocol (number of repetitions, input variation strategy).
  2. [Methods] The manuscript would benefit from a table summarizing the 213 metrics by category and the exact stability criteria applied to each.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed and constructive comments, which have prompted us to improve the clarity and transparency of our manuscript. Below we respond point by point to the major comments.

read point-by-point responses

  1. Referee: [Evaluation Procedures] Evaluation Procedures section: the replication tests rely on fixed conversation logs and standardized prompts. To support the claim that individual-score instability precludes real-time interpretation, the regime must reproduce sources of fluctuation present in live systems (changing history, user-specific phrasing, multi-turn drift, prompt sensitivity). Without systematic variation in context length or user population, the observed instability may be an artifact of the static test design rather than a general property of LLM-inferred states.

    Authors: We appreciate the referee’s emphasis on ecological validity. The fixed-log design was deliberately chosen to isolate LLM inference stability under replicable, controlled conditions, thereby providing a conservative baseline that can be directly compared across models. Instability observed even in this standardized regime indicates that reliability cannot be presupposed; additional sources of fluctuation in live deployments would be expected to increase rather than decrease variability. We agree that systematic variation of context length, user phrasing, and multi-turn dynamics would further strengthen the claims. Accordingly, we will add a dedicated Limitations subsection in the Discussion that explicitly acknowledges the static-test constraint and outlines a protocol for future dynamic-validation studies. This constitutes a partial revision. revision: partial

  2. Referee: [Results] Results section, paragraph reporting the 31/213 count: the manuscript states that only 31 metrics met the individual-stability criteria, yet provides no explicit reliability thresholds, ICC values, or statistical tests used to classify stability. Without these details it is difficult to judge whether the threshold is conservative enough to justify the strong claim that such scores cannot be used in real-time adaptive systems.

    Authors: We thank the referee for identifying this presentational gap. The Methods section defines the stability criteria (ICC(2,1) > 0.75 together with cross-model consistency), yet these thresholds were not restated with sufficient prominence in the Results paragraph that reports the 31/213 figure. In the revised manuscript we will (a) restate the exact ICC threshold and accompanying statistical decision rules in the Results section, (b) report summary ICC statistics for the retained and discarded metrics, and (c) add a supplementary table listing the 31 qualifying metrics with their ICC values. These additions will allow readers to assess the conservatism of the chosen cut-off directly. revision: yes

Circularity Check

0 steps flagged

Empirical replication study with direct measurements; no derivation chain or self-referential reductions

full rationale

The paper conducts an empirical replication evaluation of 213 metrics across three LLMs using observed repeatability on fixed inputs. No equations, fitted parameters, or derivations are present that reduce claims to inputs by construction. The central finding on individual-score instability is reported as a direct measurement outcome rather than a self-definitional or self-citation-dependent result. The proposed framework is a procedural recommendation grounded in the study's own data collection, with no load-bearing reliance on prior author work that would create circularity. This is a standard honest empirical measurement paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The work relies on standard psychometric notions of reliability without introducing new constructs.

pith-pipeline@v0.9.0 · 5826 in / 1098 out tokens · 33063 ms · 2026-05-20T19:09:39.715881+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · 4 internal anchors

  1. [1]

    On the Opportunities and Risks of Foundation Models

    Amershi, Saleemaet al., (2019). “Guidelines for human-AI interaction”,Proceedings of the 2019 chi conference on human factors in computing systems, pp. 1–13. Barrett, Lisa Feldman (2017).How emotions are made: The secret life of the brain, Pan Macmillan. Bender, Emilyet al., (2021). “On the Dangers of Stochastic Parrots”,FAccT, Binns, Reubenet al., (2018)...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1016/j.specom.2008.12.003 2019

  2. [2]

    Calibrated Trust in Human-AI Collaboration

    Chiou, Elizabeth Ket al., (2021). “Calibrated Trust in Human-AI Collaboration”,arXiv preprint arXiv:2106.05684,available at:https://arxiv.org/abs/2106.05684. Cowen, Alan S. and Keltner, Dacher (2020). “Self-report captures 27 distinct categories of emotion bridged by continuous gradients”,Nature Communications, Vol. 11, pp. 1–10.doi:10.1038/ s41467-019-12...

    work page arXiv 2021

  3. [3]

    Emotion recognition in human-computer interaction

    IEEE, pp. 1989–1992. Cowie, Roddyet al., (2001). “Emotion recognition in human-computer interaction”,IEEE Signal processing magazine, Vol. 18 No. 1, pp. 32–80. Cronbach, Lee Joseph (1972). “The dependability of behavioral measurements”,Theory of gener- alizability for scores and profiles, pp. 1–33. Danescu-Niculescu-Mizil, Cristianet al., (2011). “A Compu...

    work page 1989

  4. [4]

    Consistency and Accuracy of GPT-4 Models on Repeated Mental Health Assessments

    Doe, John and Smith, Jane (2025). “Consistency and Accuracy of GPT-4 Models on Repeated Mental Health Assessments”,Journal of Medical Internet Research, Vol. 27 No. 1, e69910. Dourish, Paul (2001).Where the action is: the foundations of embodied interaction, MIT press. Dziri, Nouhaet al., (2019). “On the Evaluation of Generative Dialogue Models”,NAACL, Ei...

    work page arXiv 2025

  5. [5]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Gehman, Samuelet al., (2020). “Toxicity in Language Models”,Findings of ACL, Ghandeharioun, Asmaet al., (2019). “Emma: An emotion-aware wellbeing chatbot”,2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, pp. 1–7. Giles, Howard, Coupland, Nikolas, and Coupland, Justine (1991).Contexts of Accommodation, Camb...

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  6. [6]

    Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP.arXiv preprint arXiv:2212.14024, 2022

    Kay, Judy (2006). “Scrutable Adaptation: Because We Can and Must”,Adaptive Hypermedia and Adaptive Web-Based Systems. Springer, pp. 11–19. Khattab, Omaret al., (2022). “Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive NLP”,arXiv preprint arXiv:2212.14024, Khattab, Omaret al., (2024). “DSPy: Compiling Declarative ...

    work page arXiv 2006

  7. [7]

    Motivated Closing of the Mind

    Kruglanski, Arie W. and Webster, Donna M. (1996). “Motivated Closing of the Mind”,Psycholog- ical Review,doi:10.1037/0033-295X.103.2.263. Krzeminska, Izabella and Rzeznik, Jakub (2021). “Personality-Based Lexical Differences in Services Adaptation Process.”Technium, Vol. 3 No

    work page doi:10.1037/0033-295x.103.2.263 1996

  8. [8]

    Multimodal Recognition of Users States at Human-AI Interaction Adaptation

    Krzemi´ nska, Izabella (2025). “Multimodal Recognition of Users States at Human-AI Interaction Adaptation.”Technium, Vol

    work page 2025

  9. [9]

    I adjust, therefore I trust? Language accommodation and trust in conversational AI

    Lee, Hee, Kim, Yoojung, and Sundar, S. Shyam (2022). “I adjust, therefore I trust? Language accommodation and trust in conversational AI”,Computers in Human Behavior, Vol. 127, p. 107045.doi:10.1016/j.chb.2021.107045. Lee, Minaet al., (2022). “Linguistic Accommodation in Human–AI Interaction”,PNAS,doi: 10.1073/pnas.2117415119. Liao, Q.et al., (2021). “Cog...

    work page doi:10.1016/j.chb.2021.107045 2022

  10. [10]

    GPT-4o System Card

    Mrkˇ si´ c, Nikolaet al., (2017). “Neural Belief Tracking”,ACL, 26 Neuberg, Stevenet al., (1997). “Need for Closure and Processing”,Journal of Personality and Social Psychology, Nunnally, Jum C (1975). “Psychometric theory—25 years ago and now”,Educational Researcher, Vol. 4 No. 10, pp. 7–21. O’Brien, Heather L and Toms, Elaine G (2008a). “What is user en...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1002/asi.20801 2017

  11. [11]

    A Circumplex Model of Affect

    — (1980). “A Circumplex Model of Affect”,Journal of Personality and Social Psychology, Vol. 39 No. 6, pp. 1161–1178.doi:10.1037/h0077714. Sacks, Harvey, Schegloff, Emanuel A, and Jefferson, Gail (1974). “A simplest systematics for the organization of turn-taking for conversation”,language, Vol. 50 No. 4, pp. 696–735. Scherer, Klaus R (2009). “The dynamic ...

    work page doi:10.1037/h0077714 1980

  12. [12]

    Recognising Realistic Emotions and Affect in Speech and Text

    Schuller, Bj¨ ornet al., (2011). “Recognising Realistic Emotions and Affect in Speech and Text”, International Conference on Affective Computing.doi:10.1109/ACII.2011.5951547. Sheng, Emilyet al., (2019). “The Woman Worked as a Babysitter: On Biases in LMs”,EMNLP, Shrout, Patrick E and Fleiss, Joseph L (1979). “Intraclass correlations: uses in assessing ra...

    work page doi:10.1109/acii.2011.5951547 2011

  13. [13]

    Ethical and social risks of harm from Language Models

    Wang, Yanet al., (2022). “A systematic review on affective computing: Emotion models, databases, and recent advances”,Information Fusion, Vol. 83, pp. 19–52. Waytz, Adam, Heafner, Jeremy, and Epley, Nicholas (2014). “The mind in the machine: An- thropomorphism increases trust in an autonomous vehicle”,Journal of Experimental Social Psychology, Vol. 52, pp...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1016/j.jesp.2014.01.005 2022

  14. [14]

    Emotion recognition from multiple modalities: Fundamentals and methodologies

    Zhao, Sichenget al., (2021). “Emotion recognition from multiple modalities: Fundamentals and methodologies”,IEEE Signal Processing Magazine, Vol. 38 No. 6, pp. 59–73. 28 Table XIV: ICC3 and ICC3K values for all models across pipelines and metrics. Cell colours indicate reliability: Excellent (0.90-1.0), Good (0.75-<0.90), Moderate (0.50-<0.75), Poor (<0.5...

    work page 2021