pith. sign in

arxiv: 2605.21739 · v2 · pith:REVZTB6Dnew · submitted 2026-05-20 · 💻 cs.AI

AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence

Kate M. Lubrano , Faisal Sayed , Ankita Rathod , Akshansh , Craver Corbyn Thomas-Smith , Mark E. Whiting , Karina Nguyen This is my paper

Pith reviewed 2026-05-22 08:49 UTC · model grok-4.3

classification 💻 cs.AI
keywords AttuneBenchemotional intelligenceLLM benchmarksmulti-turn conversationspreference predictionemotion recognitionresponse quality
0
0 comments X

The pith

Emotionally intelligent LLM behavior decomposes into independent capabilities rather than one unified skill.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces AttuneBench to measure how LLMs handle emotional states across real multi-turn conversations. It collects 200 genuine chats where participants annotate their own emotional state, the model's behavior, and their preferred responses at each turn. When eleven models are scored on emotion recognition, behavioral classification, preference prediction, and response quality, the resulting rankings show little overlap. This matters because it indicates that emotional intelligence in conversation requires distinct skills, particularly the ability to predict what response a specific user wants in context, which single-turn or synthetic tests cannot reveal.

Core claim

Across eleven evaluated models, rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent. This shows that emotionally intelligent behavior decomposes into separable capabilities. Preference alignment and response-quality judgments discriminate models more effectively than emotion-label accuracy. These results indicate that emotionally intelligent behavior requires predicting what kind of response a specific user wants in context, a distinction aggregate scoring obscures and single-turn or synthetic formats cannot capture across turns.

What carries the argument

AttuneBench, a benchmark built from 200 genuine multi-turn human-model conversations in which participants provided turn-by-turn annotations of their emotional state, the model's behavior, and preferred responses.

If this is right

  • Emotionally intelligent responses depend on predicting context-specific user preferences rather than general emotion recognition alone.
  • Aggregate scoring of emotional intelligence can hide important model differences in handling ongoing conversations.
  • Single-turn and synthetic benchmarks are insufficient for assessing how models adapt emotional responses across multiple turns.
  • Diagnosing model-specific strengths and failure modes requires separate evaluation of each capability using real annotated conversations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Future model training may benefit from targeting preference prediction and response alignment as distinct objectives separate from emotion labeling.
  • Extending AttuneBench to more diverse participant groups could test whether these separable capabilities remain independent across cultural or demographic differences.
  • If preference alignment continues to discriminate models strongly, user satisfaction in long-term conversational applications may improve more from better context prediction than from higher emotion recognition accuracy.

Load-bearing premise

Participant-provided turn-by-turn annotations of their own emotional states, the model's behavior, and preferred responses are accurate, consistent, and sufficient to measure genuine emotional intelligence.

What would settle it

Re-annotating the same 200 conversations with independent third-party judges and observing that model rankings across the four tasks become strongly correlated would indicate the separability finding depends on self-annotation rather than reflecting distinct capabilities.

Figures

Figures reproduced from arXiv: 2605.21739 by Akshansh, Ankita Rathod, Craver Corbyn Thomas-Smith, Faisal Sayed, Karina Nguyen, Kate M. Lubrano, Mark E. Whiting.

Figure 1
Figure 1. Figure 1: AttuneBench pipeline. HPs converse with an Original Model and annotate each turn (Phases 1–2); Evaluated Models are scored against the HP-grounded annotations (Phase 3). AttuneBench is designed around three core questions: (1) how accurately can LLMs infer a user’s emo￾tional state across a multi-turn conversation; (2) how well do LLMs select or produce responses that align with human preferences in emotio… view at source ↗
Figure 2
Figure 2. Figure 2: Per-model Composite (left) and Pairwise Accuracy (right). Composite range is narrow (4 pt) but 35/55 model pairs are distinguished; Pairwise spans wider (47/55). Error bars: 95% percentile bootstrap CIs (10,000 resamples; per-conversation, 𝑛 = 200). Dashed line in (b) = chance (1/3). Models compress into a narrow Composite range, but diverge sharply by metric: aggregate scoring obscures statistically signi… view at source ↗
Figure 6
Figure 6. Figure 6: Conversation interface presented to participants. Each session consisted of a single multi-turn conversation between the participant and an LLM on the assigned topic. 22 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Conversation-wide Assessment page of post-conversation annotation interface. Participants rated general per￾formance using the Four Branch Model and answered questions about their overall conversational goals and satisfaction. They were able to reference the conversation transcript while rating [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Binary Judgment page of the post-conversation annotation interface. For selected applicable binary questions, participants recorded both the observed model behavior ("In Actual Response"), as well as their preferred behavior ("Preference"). 23 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Pairwise Ranking page of the post-conversation annotation interface. Considering three different response options, participants selected their preferred response for four pairwise comparison questions, which varied from turn to turn. 24 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Human-baseline pilot accuracy (3 annotators, 7 conversations) compared against EM benchmarks from the full Default Mode evaluation. (a) On Pairwise Accuracy, the strongest annotator (anno 2: 0.722) exceeds the per-conversation best EM mean (0.665). (b) On Binary HP Accuracy, all three annotators sit at or below the per-conversation best EM mean (0.853), within 4 percentage points at most. Annotators brack… view at source ↗
Figure 11
Figure 11. Figure 11: Per-(annotator, conversation) accuracy across the 7 pilot conversations. (a) Pairwise Accuracy and (b) Binary HP Accuracy. Red border marks the self-rating cell: anno 3 was the original HP for conversation df8737a9 (A00002). Their accuracy on that conversation is comparable to their mean elsewhere; see §I.3. 34 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p034_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Composite Score per model, broken out by evaluation mode (Default, Verbose, Omniscient). Most models cluster within a 1–2 point band across modes; the Opus 4.7 Omniscient outlier on Composite is a downstream artifact of the Draft Judge collapse documented in the main paper. Within-family rank flip: Opus 4.6 ↔ Opus 4.7 under Verbose Mode.. On the Default Mode 200- conversation evaluation, Opus 4.6 leads Op… view at source ↗
Figure 13
Figure 13. Figure 13: Verbose – Default difference per model (rows) and metric (columns). Positive (blue) cells indicate Verbose Mode improves on the metric; negative (red) cells indicate degradation. The Mistral Large Pairwise/Composite degradation is the most prominent off-diagonal cell [PITH_FULL_IMAGE:figures/full_fig_p038_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Omniscient – Default difference per model (rows) and metric (columns). The Opus 4.7 Draft Judge collapse appears as the single most extreme cell on the matrix; most other cells are near-zero, supporting the main paper’s conclusion that profile access does not improve performance. 38 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p038_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Composite-rank dotplot across modes (rank 1 = best). Most models retain their rank to within ±1 position; rank shifts of ≥ 2 positions occur for Mistral Large and Sonnet 4.6. J.3 Spearman Correlation Matrix Across Primary Metrics [PITH_FULL_IMAGE:figures/full_fig_p039_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Turn-level metric grid: Emotion F1, Emotion VA, Binary OM/HP Accuracy, Pairwise Accuracy, Kendall 𝜏 (Default Mode) [PITH_FULL_IMAGE:figures/full_fig_p040_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Post-conversation metric grid: PANAS Norm, Q1 Goal Identification, Q2 Emotion Clarity, Q3 Conversational Fit, Q3 Follow-up, Four-Branch (Default Mode). 40 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p040_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Four-Branch MAE per branch (Perceiving, Understanding, Facilitating, Managing) for each model. “Under￾standing” is consistently the hardest branch and “Perceiving” the easiest, a shared pattern across models. 41 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p041_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Binary OM agreement structure: per-model, per-question heatmap of HP-vs-EM binary judgments, Default Mode. Darker cells indicate higher agreement. 42 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p042_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Pairwise win-rate matrix: rows = EM A, columns = EM B, cell = fraction of conversations where EM A’s predicted preference ranking is closer to HP ground truth than EM B’s (Default Mode). The Opus family advantage is visible as dark Opus rows [PITH_FULL_IMAGE:figures/full_fig_p043_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Per-model bars for the four conversation-wide questions (Q1 Goals, Q2 Clarity, Q3 Fit, Q3 Follow-up) used in the main paper Section 6 discussion of holistic comprehension. The Sonnet 4.6 / Opus 4.7 effective-zero on Q3 Follow-up is the rightmost panel. 43 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p043_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Composite Score per model under Verbose Mode evaluation [PITH_FULL_IMAGE:figures/full_fig_p044_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Turn-level metric grid under Verbose Mode. 44 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p044_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Post-conversation metric grid under Verbose Mode. J.6 Omniscient Mode Figure Gallery The Omniscient Mode evaluation provides each EM with the HP psychometric profile at evaluation time on a 25-conversation subsample. The figures support the Omniscient-mode findings discussed in main-paper Section 6 and Section J.2: the Opus 4.7 Draft Judge collapse ( [PITH_FULL_IMAGE:figures/full_fig_p045_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Turn-level metric grid under Omniscient Mode [PITH_FULL_IMAGE:figures/full_fig_p046_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Post-conversation metric grid under Omniscient Mode. 46 Pareto.ai [PITH_FULL_IMAGE:figures/full_fig_p046_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Per-conversation Composite Score (x-axis) plotted against per-turn metric (y-axis), Default Mode. The wide horizontal spread relative to vertical spread illustrates that conversation-level difficulty (Section 6) is the largest single source of variance in the dataset, exceeding model-level variance. K Cost and Runtime [PITH_FULL_IMAGE:figures/full_fig_p047_27.png] view at source ↗
read the original abstract

Emotional intelligence (EI), the ability to perceive, understand, and respond appropriately to others' emotional states, is central to human communication, and increasingly important to assess as LLMs assume conversational roles in everyday life. Existing EI benchmarks rely on synthetic prompts, single-turn cases, or third-party annotation. These approaches do not directly measure how models infer and respond to a participant's emotional state over the course of a real conversation. We introduce AttuneBench, a benchmark grounded in 200 genuine multi-turn human-model conversations in which participants conversed with anonymized LLMs and provided turn-by-turn annotations of their emotional state, the model's behavior, and their preferred responses. Across 11 evaluated models, we find that model rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent, indicating that emotionally intelligent behavior decomposes into separable capabilities. Preference alignment and response-quality judgments are substantially more model-discriminating than emotion-label accuracy. These results indicate that emotionally intelligent behavior requires predicting what kind of response a specific user wants in context, a distinction that aggregate scoring can obscure and that single-turn or synthetic formats cannot directly capture across turns. AttuneBench provides a framework for assessing each of these capabilities and for diagnosing model-specific strengths and failure modes in emotionally salient conversation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces AttuneBench, a benchmark using 200 real multi-turn human-LLM conversations in which participants provide turn-by-turn self-annotations of their emotional state, the model's behavior, and preferred responses. It evaluates 11 models and reports that rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent, indicating that emotionally intelligent behavior decomposes into separable capabilities, with preference alignment and response-quality judgments being more model-discriminating than emotion-label accuracy.

Significance. If the self-annotations can be shown to be reliable ground truth, the result would be significant for LLM evaluation: it would demonstrate that aggregate EI scores obscure distinct sub-capabilities and that context-specific preference prediction is a separable and more discriminative dimension than label accuracy alone. The benchmark supplies a concrete framework for diagnosing model strengths and failure modes in naturalistic, multi-turn emotional conversation.

major comments (2)
  1. [Methods / Dataset Construction] Methods / Dataset Construction section: The manuscript provides no inter-annotator agreement statistics, test-retest consistency checks, or external validation (e.g., third-party raters or physiological correlates) for the participant self-annotations of emotional state, model behavior, and preferred responses. Because the central independence claim rests on these annotations serving as reliable ground truth, the absence of reliability metrics leaves open the possibility that observed ranking independence is an artifact of label noise rather than genuine separability of EI capabilities.
  2. [Results] Results section (rankings and independence analysis): The paper states that rankings are 'largely independent' but does not report quantitative measures (e.g., Spearman rank correlations between the four tasks, or a statistical test for independence). Without these numbers or a pre-specified threshold, it is difficult to assess whether the observed pattern supports the strong claim of separable capabilities or merely reflects moderate correlations that could still be consistent with a single underlying factor.
minor comments (2)
  1. [Abstract / Introduction] The abstract and introduction would benefit from a brief comparison table contrasting AttuneBench with prior EI benchmarks (synthetic, single-turn, third-party) to clarify the precise novelty of the multi-turn self-annotation design.
  2. [Figures] Figure captions and axis labels for the ranking plots should explicitly state the number of conversations per model and any statistical controls applied when computing preference-prediction accuracy.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed and constructive review. We address each major comment below, indicating where we will revise the manuscript to incorporate the feedback and where we provide clarification or additional analysis.

read point-by-point responses

  1. Referee: Methods / Dataset Construction section: The manuscript provides no inter-annotator agreement statistics, test-retest consistency checks, or external validation (e.g., third-party raters or physiological correlates) for the participant self-annotations of emotional state, model behavior, and preferred responses. Because the central independence claim rests on these annotations serving as reliable ground truth, the absence of reliability metrics leaves open the possibility that observed ranking independence is an artifact of label noise rather than genuine separability of EI capabilities.

    Authors: We agree that reliability of the annotations is central to interpreting the results. Because these are self-annotations provided by the participants themselves on their own emotional states, preferred responses, and perceptions of model behavior, conventional inter-annotator agreement among independent raters does not apply; each conversation has only one annotator by design. We did not collect test-retest data or physiological correlates. In the revised manuscript we will add an explicit discussion in the Methods and Limitations sections explaining why participant self-report is the most direct ground truth for subjective emotional experience and preference in this setting, while acknowledging the absence of the additional validation metrics suggested and identifying them as valuable directions for future extensions of the benchmark. revision: partial

  2. Referee: Results section (rankings and independence analysis): The paper states that rankings are 'largely independent' but does not report quantitative measures (e.g., Spearman rank correlations between the four tasks, or a statistical test for independence). Without these numbers or a pre-specified threshold, it is difficult to assess whether the observed pattern supports the strong claim of separable capabilities or merely reflects moderate correlations that could still be consistent with a single underlying factor.

    Authors: We appreciate this observation and agree that quantitative support would strengthen the independence claim. Because we have per-model scores on all four tasks for the 11 evaluated models, we can compute the relevant statistics from the existing data. In the revised Results section we will report Spearman rank correlations between the four task rankings, include a pre-specified threshold for considering rankings 'largely independent,' and add a brief statistical note on the observed pattern. These additions will make the separability argument more precise without altering the original findings. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark with observational findings

full rationale

This is an empirical benchmark paper that collects real multi-turn conversations and participant annotations, then reports model evaluation results as direct observations. The central claim of largely independent rankings across EI capabilities is presented as a finding from the data rather than any derivation, equation, or fitted parameter that reduces to its inputs by construction. No mathematical steps, self-definitional quantities, or load-bearing self-citations appear in the provided text; the methodology is grounded in external data collection and model testing against that data.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper introduces a new annotated conversation dataset and four evaluation axes but does not rely on fitted parameters, unstated axioms, or new postulated entities beyond standard benchmark construction.

pith-pipeline@v0.9.0 · 5780 in / 1116 out tokens · 27124 ms · 2026-05-22T08:49:46.795571+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.