Pith. sign in

REVIEW 5 major objections 4 minor 71 references

Frontier chat models are statistically indistinguishable on empathy but separate by up to 9x on tracking user intent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:14 UTC pith:GTYRKZVP

load-bearing objection A reusable multi-turn evaluation framework with a genuinely useful design idea, but the headline objective findings rest on same-family labels and judging, and the abstract overstates what the data support. the 5 major comments →

arxiv 2607.10428 v2 pith:GTYRKZVP submitted 2026-07-11 cs.CL

Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

Jinglan Gong , Jiefan Lu , Hewei Guo , Kehan Li , Zhiyuan Han , Jihang Jiang , Wenwen Tong , Lewei Lu This is my paper
classification cs.CL
keywords multi-turn dialogue evaluationuser simulationintent trackingLLM-as-judgefinal-intent completion ratepersona formatreasoning phase transitionwarm-up effect
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that the cap that still separates modern chat models is not how warm or human they sound, but how well they hold onto a user's evolving intent across a long conversation. To make that case, it builds a benchmark in which a persona-grounded simulator, the model being tested, and an independent judge are three decoupled roles, and runs 3,400 dialogues through 17 models. Its headline finding is a split: on subjective scores — empathy, persona alignment, naturalness — closed and open models cluster in a narrow band, while on objective measures of latent-intent tracking the same models spread by up to about nine times. A second finding is that enabling reasoning is a near phase transition for objective tracking on long free-text personas, while leaving subjective scores flat. The paper also shows that persona format is a first-order design choice: switching from a structured demographic profile to a long conversational paragraph can flip which model comes out on top.

Core claim

On its own terms, the paper establishes that multi-turn dialogue quality is not one skill but at least two. Perceived interaction quality — empathy, persona alignment, anthropomorphic naturalness — has nearly converged across frontier models: almost every model lands in a narrow band. But objective trajectory tracking — whether the model can predict the user's latent intent turn by turn and eventually steer the conversation to the user's stated final goal — separates the same models by up to about nine times. The paper also claims this separation is driven by reasoning: turning on chain-of-thought in an open-source family lifts latent-intent accuracy from roughly 0.15 to 0.77 on long free-te

What carries the argument

The load-bearing mechanism is the decoupled three-party protocol: a persona-grounded user simulator that emits gold labels (explicit intent, latent intent, emotion, and a self-reported final-intent progress) for every turn; the target model that must first predict those labels and then generate a response; and an independent judge that scores subjective dimensions and adjudicates final-intent completion. The central object is the ChatSEED — a structured starting point bundling persona, topic, initial emotion, explicit/latent intent, and a single final-intent sentence describing the user's goal — around which each 10-turn dialogue is generated. From this come two trajectory-level metrics: int

Load-bearing premise

The benchmark's gold labels and primary judge both come from the same model family, so without human validation the 9x objective gap and the reasoning phase transition could be artifacts of that family's labeling behavior rather than real properties of the target models.

What would settle it

Take a stratified sample of the long-context dialogues and have human annotators (or a second, independent model family) label explicit intent, latent intent, and final-intent completion. If the human-labeled objective spread is materially smaller than 9x, or if the cross-family judge reverses the FICR ordering, the central claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Subjective Likert scores should no longer be used as the primary separator of frontier chat models; objective trajectory metrics carry the signal.
  • Evaluating on long free-text personas rather than structured demographic profiles is necessary to expose differences in intent tracking and goal completion.
  • Enabling reasoning in open-source models is a direct, large lever for latent-intent accuracy on long contexts, while costing little on perceived quality.
  • FICR is a cross-judge-stable scalar and a candidate anchor for cross-paper comparison of goal completion.
  • Persona source and format should be reported as an experimental variable, since it can flip model rankings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the objective gap generalizes beyond the benchmark's label source, the frontier race may be re-framed: after style and empathy saturate, the remaining differentiator is long-horizon goal coherence, not surface naturalness.
  • The reasoning phase transition suggests a testable cost-saving extension: apply reasoning selectively at intent-shift moments rather than every turn, and measure whether the accuracy gain is preserved.
  • The persona-format flip is consistent with label leakage from structured demographics; a direct test would strip demographic attributes from personas and see whether the objective gap reopens.
  • The warm-up effect (early turns scored lower) implies practical system design should front-load rapport-building, and evaluation protocols should avoid unweighted late-turn-only scores.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces EYT-Bench, a multi-turn dialogue benchmark built around three decoupled components: a persona-grounded user simulator, a target model that first predicts user intent/emotion labels and then generates a response, and an LLM judge. Experiments cover 17 target models x 2 persona pools x 100 dialogues (3,400 dialogues) and report four headline findings: (i) closed and open models are statistically indistinguishable on subjective dimensions but separate by up to 9x on objective intent-tracking; (ii) reasoning produces a phase transition for objective tracking on long-context PersonaMem-v2 personas; (iii) persona format strongly affects trajectory spread and FICR; (iv) a warm-up effect appears in 16/17 models. The paper also introduces trajectory-level metrics (intent drift and FICR) and a cross-judge ablation.

Significance. If the findings hold, the benchmark would be a useful contribution: it uses two human-curated persona pools, treats persona format as an experimental variable, introduces trajectory-level objective metrics, and includes useful ablations (alpha sensitivity, cross-judge calibration). The large scale (3,400 dialogues) and the public release of code, pools, and prompts make it reproducible in principle. However, the central claims currently outrun the evidence: the 'statistically indistinguishable' statement is not backed by significance tests; the 'decoupled' design is contradicted by the simulator and primary judge sharing the same base model; and the objective gold labels are generated by the simulator itself with no independent validation. These issues are load-bearing because findings (i)-(iii) rest on the objective metrics.

major comments (5)
  1. [Abstract; §4.4; Table 4; Limitations] The abstract and §4.10 claim that closed and open models are 'statistically indistinguishable' on subjective dimensions, but no significance tests, confidence intervals, or effect-size statistics are reported. N=100 per pool is large enough for meaningful CIs, and the paper's own Limitations state that differences smaller than ~0.05 on subjective scales 'should not be interpreted as significant.' The authors need to show that the observed band differences are within that threshold for all relevant pairwise comparisons. Also, the 'up to 9x' in the abstract is a ratio of accuracies near 0.08 vs 0.80; a ratio of small proportions is not a meaningful effect size and should be replaced by absolute differences with uncertainty.
  2. [§1, §3.1, §4.1, Limitations] The paper's central design claim is that the simulator, target, and judge are 'required by config to be disjoint at the model-family level' to eliminate self-preference (Panickssery et al. 2024). Yet §4.1 states that both the simulator and the primary judge are gemini-3.1-pro-preview-thinking, and the Limitations explicitly admit this sharing. The cross-judge ablation (Table 14) swaps only the judge on 46 dialogues and does not remedy the primary-results violation. The decoupling claim, as stated in §1 and §3.1, is therefore unsupported by the actual configuration.
  3. [§3.3, Algorithm 1, §3.5.1, §3.5.2, §L] The objective metrics rest on labels generated by the simulator itself: Algorithm 1 line 7 has the simulator emit per-turn gold labels and final-intent progress, and FICR is adjudicated by the judge, which shares the simulator's base model. There is no human validation of the objective labels (the human pilot in §L covers only subjective dimensions and has an acknowledged anchoring risk), and no cross-simulator ablation is provided. Consequently, per-turn accuracy and FICR measure alignment with Gemini-3.1-Pro's labeling distribution, not with any independent ground truth. The 9x objective gap and the reasoning phase transition could be artifacts of label-source alignment rather than properties of the target models.
  4. [Figure 8 vs Table 11; §3.5.3 vs Figure 5] There is a direct internal inconsistency in the objective taxonomy. The perception prompt in Figure 8 lists only 10 emotion labels (Anxious, Frustrated, Helpless, Angry, Ashamed, Guilty, Expectant, Relieved, Empathetic, Neutral), whereas Table 11 and §3.5.1 specify a 15-class emotion taxonomy. Similarly, Figure 8 lists 7 latent-intent classes, while Table 11 lists 8 (missing Identity Exploration). If the target models were prompted with the shorter lists, the reported emotion and latent-intent accuracies are computed against a label space the models were not asked to use. In addition, §3.5.3 describes a 3-point Likert scale {0,0.5,1}, but the rubric prompts in Figure 5 score each sub-indicator as 0/1 and sum to integer 0-5. These inconsistencies must be resolved for the objective metrics to be valid and reproducible.
  5. [Abstract; §4.7; Table 7; Limitations] The abstract's finding (ii) states that 'reasoning is a phase transition for objective tracking on long-context personas.' The only paired thinking-on/off evidence is within the Gemma-4 family (Table 7), and the Limitations explicitly restrict the 'quantitative phase-transition claim' to that family. The abstract and §4.10 generalize beyond the evidence. At minimum, the claim should be qualified as 'for the Gemma-4 family' in the abstract, or the authors should add paired reasoning ablations in at least one additional model family.
minor comments (4)
  1. [Limitations vs Table 14 caption] The Limitations state the cross-judge ablation was computed on a 'stratified 42-pair subsample,' but Table 14 and §4.9 say n=46 paired dialogues. Please reconcile the number.
  2. [§4.8, Table 8] The main text reports warm-up magnitudes of -0.01 to -0.74, while Table 8 reports values like -8.2 to -14.9 on a 0-100 scale. The text appears to mix raw 0-5 units with rescaled 0-100 values. Make the unit conversion explicit.
  3. [Appendix H/J and Table 9/14] The table numbering is inconsistent: the cross-judge ablation appears as Table 9 in one place and as Table 14 in another. Please renumber all appendix tables consistently.
  4. [§3.5.1, Figure 8] The perception prompt in Figure 8 contains a detailed chain-of-thought description, but the paper does not specify whether the target model receives this exact prompt for all models or whether the JSON output format is enforced for all APIs. Clarify reproducibility of the perception stage across heterogeneous APIs.

Circularity Check

2 steps flagged

Objective metrics are, by construction, agreement with a Gemini-generated label source; FICR is the same family's self-adjudicated goal verdict, so the central objective separation and completion findings are partially self-referential absent independent validation.

specific steps
  1. self definitional [§3.5.1 'Objective metrics' + §4.1 'Simulator and primary judge' + Limitations]
    "Per turn we compare the simulator-emitted gold labels against the target's predictions for explicit intent (12 classes), latent intent (8 classes), and emotion (15 classes...). ... Both the simulator and the primary judge are gemini-3.1-pro-preview-thinking ... the simulator and primary judge share the Gemini-3.1-Pro-Thinking base; Table 14 bounds the residual self-preference at ≤0.66 Likert points but does not eliminate it."

    The paper's flagship objective finding — up to 9× separation on 'objective intent-tracking' — is computed as agreement with labels emitted by one simulator family, with no independent human or second-family validation of those labels. 'Objective intent-tracking' is therefore defined as 'agreement with gemini-3.1-pro's label distribution'; a high score means the target shares Gemini's priors, not that it independently tracks a human-centered intent. The 9× gap and the reasoning phase transition are, by construction, distances to that same label source and do not independently establish a property of the target models.

  2. self definitional [§3.5.2 'Trajectory metrics' (FICR) + Limitations]
    "After the final turn the judge is asked, conditioned on the full transcript and the ChatSEED's final-intent text, to decide whether the assistant has meaningfully helped the user reach the stated goal, and to assign a 1–5 satisfaction score. FICR is the resulting completion rate... Fourth, FICR is adjudicated by an LLM, not by humans — a 200-dialogue multi-annotator study (Section L) is scheduled."

    The same Gemini-family model that generates the ChatSEED and plays the user simulator also adjudicates whether the final intent was reached. FICR has no external world-state check (unlike τ-bench's database verification); it is simply the judge's own verdict about a goal the same model family authored. Thus 'final-intent completion rate' is defined by the judge's decision, and the reported FICR saturation/spread findings are properties of Gemini's self-consistency rather than independently verified goal completion.

full rationale

Most of the pipeline is not circular: the perception/generation decoupling, warm-up weighting, α-sweep, and cross-judge calibration are self-contained experimental choices. The circularity is concentrated on the ground-truth side. The objective labels and final-intent adjudication both come from the gemini-3.1-pro family, while §1 and §3.1 promise family-level disjointness; §4.1 and the Limitations acknowledge this. The cross-judge ablation (n=46 paired dialogues) and the 59-turn-cell human pilot do not validate the objective labels — the pilot is subjective-only, shows judge scores to annotators, and has an acknowledged anchoring risk. The headline findings (i)–(iii) therefore reduce, at least partially, to properties of a self-generated reference: the 9× objective gap, the reasoning phase transition, and the FICR spread are measured against Gemini-authored labels and Gemini adjudication. Because the paper explicitly discloses these limitations and the rankings are not literally fitted parameters renamed as predictions, the score is 6 rather than 8 or 10.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 4 invented entities

The central claims rest on LLM-generated labels and adjudication, a hand-chosen evaluation configuration (α, T_max, early-stop k, taxonomies, embedding choice), and the unverified assumption that a same-family simulator/judge pair provides unbiased ground truth. Most of these are disclosed in the Limitations, but they are load-bearing for the headline findings.

free parameters (9)
  • Warm-up weight α = 0.10 (main); swept over 0.05, 0.15
    Hand-chosen early-turn weight in Eq. (4); main results use α=0.10. The sweep shows no rank flips, so subjective findings are not sensitive, but weighted scores in Tables 4/5/8 are defined by this choice.
  • Early-stop threshold k = 2
    Dialogue early-stops after two consecutive 'achieved' simulator reports (Algorithm 1); affects dialogue length distribution and FICR.
  • Turn budget T_max = 10
    Middle-of-the-road choice justified by τ-bench, MultiChallenge, MULTI-Bench; affects trajectory length and drift metric.
  • Persona dedup cosine threshold = 0.85
    Sampling choice on all-MiniLM-L6-v2 embeddings; affects pool content and diversity.
  • Number of ChatSEEDs per pool = 100
    Sample size per pool; the authors caution that objective differences below ~0.03 are not significant.
  • Emotion taxonomy size = 15 classes (7 negative, 6 positive, 2 neutral)
    Author-defined expansion from 10 labels; directly sets the difficulty of emotion-accuracy metrics.
  • Intent taxonomy sizes = 12 explicit / 8 latent
    Author-defined label sets; per-turn accuracy figures depend entirely on this schema.
  • Intent-drift encoder = unnamed sentence-transformer
    Eq. (1) defines drift via v(·) but never names the embedding model; drift values depend on this choice.
  • Random seed = 20260521
    Used throughout for sampling and dialogue generation; ensures deterministic reruns only if the cache is released.
axioms (6)
  • domain assumption LLM judges produce scores that approximate human judgments
    §3.5.3 and §4.9 rely on Gemini-3.1-Pro-Thinking for all subjective scores and FICR; the human pilot covers 59 turn-cells with anchoring risk, and no human validation exists for objective labels.
  • domain assumption The user simulator generates realistic, unbiased user behavior rather than template fallbacks
    Algorithm 1 relies on the simulator to emit natural user turns and all gold labels; the paper cites prior 'polite simulator' concerns but provides no human naturalness check.
  • domain assumption Model-family disjointness removes self-preference
    §1 claims simulator/judge/target are disjoint at the model-family level, but §4.1 shows simulator and primary judge share the gemini-3.1-pro base; the assumption is not satisfied in the main configuration.
  • domain assumption The two persona pools are comparable facets of persona realism and the 100-ChatSEED samples are representative
    §3.2 samples 500 records from Nemotron-Personas-USA and PersonaMem-v2 and treats pool as an experimental variable; the 'first 100' ChatSEEDs are not described as a stratified sample from the pools.
  • standard math Embedding cosine similarity reflects semantic intent alignment
    Eq. (1) defines intent drift via cosine distance; the specific encoder is unnamed, and no validation that embedding distances track human judgments of intent shift.
  • domain assumption The warm-up weighting scheme from Gooding and Grefenstette applies to LLM-judged scores
    Eq. (4) weighting is borrowed from prior work; the observed warm-up is in the same direction as the judge's own position lenience, which the paper documents in Section L.
invented entities (4)
  • Latent-intent taxonomy (8 classes) no independent evidence
    purpose: Defines the objective label space behind the 9x separation finding
    Constructed label set (Emotional Resonance, Seeking Validation, etc.); labels are Gemini-generated with no human agreement study or external benchmark anchor.
  • Final-intent completion rate (FICR) no independent evidence
    purpose: Trajectory-level goal-completion metric and headline objective signal
    Judge-adjudicated completion of a simulator-defined final intent; no human multi-annotator study yet (Limitations: 'a 200-dialogue multi-annotator study is scheduled').
  • Intent-drift metric no independent evidence
    purpose: Trajectory-level alignment signal in the spirit of EMPA
    Defined in Eq. (1) via an unnamed sentence-transformer; no external validation of the embedding's semantic sensitivity.
  • Warm-up effect no independent evidence
    purpose: Named finding claimed in 16 of 17 models
    Early-minus-late score differences; confounded with the judge's own position lenience documented in Section L.

pith-pipeline@v1.3.0-alltime-deepseek · 24313 in / 18240 out tokens · 191950 ms · 2026-08-02T07:14:43.483714+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging." pith.science (2026). https://pith.science/paper/GTYRKZVP

@misc{pith2026260710428,
  author       = {Pith},
  title        = {Pith review of: Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GTYRKZVP}},
  note         = {Machine review of arXiv:2607.10428}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Evaluating large language models (LLMs) as multi-turn conversational partners requires probing capabilities that single-turn benchmarks miss: persona consistency, evolving intent tracking, emotional dynamics, and goal completion across many turns. We introduce EYT-Bench, a human-centered benchmark whose evaluation protocol is built around a decoupled three-party design: a persona-grounded user simulator, a target model evaluated on both intent perception and response generation, and an independent, configurable ensemble of LLM judges. Across 3,400 dialogues with 17 target models, EYT-Bench reveals four findings that previous benchmarks miss: (i) state-of-the-art closed and open-source models are statistically indistinguishable on subjective dimensions, but separate by up to 9x on objective intent-tracking; (ii) reasoning is a phase transition for objective tracking on long-context personas but is essentially flat on subjective scores; (iii) persona format strongly affects trajectory spread, FICR (final-intent completion rate) saturates above 0.95 on Nemotron-USA but ranges from 0.53 to 0.88 on PersonaMem-v2; and (iv) the warm-up effect is observed in 16 of 17 models.

Figures

Figures reproduced from arXiv: 2607.10428 by Hewei Guo, Jiefan Lu, Jihang Jiang, Jinglan Gong, Kehan Li, Lewei Lu, Wenwen Tong, Zhiyuan Han.

Figure 1
Figure 1. Figure 1: EYT-Bench framework. ⃝1 generates a persona-conditioned ChatSEED (persona, topic, initial emotion, explicit / latent / final intent). ⃝2 The target model predicts user labels (perception stage) and generates a response (generation stage) from independent prompts. ⃝3 An LLM user simulator emits the next turn with per-turn final-intent progress. ⃝4 A third-party judge (single or multi-judge ensemble) scores … view at source ↗
Figure 1
Figure 1. Figure 1: EYT-Bench framework. ⃝1 generates a persona-conditioned ChatSEED (persona, topic, initial emotion, explicit / latent / final intent). ⃝2 The target model predicts user labels (perception stage) and generates a response (generation stage) from independent prompts. ⃝3 An LLM user simulator emits the next turn with per-turn final-intent progress. ⃝4 A third-party judge (single or multi-judge ensemble) scores … view at source ↗
Figure 2
Figure 2. Figure 2: ChatSEED examples. Each sample bundles topic, persona, initial emotion, first query, composite explicit / latent intent and final intent into a single struc￾tured record so dialogue generation is deterministic con￾ditional on the random seed. sponse) inside the target model is critical: when both prompts are merged, the prediction rubric leaks into the response distribution and inflates the subjective inte… view at source ↗
Figure 3
Figure 3. Figure 3: Three subjective dimensions, each decomposed into five sub-indicators scored on a 3-point Likert [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Persona, topic and emotion statistics across [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Judge rubric prompts for the three subjective dimensions: Empathy (left), Persona Alignment (centre), [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: System prompt for the target dialogue model, [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Perception-stage prediction prompt: the target [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 22 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2510.13747 , year=

    Interactiveomni: A unified omni-modal model for audio-visual multi-turn dialogue , author=. arXiv preprint arXiv:2510.13747 , year=

    arXiv

  2. [2]

    arXiv preprint arXiv:2606.25325 , year=

    Omni-Perception Policy Optimization for Multimodal Emotion Reasoning , author=. arXiv preprint arXiv:2606.25325 , year=

    Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:2606.27652 , year=

    MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy , author=. arXiv preprint arXiv:2606.27652 , year=

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2402.14762 , year=

    MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues , author=. arXiv preprint arXiv:2402.14762 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2511.00850 , year=

    MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models , author=. arXiv preprint arXiv:2511.00850 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2603.00552 , year=

    EMPA: Evaluating Persona-Aligned Empathy as a Process , author=. arXiv preprint arXiv:2603.00552 , year=

    arXiv

  7. [7]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  8. [8]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    DMT-RoleBench: A Dynamic Multi-Turn Dialogue Based Benchmark for Role-Playing Evaluation of Large Language Model and Agent , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  9. [9]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    EmoBench: Evaluating the Emotional Intelligence of Large Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  10. [10]

    arXiv preprint arXiv:2510.23182 , year=

    SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations , author=. arXiv preprint arXiv:2510.23182 , year=

    arXiv

  11. [11]

    arXiv preprint arXiv:2509.21856 , year=

    KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues , author=. arXiv preprint arXiv:2509.21856 , year=

    arXiv

  12. [12]

    arXiv preprint arXiv:2406.12045 , year=

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. arXiv preprint arXiv:2406.12045 , year=

    Pith/arXiv arXiv

  13. [13]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

  14. [14]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    LLM Evaluators Recognize and Favor Their Own Generations , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  15. [15]

    arXiv preprint arXiv:2509.21117 , year=

    TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them , author=. arXiv preprint arXiv:2509.21117 , year=

    arXiv

  16. [16]

    arXiv preprint arXiv:2305.17926 , year=

    Large Language Models Are Not Fair Evaluators , author=. arXiv preprint arXiv:2305.17926 , year=

    Pith/arXiv arXiv

  17. [17]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    GPTScore: Evaluate as You Desire , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  18. [18]

    arXiv preprint arXiv:2411.15594 , year=

    A Survey on LLM-as-a-Judge , author=. arXiv preprint arXiv:2411.15594 , year=

    Pith/arXiv arXiv

  19. [19]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  20. [20]

    arXiv preprint arXiv:2506.00777 , year=

    Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge , author=. arXiv preprint arXiv:2506.00777 , year=

    Pith/arXiv arXiv

  21. [21]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    SkillAggregation: Reference-free LLM-Dependent Aggregation , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  22. [22]

    arXiv preprint arXiv:2510.12462 , year=

    Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems , author=. arXiv preprint arXiv:2510.12462 , year=

    arXiv

  23. [23]

    ACM Transactions on Information Systems (TOIS) , volume=

    Challenges in Building Intelligent Open-Domain Dialog Systems , author=. ACM Transactions on Information Systems (TOIS) , volume=. 2020 , publisher=

    2020

  24. [24]

    arXiv preprint arXiv:2107.11904 , year=

    Transferable Dialogue Systems and User Simulators , author=. arXiv preprint arXiv:2107.11904 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2505.06120 , year=

    LLMs Get Lost in Multi-Turn Conversation , author=. arXiv preprint arXiv:2505.06120 , year=

    Pith/arXiv arXiv

  26. [26]

    Proceedings of the 25th ACM Conference on Economics and Computation (EC '24) , pages=

    Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? , author=. Proceedings of the 25th ACM Conference on Economics and Computation (EC '24) , pages=

  27. [27]

    arXiv preprint arXiv:2502.00640 , year=

    CollabLLM: From Passive Responders to Active Collaborators , author=. arXiv preprint arXiv:2502.00640 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2504.07114 , year=

    ChatBench: From Static Benchmarks to Human-AI Evaluation , author=. arXiv preprint arXiv:2504.07114 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2502.16761 , year=

    Language Model Fine-tuning on Scaled Survey Data for Predicting Distributions of Public Opinions , author=. arXiv preprint arXiv:2502.16761 , year=

    Pith/arXiv arXiv

  30. [30]

    Dialogue History , author=

    Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue History , author=. arXiv preprint arXiv:2505.21362 , year=

    Pith/arXiv arXiv

  31. [31]

    Nature Machine Intelligence , pages=

    Large Language Models that Replace Human Participants Can Harmfully Misportray and Flatten Identity Groups , author=. Nature Machine Intelligence , pages=. 2025 , publisher=

    2025

  32. [32]

    Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers , pages=

    Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System , author=. Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers , pages=

    2007

  33. [33]

    arXiv preprint arXiv:1612.05688 , year=

    A User Simulator for Task-Completion Dialogues , author=. arXiv preprint arXiv:1612.05688 , year=

    Pith/arXiv arXiv

  34. [34]

    Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST) , pages=

    Generative Agents: Interactive Simulacra of Human Behavior , author=. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST) , pages=

  35. [35]

    Political Analysis , volume=

    Out of One, Many: Using Language Models to Simulate Human Samples , author=. Political Analysis , volume=

  36. [36]

    arXiv preprint arXiv:2406.01633 , year=

    On Overcoming Miscalibrated Conversational Priors in LLM-Based Chatbots , author=. arXiv preprint arXiv:2406.01633 , year=

    Pith/arXiv arXiv

  37. [37]

    ACL , year=

    Personalizing Dialogue Agents: I have a dog, do you have pets too? , author=. ACL , year=

  38. [38]

    arXiv preprint arXiv:2406.20094 , year=

    Scaling Synthetic Data Creation with 1,000,000,000 Personas , author=. arXiv preprint arXiv:2406.20094 , year=

    Pith/arXiv arXiv

  39. [39]

    2025 , howpublished=

    Nemotron-Personas: A Demographically Grounded Persona Dataset , author=. 2025 , howpublished=

    2025

  40. [40]

    ACL , year=

    MuTual: A Dataset for Multi-Turn Dialogue Reasoning , author=. ACL , year=

  41. [41]

    arXiv preprint arXiv:2512.06688 , year=

    PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory , author=. arXiv preprint arXiv:2512.06688 , year=

    arXiv

  42. [42]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  43. [43]

    ACL , year=

    Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset , author=. ACL , year=

  44. [44]

    ACL , year=

    GoEmotions: A Dataset of Fine-Grained Emotions , author=. ACL , year=

  45. [45]

    Imagination, Cognition and Personality , year=

    Emotional Intelligence , author=. Imagination, Cognition and Personality , year=

  46. [46]

    Emotional Intelligence: Why It Can Matter More Than IQ , author=

  47. [47]

    arXiv preprint arXiv:2511.08394 , year=

    Interaction Dynamics as a Reward Signal for LLMs , author=. arXiv preprint arXiv:2511.08394 , year=

    arXiv

  48. [48]

    arXiv preprint , year=

    TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisons , author=. arXiv preprint , year=

  49. [49]

    arXiv preprint arXiv:2504.16939 , year=

    A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions , author=. arXiv preprint arXiv:2504.16939 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2403.15115 , year=

    Language Models in Dialogue: Conversational Maxims for Human-AI Interactions , author=. arXiv preprint arXiv:2403.15115 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2504.04717 , year=

    Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models , author=. arXiv preprint arXiv:2504.04717 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2512.10493 , year=

    Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild , author=. arXiv preprint arXiv:2512.10493 , year=

    arXiv

  53. [53]

    Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages=

    BLEU: A Method for Automatic Evaluation of Machine Translation , author=. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages=

  54. [54]

    Text Summarization Branches Out , pages=

    ROUGE: A Package for Automatic Evaluation of Summaries , author=. Text Summarization Branches Out , pages=

  55. [55]

    ACM Transactions on Information Systems , volume=

    Metaphorical User Simulators for Evaluating Task-Oriented Dialogue Systems , author=. ACM Transactions on Information Systems , volume=. 2024 , publisher=

    2024

  56. [56]

    arXiv preprint arXiv:2511.00222 , year=

    Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning , author=. arXiv preprint arXiv:2511.00222 , year=

    arXiv

  57. [57]

    arXiv preprint arXiv:2511.03508 , year=

    One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework , author=. arXiv preprint arXiv:2511.03508 , year=

    arXiv

  58. [58]

    arXiv preprint arXiv:2507.20152 , year=

    Goal Alignment in LLM-Based User Simulators for Conversational AI , author=. arXiv preprint arXiv:2507.20152 , year=

    arXiv

  59. [59]

    The Twelfth International Conference on Learning Representations (ICLR) , year=

    WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions , author=. The Twelfth International Conference on Learning Representations (ICLR) , year=

  60. [60]

    Proceedings of the 29th Symposium on Operating Systems Principles (SOSP) , pages=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the 29th Symposium on Operating Systems Principles (SOSP) , pages=

  61. [61]

    Educational and Psychological Measurement , year=

    Estimating the Reliability, Systematic Error and Random Error of Interval Data , author=. Educational and Psychological Measurement , year=

  62. [62]

    Content Analysis: An Introduction to Its Methodology , author=

  63. [63]

    Proceedings of the Royal Society of London , year=

    Notes on Regression and Inheritance in the Case of Two Parents , author=. Proceedings of the Royal Society of London , year=

  64. [64]

    Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , numpages=

    CloChat: Understanding How People Customize, Interact, and Experience Personas in Large Language Models , author=. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , numpages=

    2024

  65. [65]

    arXiv preprint arXiv:2508.10142 , year=

    Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMs , author=. arXiv preprint arXiv:2508.10142 , year=

    Pith/arXiv arXiv

  66. [66]

    arXiv preprint arXiv:2505.02018 , year=

    R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation , author=. arXiv preprint arXiv:2505.02018 , year=

    Pith/arXiv arXiv

  67. [67]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-Following Evaluation for Large Language Models , author=. arXiv preprint arXiv:2311.07911 , year=

    Pith/arXiv arXiv

  68. [68]

    Advances in Neural Information Processing Systems , volume=

    Benchmarking Complex Instruction-Following with Multiple Constraints Composition , author=. Advances in Neural Information Processing Systems , volume=

  69. [69]

    arXiv preprint arXiv:2508.06196 , year=

    EICAP: Deep Dive in Assessment and Enhancement of Large Language Models in Emotional Intelligence through Multi-Turn Conversations , author=. arXiv preprint arXiv:2508.06196 , year=

    arXiv

  70. [70]

    arXiv preprint arXiv:2505.23810 , year=

    MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation , author=. arXiv preprint arXiv:2505.23810 , year=

    arXiv

  71. [71]

    Proceedings of the ACM on Management of Data , volume=

    Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMs , author=. Proceedings of the ACM on Management of Data , volume=. 2025 , publisher=

    2025