Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

ProactiveEval unifies proactive-dialogue evaluation and finds reasoning helps planning, not guidance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A unified evaluation framework for proactive dialogue agents, built with 328 synthetic environments across six domains, shows that thinking modes improve target planning but not dialogue guidance in a 22-model comparison.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely useful unified benchmark for proactive dialogue, with a solid pipeline and broad model coverage; the empirical headline about thinking models hurting guidance is overstated and possibly inflated by the GPT-4o evaluator, but the resource itself deserves a serious referee. the 3 major comments →

arxiv 2508.20973 v1 pith:CYFWG6BB submitted 2025-08-28 cs.CL cs.AIcs.HC

ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents

Tianjian Liu , Fanqi Wan , Jiajian Guo , Xiaojun Quan This is my paper
classification cs.CL cs.AIcs.HC
keywords proactive dialogueevaluation benchmarkLLM-as-a-judgetarget planningdialogue guidancereasoning modelssynthetic data generationinteractive evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ProactiveEval is a proposal to measure 'proactivity' in dialogue the same way across every scenario that has previously been studied separately. It splits the skill into two stages—target planning (choosing a goal and sub-goals from the user's situation) and dialogue guidance (actually steering a conversation toward that goal)—and supplies a data-generation pipeline plus LLM-as-a-judge scoring. Using 328 synthetic environments across six domains and 22 models, the paper reports that DeepSeek-R1 plans best while Claude-3.7-Sonnet guides best, and that thinking/reasoning modes improve planning but not guidance, which the paper attributes to pushier, less natural opening messages from thinking models. The paper's own Limitations section acknowledges that LLM-as-a-judge may carry biases even though human agreement is high. If the framework holds up, it gives the field a common ruler and a warning that current reasoning training may not transfer to multi-turn conversational steering.

Core claim

On the paper's own terms, the central discovery is a two-part operational definition of proactive dialogue and the empirical pattern it exposes. A proactive agent must first infer, from user information and a trigger factor in the environment, a target T and sub-targets S; it must then run a multi-turn dialogue in which a simulated user with low, medium, or high agreeableness is guided to that target. ProactiveEval scores the first stage by comparing the generated plan against a reference plan (10 = better than reference), and the second by a judge's 1–10 guidance score along effectiveness, personalization, tone, engagement, and naturalness. The paper reports that thinking models consistentl

What carries the argument

The load-bearing machinery is the evaluation environment and the two-task decomposition built on it. Each environment packages user information U and trigger factor F, plus a reference target and 2–4 sub-targets; target planning scores a model's inferred plan against that reference, and dialogue guidance replays the plan through a GPT-4o-simulated user whose agreeableness is set to low, medium, or high, terminating at the target or after six turns. A synthetic-data pipeline (topic tree, target ensemble, obfuscation rewrite, noise injection, difficulty filtering) creates the 328 environments, and a diagnostic called target density—number of sub-targets per message—is used to show why thinking

Load-bearing premise

The whole comparison rests on GPT-4o: it writes the reference targets, plays every user, decides when a conversation has reached its goal, and scores all 22 models, so if its behavior or preferences differ from real users the rankings and the 'thinking doesn't help guidance' finding would not transfer.

What would settle it

Have human participants chat with a sample of the 22 models in the same 328 environments and compare human preference rankings with GPT-4o's guidance scores; if humans rank thinking models' guidance above their non-thinking counterparts, or if swapping the simulated user changes which model wins, the central guidance conclusion fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Researchers get a single yardstick: the same environment, target format, and 1–10 scoring can rank any dialogue model on both planning and guidance, so past domain-specific benchmarks become comparable.
  • Training implications: adding explicit reasoning/thinking is likely to buy planning ability but not conversational guidance; improving guidance may require controlling pacing (one or two sub-targets per message) and message naturalness.
  • Models without thinking can still lead: top-tier non-thinking models such as Claude-3.7-Sonnet and Grok-3 remain competitive or best in guidance and some planning domains, so reasoning is not a prerequisite for proactivity.
  • An explicit target matters a lot: removing the target during guidance cuts scores by roughly 10–26%, and smaller models suffer more, so target quality is a component of guidance quality.
  • The data generator can be rerun to make harder, noisier environments, which matters because the paper notes models will soon approach the ceiling of the current metrics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next test, not run in the paper, would swap GPT-4o's simulated user for a different model or for human participants; if the guidance leaderboard changes, the 'thinking hurts guidance' result is partly an artifact of this one user.
  • The judge prompt's explicit dislike of the passive phrase 'sounds like' makes the tone analysis partly circular: the 'sounds like' probe measures alignment with the judge's stated preference rather than an independent property of good conversation.
  • Target density could become a training objective or evaluation feature: the paper's own data suggest pacing, not just goal achievement, separates good steerers from pushy ones.
  • The glasses-assistant domain having no prior public benchmark suggests the synthesis pipeline could be exported to future assistant form factors (AR glasses, wearables) where public evaluation data do not yet exist.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ProactiveEval proposes a unified evaluation framework for proactive dialogue agents, decomposing proactive dialogue into target planning and dialogue guidance. The paper contributes a synthetic data-generation pipeline with a topic tree, target ensembling, and difficulty refinement, yielding 328 evaluation environments across 6 domains. The authors evaluate 22 LLMs using GPT-4o as judge and simulated user, reporting that DeepSeek-R1 and Claude-3.7-Sonnet lead target planning and dialogue guidance, respectively, and that thinking behavior helps target planning but shows no measurable benefit for dialogue guidance. Human evaluation reports weighted kappa of 0.826 (target planning) and 0.721 (dialogue guidance), and repeated runs show low standard deviations. The central claim is that ProactiveEval provides a valid, unified measure of LLM proactivity.

Significance. If the central claim holds, ProactiveEval would fill a recognized gap: existing proactive dialogue benchmarks are fragmented across domains and metrics, while this framework offers a unified task decomposition, a public dataset, and a reproducible evaluation protocol. The paper ships code, includes 22 models across 6 domains, and provides human-consistency and stability evidence (weighted kappa 0.826/0.721; repeated-run std below 0.28). These are real strengths. The headline results--DeepSeek-R1 and Claude-3.7-Sonnet as leaders, and the nuanced finding that thinking helps planning more than guidance--are actionable for the community, provided the evaluator-robustness concern below is addressed. As it stands, the study's validity rests heavily on GPT-4o's dual role as judge and simulated user, and the paper's own limitations section concedes that LLM-as-a-judge biases may exist.

major comments (3)
  1. [Experiments, Protocols; Listing 12] The central comparative claim that thinking behavior has 'no measurable impact' on dialogue guidance is evaluated entirely through GPT-4o, which generates the reference targets, serves as the judge for both tasks, and plays the simulated user. The guidance rubric (Listing 12) explicitly deducts 2-4 points for messages longer than 2 sentences/50 words, for passive phrasing such as 'sounds like', and for metadata leakage. Figure 4(c) shows thinking models produce higher target density, so the headline result may reflect GPT-4o's stylistic rubric rather than an intrinsic property of reasoning. The human evaluation (n=50, kappa 0.826/0.721) validates score consistency with the judge, but not the realism of the simulated user or the rank ordering across 22 models. The Limitations section concedes 'potential biases and gaps in LLM-as-a-judge may still exist', but no experiment isolates this de
  2. [Further Analysis, 'Effects of Thinking'; Table 2] The text states 'non-thinking models outperform thinking models across all domains' in dialogue guidance. Table 2 contradicts this: in Long-term Follow-up, thinking Claude-3.7-Sonnet scores 9.23 versus 9.10 for non-thinking Claude-3.7-Sonnet, and in System Operation it scores 8.40 versus 8.36. The accurate empirical statement is that thinking models generally underperform on average and in most, but not all, domains. This overclaim is load-bearing because the paper's key practical message is that reasoning modes help planning but not conversation steering. The claim should be revised to match the table.
  3. [Environment & Target Generation; Data Refinement] The target planning evaluation measures models against reference targets that are generated by GPT-4o through high-temperature sampling and target ensembling, and are then validated by LLM judges. This is not circularity of the fitted-parameter kind, but it does mean the target planning leaderboard reflects agreement with GPT-4o's notion of a good target. Models whose planning style resembles GPT-4o may be advantaged. A small human-authored reference set or an alternative generator (e.g., a different LLM) used as a cross-check would substantially strengthen the claim that the benchmark measures general proactivity rather than GPT-4o conformity.
minor comments (5)
  1. [Appendix, 'Limiations' heading] The limitations section heading is misspelled ('Limiations'), and the appendix heading 'Details of Evalution Models' should be 'Evaluation'.
  2. [Table 5] Table 5 attributes 'DuRecDial 2.0' to 'Li et al. 2024', but the reference list's Li et al. 2024 entry is 'Hello again! ...'. The DuRecDial 2.0 citation should be Liu et al. 2021b.
  3. [Table 2 and 'Details of Evalution Models'] The non-thinking model is listed as 'Qwen-3-235B-A22B-0725' in Table 2 but as 'Qwen3-235B-A22B-2507' in the model details. This inconsistency should be resolved.
  4. [Abstract/Table 1] The domain name is sometimes 'Glasses Assistant' and sometimes 'Glass Assistants'; please standardize.
  5. [Listing 11 and Figure 2 caption] Listing 11's header has a typo ('arget planning evaluation'). Figure 2's caption repeats '(a)' and '(b)' labels, which is confusing because the subfigures already have their own labels.

Circularity Check

0 steps flagged

No significant circularity: ProactiveEval is a benchmark construction and evaluation study; its dataset synthesis, judge, and simulated-user choices are not derived from the conclusions they support, and the self-citations are not load-bearing.

full rationale

Walking the paper's derivation chain, there is no step in which a claimed result reduces to its own input by construction. The central contribution is a benchmark: a two-task decomposition of proactive dialogue (target planning and dialogue guidance), a synthetic data pipeline (topic tree, environment generation, target ensemble, refinement), and an LLM-as-a-judge evaluation protocol. The headline findings ('DeepSeek-R1 and Claude-3.7-Sonnet exhibit exceptional performance on target planning and dialogue guidance tasks, respectively'; 'thinking shows no measurable impact on dialogue guidance') are empirical measurements on 22 models, not derivations from fitted parameters. The judge and simulated user are both GPT-4o ('we employ GPT-4o as the judge model for both tasks, which also serves as simulated user in dialogue guidance'), and this creates a legitimate construct-validity/robustness concern: the reference targets, the judge's stylistic preferences, and the simulated user's behavior all come from one model, so the leaderboard partly measures conformity to GPT-4o's notion of a good target and a good conversation. However, this is not circularity in the technical sense used here: no equation is identical to an input, no fitted parameter is relabeled as a prediction, and no prohibited uniqueness or ansatz is imported from a self-citation. The dialogue-guidance rubric in Listing 12 explicitly penalizes messages over two sentences/50 words, passive phrasing like 'sounds like', and metadata leakage; since Figure 4(c) shows thinking models have higher target density, the conclusion that thinking does not help guidance is partly an artifact of that rubric. But this is an evaluation-design bias, not a circular derivation: the metric is stated a priori and applied uniformly, and the paper's own Limiations section concedes 'potential biases and gaps in LLM-as-a-judge may still exist'. External anchors exist: human evaluation on 50 samples (weighted kappa 0.826/0.721) and an IFEval comparison. Self-citations (e.g., Liu et al. 2024b ComPeer; Wan et al. 2023 Explore-Instruct) appear as domain sources or methodological inspiration and are not used to justify the framework's decomposition or to exclude alternatives. No step meets the evidentiary standard for circularity, so the appropriate score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claims rest on the two-task decomposition, the scope of six domains, and the validity of GPT-4o as generator, judge, and simulated user. These are domain assumptions from prior work and the paper's own setup; they are partially validated by the human kappa and stability runs, but not independently established.

free parameters (5)
  • Maximum dialogue turns (I) = 6
    Hand-set in Experimental Setups; longer dialogues may allow models to eventually reach targets, potentially inflating guidance scores.
  • Memory window = 3 turns
    Hand-set; only the most recent 3 turns are conditioned on, which could influence guidance results for long dialogues.
  • Candidate targets per ensemble (n) = 5
    Hand-set for target ensemble; larger n could change reference target quality.
  • Evaluation temperature = 0
    Temperature set to 0 for all evaluated models; greedy decoding may not reflect typical deployment.
  • Agreeableness levels = low/medium/high
    Three hand-picked tiers of simulated user resistance in dialogue guidance.
axioms (6)
  • domain assumption Proactive dialogue can be fully decomposed into target planning and dialogue guidance.
    Adopted from Deng et al. 2023b, 2025; everything in the paper (tasks, metrics, data generation) presupposes this division.
  • domain assumption The six domains (recommendation, persuasion, ambiguous instruction, long-term follow-up, system operation, glasses assistant) are the mainstream proactive dialogue domains.
    The framework claims unity by covering these six; the choice is justified by prior work but is a scope assumption.
  • domain assumption GPT-4o-generated reference targets, refined by a target ensemble and validated by 5 LLMs, are a reliable gold standard for target planning.
    The target-planning judge compares every model output to this reference; no human gold-standard annotation is used, only a 50-sample human kappa check.
  • domain assumption A GPT-4o simulated user with Big Five agreeableness levels approximates real human behavior in proactive dialogue.
    Dialogue guidance scores depend entirely on the simulated user; this is not validated beyond the paper's own internal checks.
  • domain assumption Obfuscation rewriting and noise injection increase difficulty without changing the reference target's validity.
    The final-check step uses LLMs to confirm the target remains best, but the assumption that harder-to-predict environments remain fair is stated, not proven.
  • domain assumption LLM-as-a-judge scores are proportional to human judgments.
    Supported only by weighted kappa on 50 samples (0.826 and 0.721); not a general proof of unbiasedness.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents." pith.science (2026). https://pith.science/paper/CYFWG6BB

@misc{pith2026250820973,
  author       = {Pith},
  title        = {Pith review of: ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYFWG6BB}},
  note         = {Machine review of arXiv:2508.20973}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Proactive dialogue has emerged as a critical and challenging research problem in advancing large language models (LLMs). Existing works predominantly focus on domain-specific or task-oriented scenarios, which leads to fragmented evaluations and limits the comprehensive exploration of models' proactive conversation abilities. In this work, we propose ProactiveEval, a unified framework designed for evaluating proactive dialogue capabilities of LLMs. This framework decomposes proactive dialogue into target planning and dialogue guidance, establishing evaluation metrics across various domains. Moreover, it also enables the automatic generation of diverse and challenging evaluation data. Based on the proposed framework, we develop 328 evaluation environments spanning 6 distinct domains. Through experiments with 22 different types of LLMs, we show that DeepSeek-R1 and Claude-3.7-Sonnet exhibit exceptional performance on target planning and dialogue guidance tasks, respectively. Finally, we investigate how reasoning capabilities influence proactive behaviors and discuss their implications for future model development.

Figures

Figures reproduced from arXiv: 2508.20973 by Fanqi Wan, Jiajian Guo, Tianjian Liu, Xiaojun Quan.

Figure 1
Figure 1. Figure 1: Interaction diagram for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the evaluation dataset generation pipeline. The pipeline is mainly divided into (a) data synthesis: gener [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The features and statistic for ProactiveEval. The [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The related analytical results in discussion. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The examples of different dialogue guidance. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The environment change after the refinement. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ProactBench: Beyond What The User Asked For

    cs.LG 2026-05 unverdicted novelty 7.0

    ProactBench measures LLM conversational proactivity in three phases using 198 multi-agent dialogues and finds recovery behavior hard to predict from existing benchmarks.

  2. TriggerBench: Investigating Prospective Memory for Large Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    TriggerBench is a new benchmark showing prospective memory in LLMs is harder than retrospective memory, exhibits precision-recall trade-offs, and may indicate spare reasoning capacity.

  3. Do Proactive Agents Really Need an LLM to Decide When to Wake and What to Anchor?

    cs.CL 2026-05 unverdicted novelty 6.0

    A temporal-graph model on structured event streams replaces per-event LLM calls for trigger decisions in proactive agents, reporting mean F1 gains of 16.7 and 4-83x speedups.

Reference graph

Works this paper leans on

55 extracted references · 30 canonical work pages · cited by 3 Pith papers · 5 internal anchors

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Dynamic benchmarking framework for LLM-based conversational data capture

    Aluffi, P. A.; Zietkiewicz, P.; Bazzi, M.; Arderne, M.; and Murevics, V. 2025. Dynamic benchmarking framework for LLM-based conversational data capture. arXiv preprint arXiv:2502.04349

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Anthropic. 2025. Claude 3.7 Sonnet and Claude Code

    work page 2025

  5. [5]

    Bai, G.; Liu, J.; Bu, X.; He, Y.; Liu, J.; Zhou, Z.; Lin, Z.; Su, W.; Ge, T.; Zheng, B.; et al. 2024. MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7421--7454

    work page 2024

  6. [6]

    Cai, R.; Janaka, N.; Kim, H.; Chen, Y.; Zhao, S.; Huang, Y.; and Hsu, D. 2025. AiGet: Transforming Everyday Moments into Hidden Knowledge Discovery with AI Assistance on Smart Glasses. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1--26

    work page 2025

  7. [7]

    Cao, M.; Zhang, T.; Li, M.; Zhang, C.; Liu, Y.; Duan, H.; Zhang, S.; and Chen, K. 2025. Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement. arXiv:2501.12273

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    Castillo-Bolado, D.; Davidson, J.; Gray, F.; and Rosa, M. 2024. Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models. arXiv preprint arXiv:2409.20222

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    P.; and Gerosa, M

    Chaves, A. P.; and Gerosa, M. A. 2021. How should my chatbot interact? A survey on social characteristics in human--chatbot interaction design. International Journal of Human--Computer Interaction, 37(8): 729--758

    work page 2021

  10. [10]

    Chen, V.; Zhu, A.; Zhao, S.; Mozannar, H.; Sontag, D.; and Talwalkar, A. 2024. Need Help? Designing Proactive AI Assistants for Programming. arXiv preprint arXiv:2410.04596

    Pith/arXiv arXiv 2024

  11. [11]

    Cohen, J. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4): 213

    work page 1968

  12. [12]

    T.; McCrae, R

    Costa Jr, P. T.; McCrae, R. R.; and Dye, D. A. 1991. Facet scales for agreeableness and conscientiousness: A revision of the NEO Personality Inventory. Personality and individual Differences, 12(9): 887--898

    work page 1991

  13. [13]

    A.; and Antheunis, M

    Croes, E. A.; and Antheunis, M. L. 2021. Can we be friends with Mitsuku? A longitudinal study on the process of relationship formation between humans and a social chatbot. Journal of Social and Personal Relationships, 38(1): 279--300

    work page 2021

  14. [14]

    DeepMind, G. 2025. Gemini Pro

    work page 2025

  15. [15]

    Deng, Y.; Lei, W.; Lam, W.; and Chua, T.-S. 2023 a . A survey on proactive dialogue systems: Problems, methods, and prospects. arXiv preprint arXiv:2305.02750

    Pith/arXiv arXiv 2023

  16. [16]

    Deng, Y.; Liao, L.; Chen, L.; Wang, H.; Lei, W.; and Chua, T.-S. 2023 b . Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. arXiv preprint arXiv:2305.13626

    Pith/arXiv arXiv 2023

  17. [17]

    H.; Lam, W.; and Chua, T.-S

    Deng, Y.; Liao, L.; Lei, W.; Yang, G. H.; Lam, W.; and Chua, T.-S. 2025. Proactive conversational ai: A comprehensive survey of advancements and opportunities. ACM Transactions on Information Systems, 43(3): 1--45

    work page 2025

  18. [18]

    Deng, Y.; Zhang, W.; Lam, W.; Ng, S.-K.; and Chua, T.-S. 2024. Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents. In The Twelfth International Conference on Learning Representations

    work page 2024

  19. [19]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv e-prints, arXiv--2407

    2024

  20. [20]

    Fourrier, C.; Habib, N.; Lozovskaya, A.; Szafer, K.; and Wolf, T. 2024. Open LLM Leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

    2024

  21. [21]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  22. [22]

    P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al

    Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv 2024

  23. [23]

    Jin, C.; Ren, K.; Kong, L.; Wang, X.; Song, R.; and Chen, H. 2024. Persuading across Diverse Domains: a Dataset and Persuasion Large Language Model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1678--1706

    work page 2024

  24. [24]

    Y.-Y.; Turakhia, D

    Li, C.; Wu, G.; Chan, G. Y.-Y.; Turakhia, D. G.; Castelo Quispe, S.; Li, D.; Welch, L.; Silva, C.; and Qian, J. 2025 a . Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1--24

    work page 2025

  25. [25]

    Li, H.; Yang, C.; Zhang, A.; Deng, Y.; Wang, X.; and Chua, T.-S. 2024. Hello again! llm-powered personalized agent for long-term dialogue. arXiv preprint arXiv:2406.05925

    Pith/arXiv arXiv 2024

  26. [26]

    Li, R.; Zhu, C.; Xu, B.; Wang, X.; and Mao, Z. 2025 b . Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach. arXiv preprint arXiv:2504.15784

    Pith/arXiv arXiv 2025

  27. [27]

    Li, X.; Yu, Z.; Zhang, Z.; Chen, X.; Zhang, Z.; Zhuang, Y.; Sadagopan, N.; and Beniwal, A. 2025 c . When thinking fails: The pitfalls of reasoning for instruction-following in llms. arXiv preprint arXiv:2505.11423

    Pith/arXiv arXiv 2025

  28. [28]

    Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    Pith/arXiv arXiv 2024

  29. [29]

    Liu, S.; Zheng, C.; Demasi, O.; Sabour, S.; Li, Y.; Yu, Z.; Jiang, Y.; and Huang, M. 2021 a . Towards emotional support dialog systems. arXiv preprint arXiv:2106.01144

    Pith/arXiv arXiv 2021

  30. [30]

    Liu, T.; Zhao, H.; Liu, Y.; Wang, X.; and Peng, Z. 2024 b . Compeer: A generative conversational agent for proactive peer support. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 1--22

    work page 2024

  31. [31]

    Liu, Z.; Wang, H.; Niu, Z.-Y.; Wu, H.; and Che, W. 2021 b . DuRecDial 2.0: A bilingual parallel corpus for conversational recommendation. arXiv preprint arXiv:2109.08877

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  32. [32]

    Lu, Y.; Yang, S.; Qian, C.; Chen, G.; Luo, Q.; Wu, Y.; Wang, H.; Cong, X.; Zhang, Z.; Lin, Y.; et al. 2024. Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance. arXiv preprint arXiv:2410.12361

    Pith/arXiv arXiv 2024

  33. [33]

    Meta AI . 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation

    work page 2025

  34. [34]

    Niu, C.; Wang, X.; Cheng, X.; Song, J.; and Zhang, T. 2024. Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8724--8741

    work page 2024

  35. [35]

    OpenAI. 2024. Learning to Reason with LLMs

    work page 2024

  36. [36]

    Qian, C.; He, B.; Zhuang, Z.; Deng, J.; Qin, Y.; Cong, X.; Zhang, Z.; Zhou, J.; Lin, Y.; Liu, Z.; et al. 2024. Tell me more! towards implicit user intention understanding of language model driven agents. arXiv preprint arXiv:2402.09205

    Pith/arXiv arXiv 2024

  37. [37]

    Wan, F.; Huang, X.; Yang, T.; Quan, X.; Bi, W.; and Shi, S. 2023. Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Exploration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 9435--9454

    work page 2023

  38. [38]

    Wan, Q.; Hu, S.; Zhang, Y.; Wang, P.; Wen, B.; and Lu, Z. 2024. ``It Felt Like Having a Second Mind''': Investigating Human-AI Co-creativity in Prewriting with Large Language Models. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1): 1--26

    work page 2024

  39. [39]

    Wang, J.; Cheng, Y.; Lin, D.; Leong, C.; and Li, W. 2023. Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 1132--1143

    work page 2023

  40. [40]

    Wang, R.; Yu, H.; Zhang, W.; Qi, Z.; Sap, M.; Neubig, G.; Bisk, Y.; and Zhu, H. 2024. SOTOPIA-pi: Interactive Learning of Socially Intelligent Language Agents. arXiv preprint arXiv:2403.08715

    Pith/arXiv arXiv 2024

  41. [41]

    Wang, X.; Shi, W.; Kim, R.; Oh, Y.; Yang, S.; Zhang, J.; and Yu, Z. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint arXiv:1906.06725

    Pith/arXiv arXiv 2019

  42. [42]

    Xu, F.; Hao, Q.; Zong, Z.; Wang, J.; Zhang, Y.; Wang, J.; Lan, X.; Gong, J.; Ouyang, T.; Meng, F.; et al. 2025. Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models. arXiv preprint arXiv:2501.09686

    Pith/arXiv arXiv 2025

  43. [43]

    Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025 a . Qwen3 Technical Report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  44. [44]

    Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  45. [45]

    Yang, B.; Guo, Y.; Xu, L.; Yan, Z.; Chen, H.; Xing, G.; and Jiang, X. 2025 b . Socialmind: Llm-based proactive ar social assistive system with human-like perception for in-situ live interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(1): 1--30

    work page 2025

  46. [46]

    Yao, S.; Shinn, N.; Razavi, P.; and Narasimhan, K. 2024. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045

    Pith/arXiv arXiv 2024

  47. [47]

    Zhang, Q.; Wang, Y.; YU, T.; Jiang, Y.; Wu, C.; Li, L.; Wang, Y.; Jiang, X.; Shang, L.; Tang, R.; Lyu, F.; and Ma, C. 2025. RevisEval: Improving LLM -as-a-Judge via Response-Adapted References. In The Thirteenth International Conference on Learning Representations

    work page 2025

  48. [48]

    Zhang, T.; Huang, C.; Deng, Y.; Liang, H.; Liu, J.; Wen, Z.; Lei, W.; and Chua, T.-S. 2024 a . Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulation. arXiv preprint arXiv:2403.06769

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  49. [49]

    Zhang, T.; Qin, P.; Deng, Y.; Huang, C.; Lei, W.; Liu, J.; Jin, D.; Liang, H.; and Chua, T.-S. 2024 b . CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 10746--10766

    work page 2024

  50. [50]

    Zhang, T.; Zhang, X.; Zhao, J.; Zhou, L.; and Jin, Q. 2024 c . ESCoT: Towards Interpretable Emotional Support Dialogue Systems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 13395--13412

    work page 2024

  51. [51]

    Zhang, X.; Deng, Y.; Ren, Z.; Ng, S.-K.; and Chua, T.-S. 2024 d . Ask-before-plan: Proactive language agents for real-world planning. arXiv preprint arXiv:2406.12639

    Pith/arXiv arXiv 2024

  52. [52]

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595--46623

    work page 2023

  53. [53]

    Zheng, Z.; Liao, L.; Deng, Y.; Lim, E.-P.; Huang, M.; and Nie, L. 2024. Thoughts to target: Enhance planning for target-driven conversation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 21108--21124

    work page 2024

  54. [54]

    Zhou, J.; Lu, T.; Mishra, S.; Brahma, S.; Basu, S.; Luan, Y.; Zhou, D.; and Hou, L. 2023 a . Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    Pith/arXiv arXiv 2023

  55. [55]

    Zhou, X.; Zhu, H.; Mathur, L.; Zhang, R.; Yu, H.; Qi, Z.; Morency, L.-P.; Bisk, Y.; Fried, D.; Neubig, G.; et al. 2023 b . Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.