REVIEW 6 cited by
Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
We introduce a dynamic benchmarking system for conversational agents that evaluates their performance through a single, simulated, and lengthy user$\leftrightarrow$agent interaction. The interaction is a conversation between the user and agent, where multiple tasks are introduced and then undertaken concurrently. We context switch regularly to interleave the tasks, which constructs a realistic testing scenario in which we assess the Long-Term Memory, Continual Learning, and Information Integration capabilities of the agents. Results from both proprietary and open-source Large-Language Models show that LLMs in general perform well on single-task interactions, but they struggle on the same tasks when they are interleaved. Notably, short-context LLMs supplemented with an LTM system perform as well as or better than those with larger contexts. Our benchmark suggests that there are other challenges for LLMs responding to more natural interactions that contemporary benchmarks have heretofore not been able to capture.
Forward citations
Cited by 6 Pith papers
-
ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents
A unified evaluation framework for proactive dialogue agents, built with 328 synthetic environments across six domains, shows that thinking modes improve target planning but not dialogue guidance in a 22-model comparison.
-
HACo-Det: A Study Towards Fine-Grained Machine-Generated Text Detection under Human-AI Coauthoring
Current machine-generated text detectors, especially metric-based ones, perform poorly on word-level detection in coauthored texts, while finetuned DeBERTa achieves strong but imperfect performance.
-
IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems
IntellAgent uses a policy graph, synthetic event generation, and user simulation to automatically create benchmarks that rank conversational AI agents similarly to tau-bench.
-
Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Tasks
LLM-as-judge scoring of multi-round lateral thinking tasks can be fooled by answer leakage and question substitution, so response-based metrics may overstate reasoning ability.
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
-
Dynamic benchmarking framework for LLM-based conversational data capture
An LLM-based framework that benchmarks conversational data capture using synthetic users, applied to loan applications, shows adaptive follow-up questions improve extraction accuracy.
Discussion (0). Continue with ORCID to comment.