ALEM benchmark reveals LLM agents achieve only ~6% normalized return in open-ended multi-agent settings, with communication as the main driver of coordination and individual task competence not implying coordination competence.
2601.18137 , archivePrefix =
12 Pith papers cite this work. Polarity classification is still indexing.
12
Pith papers citing it
citation-role summary
dataset 1
citation-polarity summary
years
2026 12roles
dataset 1polarities
use dataset 1representative citing papers
Boiling the Frog is a new stateful multi-turn benchmark that finds an aggregate 44.4% strict attack success rate for incremental safety violations across nine AI models, with rates ranging from 20.5% to 92.9%.
GTA-2 benchmark shows frontier models achieve below 50% on atomic tool tasks and only 14.39% success on realistic long-horizon workflows, with execution harnesses like Manus providing substantial gains.
A separate memory agent that selectively injects reminders into an unmodified action agent improves long-horizon task performance by up to 8.3 percentage points.
Trip+ benchmark evaluates language model agents on generating and revising personalized minute-level travel itineraries under dynamic interactions, finding consistent gaps where models produce feasible but exhausting plans that ignore traveler profiles.
PIVOT refines LLM agent trajectories through plan-inspect-evolve-verify stages using environment feedback, yielding up to 94% relative gains in constraint satisfaction and 3-5x token efficiency over prior refinement methods.
A two-axis taxonomy of student entropy and teacher-student divergence identifies informative tokens in on-policy distillation, allowing near-full performance with 10-50% of tokens.
CivBench trains models on turn-level states in Civilization V to predict victory probabilities, providing a progress-based evaluation of LLM strategic capabilities across 307 games with 7 models.
On-policy reverse-KL self-distillation from a “be concise” teacher cuts reasoning length up to 56% while preserving or slightly improving math accuracy across several models.
AtomisticSkills is a new harness framework with 100+ human-curated skills that lets general AI agents perform atomistic research tasks including simulations, screening, and analysis, shown on electrolyte design, CO2 capture, drug screening, and catalyst tasks.
Interactive evaluation of AI must be reframed as a distinct paradigm that maps interaction trajectories to judgments on process, recoverability, coordination, robustness, and system performance, supported by a two-axis taxonomy and design principles.
On 190 tasks and a 12-direction cross-modal diagnostic, seven embedding models frequently fail to honor explicit target-modality instructions: retrieval is biased toward the query modality and instruction-induced shifts are misaligned.
citing papers explorer
-
Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
ALEM benchmark reveals LLM agents achieve only ~6% normalized return in open-ended multi-agent settings, with communication as the main driver of coordination and individual task competence not implying coordination competence.
-
Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
Boiling the Frog is a new stateful multi-turn benchmark that finds an aggregate 44.4% strict attack success rate for incremental safety violations across nine AI models, with rates ranging from 20.5% to 92.9%.
-
GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows
GTA-2 benchmark shows frontier models achieve below 50% on atomic tool tasks and only 14.39% success on realistic long-horizon workflows, with execution harnesses like Manus providing substantial gains.
-
Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents
A separate memory agent that selectively injects reminders into an unmodified action agent improves long-horizon task performance by up to 8.3 percentage points.
-
Trip+: Benchmarking Agents in Personalized Interactive Travel Planning
Trip+ benchmark evaluates language model agents on generating and revising personalized minute-level travel itineraries under dynamic interactions, finding consistent gaps where models produce feasible but exhausting plans that ignore traveler profiles.
-
PIVOT: Bridging Planning and Execution in LLM Agents via Trajectory Refinement
PIVOT refines LLM agent trajectories through plan-inspect-evolve-verify stages using environment feedback, yielding up to 94% relative gains in constraint satisfaction and 3-5x token efficiency over prior refinement methods.
-
TIP: Token Importance in On-Policy Distillation
A two-axis taxonomy of student entropy and teacher-student divergence identifies informative tokens in on-policy distillation, allowing near-full performance with 10-50% of tokens.
-
CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V
CivBench trains models on turn-level states in Civilization V to predict victory probabilities, providing a progress-based evaluation of LLM strategic capabilities across 307 games with 7 models.
-
CRISP: Compressed Reasoning via Iterative Self-Policy Distillation
On-policy reverse-KL self-distillation from a “be concise” teacher cuts reasoning length up to 56% while preserving or slightly improving math accuracy across several models.
-
Harnessing AtomisticSkills for Agentic Atomistic Research
AtomisticSkills is a new harness framework with 100+ human-curated skills that lets general AI agents perform atomistic research tasks including simulations, screening, and analysis, shown on electrolyte design, CO2 capture, drug screening, and catalyst tasks.
-
Interactive Evaluation Requires a Design Science
Interactive evaluation of AI must be reframed as a distinct paradigm that maps interaction trajectories to judgments on process, recoverability, coordination, robustness, and system performance, supported by a two-axis taxonomy and design principles.
-
MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models
On 190 tasks and a 12-direction cross-modal diagnostic, seven embedding models frequently fail to honor explicit target-modality instructions: retrieval is biased toward the query modality and instruction-induced shifts are misaligned.