OCB introduces File Fidelity Q&A and Domain Q&A tracks for LLM evaluation on native office files, reporting ~59.3% on domain reasoning for top systems with releases of data and tooling.
hub
OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows
15 Pith papers cite this work. Polarity classification is still indexing.
15
Pith papers citing it
abstract
Autonomous agents powered by large language models (LLMs) are increasingly deployed in real-world applications requiring complex, long-horizon workflows. However, existing benchmarks predominantly focus on atomic tasks that are self-contained and independent, failing to capture the long-term contextual dependencies and multi-interaction coordination required in realistic scenarios. To address this gap, we introduce OdysseyBench, a comprehensive benchmark for evaluating LLM agents on long-horizon workflows across diverse office applications including Word, Excel, PDF, Email, and Calendar. Our benchmark comprises two complementary splits: OdysseyBench+ with 300 tasks derived from real-world use cases, and OdysseyBench-Neo with 302 newly synthesized complex tasks. Each task requires agent to identify essential information from long-horizon interaction histories and perform multi-step reasoning across various applications. To enable scalable benchmark creation, we propose HomerAgents, a multi-agent framework that automates the generation of long-horizon workflow benchmarks through systematic environment exploration, task generation, and dialogue synthesis. Our extensive evaluation demonstrates that OdysseyBench effectively challenges state-of-the-art LLM agents, providing more accurate assessment of their capabilities in complex, real-world contexts compared to existing atomic task benchmarks. We believe that OdysseyBench will serve as a valuable resource for advancing the development and evaluation of LLM agents in real-world productivity scenarios. In addition, we release OdysseyBench and HomerAgents to foster research along this line.
hub tools
citation-role summary
background 2
dataset 1
citation-polarity summary
representative citing papers
A new native-runtime benchmark reveals that current frontier AI agents succeed on at most 62 percent of realistic long-horizon CLI tasks.
CUJBench is the first benchmark for cross-modal LLM-agent failure diagnosis, reporting 19.7% accuracy and identifying evidence attribution as the core bottleneck across six models.
Only two of ten advanced AI agents finish a 500-day simulated CEO challenge above the starting cash, and none surpass a hand-tuned rule-based baseline.
GauntletBench reveals frontier AI agents achieve 19.1% success on 100 tasks in video editing, 3D modeling, and similar tools versus over 80% for humans, exposing limitations in overlooked capabilities.
ALEM benchmark reveals LLM agents achieve only ~6% normalized return in open-ended multi-agent settings, with communication as the main driver of coordination and individual task competence not implying coordination competence.
SentinelBench is a new benchmark for time-evolving monitoring tasks in web environments, measuring task completion, reaction time, and resource use with baselines from three models and two harnesses.
GTA-2 benchmark shows frontier models achieve below 50% on atomic tool tasks and only 14.39% success on realistic long-horizon workflows, with execution harnesses like Manus providing substantial gains.
An empirical study evaluating tool-augmented LLM agents on 243 real-world energy analytics problems across data retrieval, knowledge interpretation, and quantitative modeling using domain-specific tools and multi-dimensional scoring.
EnergyAgentBench, a 70-task live-data agentic benchmark, finds Claude Sonnet 4.6 best overall and Haiku 4.5 best on long-horizon siting among nine LLM agents.
Opal enables private long-term memory for personal AI by decoupling reasoning to a trusted enclave with a lightweight knowledge graph and piggybacking reindexing on ORAM accesses.
Finch is a new benchmark with 172 composite workflows and 384 tasks from real enterprise data that shows top AI models like GPT-5.1 Pro pass only 38.4% of workflows under human evaluation.
citing papers explorer
-
Office Comprehension Benchmark
OCB introduces File Fidelity Q&A and Domain Q&A tracks for LLM evaluation on native office files, reporting ~59.3% on domain reasoning for top systems with releases of data and tooling.
-
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
A new native-runtime benchmark reveals that current frontier AI agents succeed on at most 62 percent of realistic long-horizon CLI tasks.
-
CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend
CUJBench is the first benchmark for cross-modal LLM-agent failure diagnosis, reporting 19.7% accuracy and identifying evidence attribution as the core bottleneck across six models.
-
CEO-Bench: Can Agents Play the Long Game?
Only two of ten advanced AI agents finish a 500-day simulated CEO challenge above the starting cash, and none surpass a hand-tuned rule-based baseline.
-
Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments
GauntletBench reveals frontier AI agents achieve 19.1% success on 100 tasks in video editing, 3D modeling, and similar tools versus over 80% for humans, exposing limitations in overlooked capabilities.
-
Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
ALEM benchmark reveals LLM agents achieve only ~6% normalized return in open-ended multi-agent settings, with communication as the main driver of coordination and individual task competence not implying coordination competence.
-
SentinelBench: A Benchmark for Long-Running Monitoring Agents
SentinelBench is a new benchmark for time-evolving monitoring tasks in web environments, measuring task completion, reaction time, and resource use with baselines from three models and two harnesses.
-
GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows
GTA-2 benchmark shows frontier models achieve below 50% on atomic tool tasks and only 14.39% success on realistic long-horizon workflows, with execution harnesses like Manus providing substantial gains.
-
How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?
An empirical study evaluating tool-augmented LLM agents on 243 real-world energy analytics problems across data retrieval, knowledge interpretation, and quantitative modeling using domain-specific tools and multi-dimensional scoring.
-
EnergyAgentBench: Benchmarking LLM Agents on Live Energy Infrastructure Data
EnergyAgentBench, a 70-task live-data agentic benchmark, finds Claude Sonnet 4.6 best overall and Haiku 4.5 best on long-horizon siting among nine LLM agents.
-
Opal: Private Memory for Personal AI
Opal enables private long-term memory for personal AI by decoupling reasoning to a trusted enclave with a lightweight knowledge graph and piggybacking reindexing on ORAM accesses.
-
Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows
Finch is a new benchmark with 172 composite workflows and 384 tasks from real enterprise data that shows top AI models like GPT-5.1 Pro pass only 38.4% of workflows under human evaluation.
- PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflows
- ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks
- Tools as Continuous Flow for Evolving Agentic Reasoning